Data information acquisition method, device, computer equipment and storage medium
By configuring and executing multiple information crawling programs in parallel, the problem of low data information acquisition efficiency in the prior art is solved, and massive data information is efficiently acquired and equipment burden is reduced.
Patent Information
- Application Number
- CN201811559342.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2018-12-19
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2038-12-19
AI Technical Summary
The existing data information acquisition methods are inefficient, resulting in a long acquisition time and a large burden on the equipment.
By receiving the configuration information input by the user, the preset crawler template is configured, multiple information crawler programs are generated, and executed in parallel to obtain multiple data information, and integrated to obtain target data information.
It greatly reduces the operating pressure of equipment and improves the efficiency of data and information acquisition.
Smart Images

Figure CN109582853B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a data information acquisition method, device, computer equipment and storage medium. Background Art
[0002] When acquiring data information, since the amount of data to be acquired is large, it is necessary to execute an information crawling program to obtain relevant information. However, the time required to acquire information by executing a crawling program is long, and executing a crawling program will bring a heavy burden to the execution device. Therefore, the existing data information acquisition method has the problem of low information acquisition efficiency. Summary of the invention
[0003] The embodiments of the present invention provide a data information acquisition method, device, computer equipment and storage medium, aiming to solve the problem of low information acquisition efficiency in the data information acquisition method in the existing technical methods.
[0004] In a first aspect, an embodiment of the present invention provides a method for acquiring data information, which includes:
[0005] If the configuration information input by the user is received, the preset crawling program template is configured according to the configuration information to obtain multiple information crawling programs that have been configured;
[0006] Generate prompt information according to the quantity information in the configuration information to remind the user that the program configuration has been completed;
[0007] If execution information fed back by the user according to the prompt information is received, multiple information crawling programs are executed in parallel to obtain corresponding multiple copies of data information and store them;
[0008] The obtained multiple pieces of data information are integrated according to the configuration information to obtain target data information.
[0009] In a second aspect, an embodiment of the present invention provides a data information acquisition device, comprising:
[0010] A program configuration unit, configured to configure a preset crawling program template according to the configuration information to obtain a plurality of information crawling programs that have been configured if configuration information input by a user is received;
[0011] A prompt information sending unit, used to generate prompt information according to the quantity information in the configuration information to remind the user that the program configuration has been completed;
[0012] A data information acquisition unit, configured to execute multiple information crawling programs in parallel to acquire and store corresponding multiple copies of data information upon receiving execution information fed back by the user according to the prompt information;
[0013] The data information integration unit is used to integrate the obtained multiple pieces of data information according to the configuration information to obtain target data information.
[0014] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data information acquisition method described in the first aspect when executing the computer program.
[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the data information acquisition method described in the first aspect above.
[0016] The embodiment of the present invention provides a data information acquisition method, device, computer equipment and storage medium. The preset crawling program template is configured by the configuration information input by the user to obtain multiple information crawling programs, the multiple information crawling programs are executed in parallel according to the execution information fed back by the user to obtain multiple data information, the multiple data information are integrated to obtain the target data information, and the multiple information crawling programs are executed in parallel to obtain massive data information, and the operating pressure of the equipment can be greatly reduced, which greatly improves the efficiency of enterprises in obtaining data information. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.
[0018] Figure 1 A schematic diagram of a flow chart of a method for acquiring data information provided by an embodiment of the present invention;
[0019] Figure 2 A schematic diagram of a sub-process of a method for acquiring data information provided by an embodiment of the present invention;
[0020] Figure 3 A schematic diagram of another sub-process of the data information acquisition method provided by an embodiment of the present invention;
[0021] Figure 4 A schematic diagram of another sub-process of the data information acquisition method provided by an embodiment of the present invention;
[0022] Figure 5A schematic diagram of another sub-process of the data information acquisition method provided by an embodiment of the present invention;
[0023] Figure 6 A schematic block diagram of a data information acquisition device provided by an embodiment of the present invention;
[0024] Figure 7 A schematic block diagram of a subunit of a data information acquisition device provided in an embodiment of the present invention;
[0025] Figure 8 A schematic block diagram of another subunit of the data information acquisition device provided in an embodiment of the present invention;
[0026] Fig. 9 A schematic block diagram of another subunit of the data information acquisition device provided in an embodiment of the present invention;
[0027] Fig.10 A schematic block diagram of another subunit of the data information acquisition device provided in an embodiment of the present invention;
[0028] Fig.11 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0030] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0031] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0032] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0033] See also Figure 1 , Figure 1 1 is a flow chart of a data information acquisition method provided by an embodiment of the present invention. The data information acquisition method is applied to a user terminal, and the method is executed by an application software installed in the user terminal. The user terminal is a terminal device used to execute the data information acquisition method to acquire data information, such as a desktop computer, a laptop computer, a tablet computer or a mobile phone.
[0034] like Figure 1 As shown, the method includes steps S110 to S140.
[0035] S110: If the configuration information input by the user is received, the preset crawling program template is configured according to the configuration information to obtain a plurality of information crawling programs that have been configured.
[0036] If the configuration information input by the user (user of the user terminal) is received, the preset crawler program template is configured according to the configuration information to obtain multiple information crawler programs that have been configured. In order to obtain a large amount of data information in a short period of time, it is necessary to execute multiple information crawler programs at the same time to obtain data information to improve the efficiency of information acquisition. Before obtaining data information, the preset crawler program template can be configured through the configuration information to obtain multiple information crawler programs. The preset crawler program template is a pre-designed program template, and the parameter values of each parameter in the preset crawler program template are all empty.
[0037] Specifically, the configuration information is the information input by the user for configuring the preset crawler program template, the configuration information includes quantity information, matrix information, owner information, data type, interface information, etc., wherein the quantity information is the number of information crawlers required to run concurrently, the matrix information is the information used to configure the longitude and latitude information in the preset crawler program template, the owner information is the information used to configure the Key file in the preset crawler program template, each owner corresponds to a Key value in a Key file, the data type is the type information used to configure the type of data information to be collected, and the interface information is the information used to configure the interface required for data interaction of the preset crawler program template.
[0038] The parameters configured in the information crawling program include longitude and latitude information, Key file, acquisition type, and acquisition interface, wherein the longitude and latitude information is the matrix range information used to collect data information in the corresponding area, and the longitude and latitude information contains one or more sets of longitude and latitude coordinates consisting of two coordinate points, and the line connecting the two coordinate points contained in a set of longitude and latitude coordinates is neither on the longitude nor on the latitude, and a rectangular area can be determined by the two coordinate points in a set of longitude and latitude coordinates, which is the area used to collect the corresponding data information; a Key file contains a Key value, and the Key value is the identification code corresponding to the acquirer of the corresponding data information; the acquisition type is the type information used to acquire the corresponding type of data information, for example, the acquisition type can be configured as hotel, food, scenic spot, etc.; the acquisition interface is the network address information of the interface for acquiring the corresponding data information, and each acquisition interface corresponds to a server end, and the acquisition interface can be used to interact with the corresponding server end.
[0039] In one embodiment, if Figure 2 As shown, step S110 includes sub-steps S111, S112, S113, S114 and S115.
[0040] S111. Copy the preset crawling program template according to the quantity information in the configuration information to obtain multiple crawling program templates.
[0041] Obtain the quantity information in the configuration information, and copy the preset crawler program template according to the quantity information to obtain a corresponding number of crawler program templates, that is, the number of copied crawler program templates is the same as the data information, and the parameter values of each parameter in the preset crawler program template are all empty, that is, the parameter values in the multiple copied crawler program templates are all empty.
[0042] S112. Configure the longitude and latitude information in multiple crawler templates respectively according to the matrix information in the configuration information.
[0043] The longitude and latitude information in multiple crawler templates are configured respectively according to the matrix information in the configuration information. The matrix information contains multiple matrices, each of which can contain multiple groups of longitude and latitude coordinates, each group of longitude and latitude coordinates contains two coordinate points, and one matrix in the matrix information can be used to configure the longitude and latitude information in one crawler template, and the longitude and latitude information in all crawler templates can be configured according to the matrix information.
[0044] S113. Configure the Key files in multiple crawler program templates respectively according to the owner information in the configuration information.
[0045] The Key files in the multiple crawler program templates are configured respectively according to the owner information in the configuration information. The owner information includes multiple acquisition person identification codes (IDs), and one identification code can be used to configure the Key file in one crawler program template. Then, the Key files in all crawler program templates can be configured according to the owner information.
[0046] S114. Configure the acquisition types in multiple crawler templates respectively according to the data types in the configuration information.
[0047] The acquisition types in the plurality of crawler program templates are configured respectively according to the data type in the configuration information. The acquisition types in all crawler program templates only contain one parameter value, and the acquisition types in all crawler program templates can be configured according to the data type.
[0048] S115. According to the interface information in the configuration information, respectively configure the acquisition interfaces in the multiple crawler program templates to obtain multiple information crawler programs.
[0049] According to the interface information in the configuration information, the acquisition interfaces in the multiple crawler program templates are configured respectively to obtain multiple information crawler programs. The acquisition interfaces in all crawler program templates only contain one parameter value. The acquisition interfaces in all crawler program templates can be configured through the interface information to obtain multiple information crawler programs.
[0050] S120: Generate prompt information according to the quantity information in the configuration information to remind the user that the program configuration is completed.
[0051] A prompt message indicating that the configuration has been completed is generated according to the quantity information in the configuration information and sent to the user, so as to remind the user that the program configuration has been completed. After configuring multiple information crawling programs, corresponding prompt messages can be generated according to the quantity information in the configuration information and sent to the user, and the user can perform subsequent operations after receiving the prompt message.
[0052] For example, if the quantity information in the configuration information is "47", a prompt message "47 information crawlers have been configured" is generated and sent to the user.
[0053] S130: If execution information fed back by the user according to the prompt information is received, multiple information crawling programs are executed in parallel to obtain and store corresponding multiple copies of data information.
[0054] If the execution information fed back by the user according to the prompt information is received, multiple information crawling programs are executed in parallel to obtain and store corresponding multiple copies of data information. Specifically, after the user receives the prompt information of the completed configuration, the execution information can be sent to execute the configured multiple information crawling programs in parallel, and each information crawling program obtains a corresponding piece of data information, so multiple copies of data information can be obtained and saved.
[0055] In one embodiment, if Figure 3 As shown, step S130 includes sub-steps S131 and S132.
[0056] S131. According to the acquisition interface of the information crawling program, the acquisition type, key file, and longitude and latitude information configured by the information crawling program are sent to the server end of the corresponding acquisition interface.
[0057] According to the acquisition interface in the information crawling program, the acquisition type, key file, and longitude and latitude information configured in the information crawling program are sent to the server side of the corresponding acquisition interface. Specifically, each acquisition interface corresponds to a server side. By sending the acquisition type, key file, and longitude and latitude information in an information crawling program to the server side corresponding to the acquisition interface of the information crawling program, a copy of the data information fed back by the server side can be obtained. After the server side verifies the Key value in the Key file, it searches for the acquisition type keyword in the rectangular area determined by each set of longitude and latitude coordinates in the longitude and latitude information, and obtains the corresponding data information and provides feedback.
[0058] S132, obtaining multiple copies of data information fed back by the server corresponding to the interface in multiple information crawling programs and storing them.
[0059] The data information fed back by the server corresponding to the acquisition interface in each information crawling program is obtained, that is, one information crawling program obtains a piece of data information, and the obtained multiple pieces of data information are stored.
[0060] S140: Integrate the obtained multiple pieces of data information according to the configuration information to obtain target data information.
[0061] The obtained multiple pieces of data information are integrated according to the configuration information to obtain the target data information. Specifically, the configuration information also includes a preset threshold value. Among the obtained multiple pieces of data information, there is data information with a large number of data items. If the number of data items in a piece of data information does not exceed the preset threshold value in the configuration information, then the piece of data information meets the requirements. According to the judgment result, all the data information that meets the requirements is obtained, and the target data is obtained by classifying and sorting according to the data type in the configuration information.
[0062] In one embodiment, if Figure 4 As shown, step S140 includes sub-steps S141 and S142.
[0063] S141. Determine whether the number of data items in each of the multiple pieces of data information is less than a preset threshold in the configuration information to obtain a determination result.
[0064] It is judged whether the number of data items in each of the multiple data information is less than the preset threshold in the configuration information to obtain a judgment result. If the number of data items in a certain data information does not exceed the preset threshold in the configuration information, then the data information meets the requirements, and the number of data items in the obtained multiple data information is judged, and the judgment result is that the number of data items in the multiple data information is less than the preset threshold or the number of data items in the multiple data information is less than the preset threshold.
[0065] For example, the preset threshold is 500, and the number of data items in a certain data information is 350, then the data information meets the requirement.
[0066] S142: If the judgment result is that the number of data items in each of the multiple pieces of data information is less than the preset threshold, the data information is classified and sorted according to the data types corresponding to the multiple pieces of data information in the configuration information to obtain the target data.
[0067] If the result of the judgment is that the number of data items in each of the multiple pieces of data information is less than the preset threshold, the data information is classified and sorted according to the data types corresponding to the multiple pieces of data information in the configuration information to obtain the target data. Specifically, since the acquisition type in the information crawling program can be configured as multiple types when acquiring data information, the data information acquired of different data types can be classified and sorted according to the corresponding data types in the configuration information, and data information of the same data type is classified into one category.
[0068] In one embodiment, if Figure 5 As shown, step S141a is further included after step S141.
[0069] S141a. If the judgment result is that the number of data items in not all of the multiple data information is less than the preset threshold, an alarm prompt message is issued to remind the user that too much data information has been sent.
[0070] If the result of the judgment is that the number of data items in not all data items in the multiple data information is less than the preset threshold, an alarm prompt message is issued to remind the user that too much data information is sent. If the number of data items in a certain data information exceeds the preset threshold in the configuration information, an alarm prompt message needs to be sent to the user to prompt the user to further process the data information that exceeds the preset threshold.
[0071] The preset crawler program template is configured through the configuration information input by the user to obtain multiple information crawlers. The multiple information crawler programs are executed in parallel according to the execution information fed back by the user to obtain multiple copies of data information. The multiple copies of data information are integrated to obtain target data information. By executing multiple information crawler programs in parallel, massive data information can be acquired, which can greatly reduce the operating pressure of the equipment and greatly improve the efficiency of the enterprise in acquiring data information.
[0072] The embodiment of the present invention further provides a data information acquisition device, which is used to execute any embodiment of the above-mentioned data information acquisition method. Figure 6 , Figure 6 1 is a schematic block diagram of a data information acquisition device provided by an embodiment of the present invention. The data information acquisition device can be configured in a user terminal.
[0073] like Figure 6 As shown, the data information acquisition device 100 includes a program configuration unit 110 , a prompt information sending unit 120 , a data information acquisition unit 130 and a data information integration unit 140 .
[0074] The program configuration unit 110 is used to configure the preset crawling program template according to the configuration information to obtain a plurality of configured information crawling programs if configuration information input by the user is received.
[0075] If the configuration information input by the user (user of the user terminal) is received, the preset crawler program template is configured according to the configuration information to obtain multiple information crawler programs that have been configured. In order to obtain a large amount of data information in a short period of time, it is necessary to execute multiple information crawler programs at the same time to obtain data information to improve the efficiency of information acquisition. Before obtaining data information, the preset crawler program template can be configured through the configuration information to obtain multiple information crawler programs. The preset crawler program template is a pre-designed program template, and the parameter values of each parameter in the preset crawler program template are all empty.
[0076] Specifically, the configuration information is the information input by the user for configuring the preset crawler program template, the configuration information includes quantity information, matrix information, owner information, data type, interface information, etc., wherein the quantity information is the number of information crawlers required to run concurrently, the matrix information is the information used to configure the longitude and latitude information in the preset crawler program template, the owner information is the information used to configure the Key file in the preset crawler program template, each owner corresponds to a Key value in a Key file, the data type is the type information used to configure the type of data information to be collected, and the interface information is the information used to configure the interface required for data interaction of the preset crawler program template.
[0077] The parameters configured in the information crawling program include longitude and latitude information, Key file, acquisition type, and acquisition interface, wherein the longitude and latitude information is the matrix range information used to collect data information in the corresponding area, and the longitude and latitude information contains one or more sets of longitude and latitude coordinates consisting of two coordinate points, and the line connecting the two coordinate points contained in a set of longitude and latitude coordinates is neither on the longitude nor on the latitude, and a rectangular area can be determined by the two coordinate points in a set of longitude and latitude coordinates, which is the area used to collect the corresponding data information; a Key file contains a Key value, and the Key value is the identification code corresponding to the acquirer of the corresponding data information; the acquisition type is the type information used to acquire the corresponding type of data information, for example, the acquisition type can be configured as hotel, food, scenic spot, etc.; the acquisition interface is the network address information of the interface for acquiring the corresponding data information, and each acquisition interface corresponds to a server end, and the acquisition interface can be used to interact with the corresponding server end.
[0078] In other inventive embodiments, Figure 7 As shown, the program configuration unit 110 includes sub-units: a program template copying unit 111 , a latitude and longitude information configuration unit 112 , a key file configuration unit 113 , an acquisition type configuration unit 114 and an interface configuration unit 115 .
[0079] The program template copying unit 111 is used to copy the preset crawling program template according to the quantity information in the configuration information to obtain multiple crawling program templates.
[0080] Obtain the quantity information in the configuration information, and copy the preset crawler program template according to the quantity information to obtain a corresponding number of crawler program templates, that is, the number of copied crawler program templates is the same as the data information, and the parameter values of each parameter in the preset crawler program template are all empty, that is, the parameter values in the multiple copied crawler program templates are all empty.
[0081] The longitude and latitude information configuration unit 112 is used to configure the longitude and latitude information in multiple crawler program templates according to the matrix information in the configuration information.
[0082] The longitude and latitude information in multiple crawler templates are configured respectively according to the matrix information in the configuration information. The matrix information contains multiple matrices, each of which can contain multiple groups of longitude and latitude coordinates, each group of longitude and latitude coordinates contains two coordinate points, and one matrix in the matrix information can be used to configure the longitude and latitude information in one crawler template, and the longitude and latitude information in all crawler templates can be configured according to the matrix information.
[0083] The key file configuration unit 113 is used to configure the key files in multiple crawler program templates respectively according to the owner information in the configuration information.
[0084] The Key files in the multiple crawler program templates are configured respectively according to the owner information in the configuration information. The owner information includes multiple acquisition person identification codes (IDs), and one identification code can be used to configure the Key file in one crawler program template. Then, the Key files in all crawler program templates can be configured according to the owner information.
[0085] The acquisition type configuration unit 114 is used to configure the acquisition types in multiple crawler program templates according to the data types in the configuration information.
[0086] The acquisition types in the plurality of crawler program templates are configured respectively according to the data type in the configuration information. The acquisition types in all crawler program templates only contain one parameter value, and the acquisition types in all crawler program templates can be configured according to the data type.
[0087] The interface configuration unit 115 is used to configure the acquisition interfaces in the multiple crawler program templates according to the interface information in the configuration information to obtain multiple information crawler programs.
[0088] According to the interface information in the configuration information, the acquisition interfaces in the multiple crawler program templates are configured respectively to obtain multiple information crawler programs. The acquisition interfaces in all crawler program templates only contain one parameter value. The acquisition interfaces in all crawler program templates can be configured through the interface information, and multiple information crawler programs with completed configuration can be obtained.
[0089] The prompt information sending unit 120 is used to generate prompt information according to the quantity information in the configuration information to remind the user that the program configuration has been completed.
[0090] A prompt message indicating that the configuration has been completed is generated according to the quantity information in the configuration information and sent to the user, so as to remind the user that the program configuration has been completed. After configuring multiple information crawling programs, corresponding prompt messages can be generated according to the quantity information in the configuration information and sent to the user, and the user can perform subsequent operations after receiving the prompt message.
[0091] The data information acquisition unit 130 is used to execute multiple information crawling programs in parallel to acquire and store corresponding multiple copies of data information if execution information fed back by the user according to the prompt information is received.
[0092] If the execution information fed back by the user according to the prompt information is received, multiple information crawling programs are executed in parallel to obtain and store corresponding multiple copies of data information. Specifically, after the user receives the prompt information of the completed configuration, the execution information can be sent to execute the configured multiple information crawling programs in parallel, and each information crawling program obtains a corresponding piece of data information, so multiple copies of data information can be obtained and saved.
[0093] In other inventive embodiments, Figure 8 As shown, the data information acquisition unit 130 includes subunits: an information sending unit 131 and a data information storage unit 132 .
[0094] The information sending unit 131 is used to send the acquisition type, key file, and longitude and latitude information configured by the information crawling program to the server end of the corresponding acquisition interface according to the acquisition interface of the information crawling program.
[0095] According to the acquisition interface in the information crawling program, the acquisition type, key file, and longitude and latitude information configured in the information crawling program are sent to the server side of the corresponding acquisition interface. Specifically, each acquisition interface corresponds to a server side. By sending the acquisition type, key file, and longitude and latitude information in an information crawling program to the server side corresponding to the acquisition interface of the information crawling program, a copy of the data information fed back by the server side can be obtained. After the server side verifies the Key value in the Key file, it searches for the acquisition type keyword in the rectangular area determined by each set of longitude and latitude coordinates in the longitude and latitude information, and obtains the corresponding data information and provides feedback.
[0096] The data information storage unit 132 is used to obtain and store multiple copies of data information fed back by the server end corresponding to the interface in multiple information crawling programs.
[0097] The data information fed back by the server corresponding to the acquisition interface in each information crawling program is obtained, that is, one information crawling program obtains a piece of data information, and the obtained multiple pieces of data information are stored.
[0098] The data information integration unit 140 is used to integrate the obtained multiple pieces of data information according to the configuration information to obtain target data information.
[0099] The obtained multiple pieces of data information are integrated according to the configuration information to obtain the target data information. Specifically, the configuration information also includes a preset threshold value. Among the obtained multiple pieces of data information, there is data information with a large number of data items. If the number of data items in a piece of data information does not exceed the preset threshold value in the configuration information, then the piece of data information meets the requirements. According to the judgment result, all the data information that meets the requirements is obtained, and the target data is obtained by classifying and sorting according to the data type in the configuration information.
[0100] In other inventive embodiments, Fig. 9 As shown, the data information integration unit 140 includes subunits: a judgment unit 141 and a classification and sorting unit 142 .
[0101] The judging unit 141 is used to judge whether the number of data items in each of the multiple pieces of data information is less than a preset threshold in the configuration information to obtain a judgment result.
[0102] It is judged whether the number of data items in each of the multiple data information is less than the preset threshold in the configuration information to obtain a judgment result. If the number of data items in a certain data information does not exceed the preset threshold in the configuration information, then the data information meets the requirements, and the number of data items in the obtained multiple data information is judged, and the judgment result is that the number of data items in the multiple data information is less than the preset threshold or the number of data items in the multiple data information is less than the preset threshold.
[0103] The classification and sorting unit 142 is used to classify and sort the data information according to the data types corresponding to the multiple data information in the configuration information to obtain the target data if the judgment result is that the number of data items in each of the multiple data information is less than a preset threshold.
[0104] If the result of the judgment is that the number of data items in each of the multiple pieces of data information is less than the preset threshold, the data information is classified and sorted according to the data types corresponding to the multiple pieces of data information in the configuration information to obtain the target data. Specifically, since the acquisition type in the information crawling program can be configured as multiple types when acquiring data information, the data information acquired of different data types can be classified and sorted according to the corresponding data types in the configuration information, and data information of the same data type is classified into one category.
[0105] In other inventive embodiments, Fig.10 As shown, the data information integration unit 140 further includes a subunit: an alarm prompt unit 141a.
[0106] The alarm prompt unit 141a is used to issue an alarm prompt message to remind the user that too much data information has been sent if the judgment result is that the number of data items in not all data information in the multiple data information is less than a preset threshold.
[0107] If the result of the judgment is that the number of data items in not all data items in the multiple data information is less than the preset threshold, an alarm prompt message is issued to remind the user that too much data information is sent. If the number of data items in a certain data information exceeds the preset threshold in the configuration information, an alarm prompt message needs to be sent to the user to prompt the user to further process the data information that exceeds the preset threshold.
[0108] The preset crawler program template is configured through the configuration information input by the user to obtain multiple information crawlers. The multiple information crawler programs are executed in parallel according to the execution information fed back by the user to obtain multiple copies of data information. The multiple copies of data information are integrated to obtain target data information. By executing multiple information crawler programs in parallel, massive data information can be acquired, which can greatly reduce the operating pressure of the equipment and greatly improve the efficiency of the enterprise in acquiring data information.
[0109] The above-mentioned data information acquisition device can be implemented in the form of a computer program. The computer program can be Fig.11 Runs on the computer device shown.
[0110] See also Fig.11 , Fig.11 It is a schematic block diagram of a computer device provided by an embodiment of the present invention.
[0111] See also Fig.11 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a non-volatile storage medium 503 and an internal memory 504 .
[0112] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the data information acquisition method.
[0113] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .
[0114] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the data information acquisition method.
[0115] The network interface 505 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0116] Among them, the processor 502 is used to run the computer program 5032 stored in the memory to achieve the following functions: if the configuration information input by the user is received, the preset crawler program template is configured according to the configuration information to obtain multiple information crawler programs that have been configured; prompt information is generated according to the quantity information in the configuration information to remind the user that the program configuration has been completed; if the execution information fed back by the user according to the prompt information is received, multiple information crawler programs are executed in parallel to obtain and store the corresponding multiple copies of data information; the multiple copies of data information obtained are integrated according to the configuration information to obtain the target data information.
[0117] In one embodiment, when the processor 502 executes the step of receiving configuration information input by a user and configuring a preset crawler program template according to the configuration information to obtain multiple information crawlers that have been configured, the processor 502 performs the following operations: copying the preset crawler program template according to the quantity information in the configuration information to obtain multiple crawler program templates; configuring the longitude and latitude information in the multiple crawler program templates according to the matrix information in the configuration information; configuring the key files in the multiple crawler program templates according to the owner information in the configuration information; configuring the acquisition type in the multiple crawler program templates according to the data type in the configuration information; configuring the acquisition interface in the multiple crawler program templates according to the interface information in the configuration information to obtain multiple information crawlers.
[0118] In one embodiment, when the processor 502 executes the steps of executing multiple information crawling programs in parallel to obtain and store corresponding multiple copies of data information if the execution information fed back by the user according to the prompt information is received, the processor 502 performs the following operations: according to the acquisition interface of the information crawling program, the acquisition type, key file, and longitude and latitude information configured by the information crawling program are sent to the server end of the corresponding acquisition interface; the multiple copies of data information fed back by the server end corresponding to the acquisition interface in the multiple information crawling programs are obtained and stored.
[0119] In one embodiment, when the processor 502 executes the step of integrating the multiple copies of data information obtained according to the configuration information to obtain the target data information, it performs the following operations: judging whether the number of data items in each of the multiple copies of data information is less than a preset threshold in the configuration information to obtain a judgment result; if the judgment result is that the number of data items in each of the multiple copies of data information is less than the preset threshold, classifying and sorting the data information according to the data types corresponding to the multiple copies of data information in the configuration information to obtain the target data.
[0120] In one embodiment, after executing the step of determining whether the number of data items in each of the multiple data messages is less than a preset threshold in the configuration information to obtain a determination result, the processor 502 further performs the following operation: if the determination result is that the number of data items in not each of the multiple data messages is less than the preset threshold, an alarm message is issued to remind the user that too much data information has been sent.
[0121] Those skilled in the art will understand that Fig.11 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Fig.11 The embodiments shown are consistent and will not be described again here.
[0122] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0123] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein the computer program implements the following steps when executed by a processor: if configuration information input by a user is received, a preset crawler program template is configured according to the configuration information to obtain multiple information crawlers that have been configured; prompt information is generated according to the quantity information in the configuration information to remind the user that the program configuration has been completed; if execution information fed back by the user according to the prompt information is received, multiple information crawlers are executed in parallel to obtain and store corresponding multiple copies of data information; and the obtained multiple copies of data information are integrated according to the configuration information to obtain target data information.
[0124] In one embodiment, if the configuration information input by the user is received, the step of configuring the preset crawler program template according to the configuration information to obtain multiple information crawler programs that have been configured includes: copying the preset crawler program template according to the quantity information in the configuration information to obtain multiple crawler program templates; configuring the longitude and latitude information in the multiple crawler program templates according to the matrix information in the configuration information; configuring the key files in the multiple crawler program templates according to the owner information in the configuration information; configuring the acquisition type in the multiple crawler program templates according to the data type in the configuration information; configuring the acquisition interface in the multiple crawler program templates according to the interface information in the configuration information to obtain multiple information crawler programs.
[0125] In one embodiment, if the execution information fed back by the user according to the prompt information is received, the steps of executing multiple information crawling programs in parallel to obtain corresponding multiple data information and storing them include: sending the acquisition type, key file, and longitude and latitude information configured by the information crawling program to the server end of the corresponding acquisition interface according to the acquisition interface of the information crawling program; obtaining multiple data information fed back by the server end corresponding to the acquisition interface in multiple information crawling programs and storing them.
[0126] In one embodiment, the step of integrating the multiple copies of data information obtained according to the configuration information to obtain the target data information includes: judging whether the number of data items in each of the multiple copies of data information is less than a preset threshold in the configuration information to obtain a judgment result; if the judgment result is that the number of data items in each of the multiple copies of data information is less than the preset threshold, classifying and arranging the data information according to the data types corresponding to the multiple copies of data information in the configuration information to obtain the target data.
[0127] In one embodiment, after the step of judging whether the number of data items in each of the multiple data information is less than the preset threshold in the configuration information to obtain a judgment result, it also includes: if the judgment result is that the number of data items in not each of the multiple data information is less than the preset threshold, an alarm prompt message is issued to remind the user that too much data information has been sent.
[0128] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.
[0129] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.
[0130] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.
[0131] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.
[0133] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A method for acquiring data information, characterized in that: include: If the configuration information input by the user is received, the preset crawling program template is configured according to the configuration information to obtain multiple information crawling programs that have been configured; Generate prompt information according to the quantity information in the configuration information to remind the user that the program configuration has been completed; If execution information fed back by the user according to the prompt information is received, multiple information crawling programs are executed in parallel to obtain corresponding multiple copies of data information and store them; Integrate the obtained multiple pieces of data information according to the configuration information to obtain target data information; The configuring the preset crawling program template according to the configuration information to obtain multiple information crawling programs that have been configured includes: Copying the preset crawling program template according to the quantity information in the configuration information to obtain multiple crawling program templates; According to the matrix information in the configuration information, respectively configure the longitude and latitude information in the plurality of crawler program templates; According to the owner information in the configuration information, the Key files in the plurality of crawler program templates are configured respectively; each owner corresponds to a Key value in the Key file; According to the data types in the configuration information, respectively configure the acquisition types in the plurality of crawler program templates; According to the interface information in the configuration information, the acquisition interfaces in the multiple crawler program templates are configured respectively to obtain multiple information crawler programs.
2. The data information acquisition method according to claim 1, characterized in that: The method of executing multiple information crawling programs in parallel to obtain and store corresponding multiple copies of data information includes: According to the acquisition interface of the information crawling program, the acquisition type, key file, and longitude and latitude information configured by the information crawling program are sent to the server end of the corresponding acquisition interface; Obtain multiple copies of data information fed back by the corresponding server side of the acquisition interface in multiple information crawling programs and store them.
3. The data information acquisition method according to claim 1, characterized in that: The step of integrating the plurality of pieces of data information obtained according to the configuration information to obtain target data information includes: Determine whether the number of data items in each of the multiple data information is less than a preset threshold in the configuration information to obtain a determination result; If the judgment result is that the number of data items in each of the multiple pieces of data information is less than the preset threshold, the data information is classified and sorted according to the data types corresponding to the multiple pieces of data information in the configuration information to obtain the target data.
4. The data information acquisition method according to claim 3, characterized in that: After determining whether the number of data items in each of the plurality of data information is less than the preset threshold in the configuration information to obtain a determination result, the method further includes: If the judgment result is that the number of data items in not every piece of data information among the multiple pieces of data information is less than the preset threshold, an alarm prompt message is issued to remind the user that too much data information has been sent.
5. A data information acquisition device, characterized in that: include: A program configuration unit, configured to configure a preset crawling program template according to the configuration information to obtain a plurality of information crawling programs that have been configured if configuration information input by a user is received; A prompt information sending unit, used to generate prompt information according to the quantity information in the configuration information to remind the user that the program configuration has been completed; A data information acquisition unit, configured to execute multiple information crawling programs in parallel to acquire and store corresponding multiple copies of data information upon receiving execution information fed back by the user according to the prompt information; A data information integration unit, used for integrating the obtained multiple pieces of data information according to the configuration information to obtain target data information; The program configuration unit includes: A program template copying unit, used for copying a preset crawling program template according to the quantity information in the configuration information to obtain a plurality of crawling program templates; A latitude and longitude information configuration unit, used to configure the latitude and longitude information in a plurality of crawler program templates respectively according to the matrix information in the configuration information; A key file configuration unit, used to configure the key files in the plurality of crawler program templates respectively according to the owner information in the configuration information; each owner corresponds to a key value in the key file; An acquisition type configuration unit, used to configure the acquisition types in a plurality of crawler program templates respectively according to the data types in the configuration information; An interface configuration unit is used to configure the acquisition interfaces in multiple crawler program templates according to the interface information in the configuration information to obtain multiple information crawler programs.
6. The data information acquisition device according to claim 5, characterized in that: The data information acquisition unit includes: An information sending unit, used to send the acquisition type, key file, and longitude and latitude information configured by the information crawling program to the server end of the corresponding acquisition interface according to the acquisition interface of the information crawling program; The data information storage unit is used to obtain and store multiple copies of data information fed back by the server side corresponding to the interface in multiple information crawling programs.
7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data information acquisition method according to any one of claims 1 to 4 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the data information acquisition method according to any one of claims 1 to 4.
Citation Information
Patent Citations
Unified management distributed web crawler system
CN108875091A
Process for fast file system crawling to support incremental file system differencing
US8078653B1
