A data extraction method and related equipment

By configuring data crawling and extraction rules for the website, the problem of low efficiency of manual data extraction is solved, efficient extraction and structured storage of target data is achieved, and data utilization efficiency is improved.

CN111475700BInactive Publication Date: 2024-11-26SHENZHEN SAIANTE TECH SERVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010180490.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-03-19
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the prior art, the manual extraction of target data is low efficiency, and it is difficult to efficiently extract and structure the required data from massive data.

Method used

By preconfiguring data crawling rules for each website, crawling target data from the website, and verifying according to the preset verification method, the data is converted into HTML format, and extracting and writing from HTML data based on the preset extraction rules Preset data table.

Benefits of technology

The efficient extraction and structured storage of target data are achieved, the data extraction efficiency is improved, and manual intervention is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111475700B_ABST
    Figure CN111475700B_ABST
Patent Text Reader

Abstract

The embodiment of the present application discloses a data extraction method and related equipment, which is applied to the field of data acquisition technology, including: crawling target data from each website based on the data crawling rules pre-configured for each website, verifying the target data according to a preset verification method, and if the verification passes, converting the crawled target data into a hypertext markup language HTML format to obtain target data in HTML format. Further, extracting field data corresponding to each preset field in a preset data table from the target data in HTML format based on the preset extraction rules, and writing each field data into the corresponding preset field column in the preset data table. By implementing the embodiment of the present application, the extraction of target data and the structured storage of target data can be efficiently realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and in particular to a data extraction method and related equipment. Background Art

[0002] At present, with the rapid development of Internet technology, various websites revolve around people's lives, and each website publishes a large amount of data (such as bid winning data, social data, insurance data, etc.). In order to effectively use these massive amounts of data, it is first necessary to collect and structure the target data in accordance with one's own business needs. At present, the main method of extracting relevant indicators is manual, which is time-consuming and labor-intensive, and the extraction efficiency is low. Therefore, how to extract target data more efficiently has become a problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide a data extraction method and related devices, which can efficiently extract target data and store the target data in a structured manner.

[0004] In a first aspect, an embodiment of the present application provides a data extraction method, which is applied to a server and includes:

[0005] crawling target data from each website based on data crawling rules preconfigured for each website;

[0006] Verifying the target data according to a preset verification method;

[0007] If the verification is passed, the crawled target data is converted into a hypertext markup language HTML format to obtain the target data in HTML format;

[0008] Extracting field data corresponding to each preset field in a preset data table from the target data in the HTML format based on preset extraction rules;

[0009] In the preset data table, each field data is written into the corresponding preset field column.

[0010] In one embodiment, the preset check mode includes odd check or even check, and the specific implementation method of checking the target data according to the preset check mode is: locating a check bit that matches the preset check mode in the target data; when the preset check mode is the odd check, detecting whether the check bit is an odd number; if it is an odd number, determining that the target data has been checked; or, when the preset check mode is the even check, detecting whether the check bit is an even number; if it is an even number, determining that the target data has been checked.

[0011] In one embodiment, the preset verification method includes information digest algorithm verification, and the specific implementation method of verifying the target data according to the preset verification method is: performing hash value calculation on the target data based on the information digest algorithm to obtain a target hash value corresponding to the target data; comparing the target hash value with the hash value of the target data transmission; if the comparison shows that the target hash value is the same as the hash value of the target data transmission, it is determined that the target data verification has passed.

[0012] In one embodiment, each preset field includes a first field and / or a second field, wherein field data written into the column to which the first field belongs is not associated with context information, and field data written into the column to which the second field belongs is associated with context information.

[0013] In one embodiment, the various preset fields include the first field, and the specific implementation method of extracting the field data corresponding to each preset field in the preset data table from the target data in the HTML format based on the preset extraction rules is: obtaining a field data crawling template pre-configured for the first field; extracting the field data corresponding to the first field in the preset data table from the target data in the HTML format based on the field data crawling template.

[0014] In one embodiment, the various preset fields include the second field, and the specific implementation method of extracting the field data corresponding to each preset field in the preset data table from the target data in the HTML format based on the preset extraction rule is:

[0015] Calling an HTML parser to extract text information from the target data in HTML format, and restoring the text information to a web page format, so that line breaks exist between paragraphs in the text information restored to the web page format;

[0016] dividing the text information restored to the webpage format into multiple segments of text information based on the line break;

[0017] Locate the second field in the multiple text information segments and locate the corresponding target text information segment;

[0018] The field data corresponding to the second field in the preset data table is extracted from the target segment text information.

[0019] In one embodiment, the second field includes a first subfield, and a specific implementation method of extracting field data corresponding to the second field in a preset data table from the target segment text information is as follows:

[0020] The field data corresponding to the first subfield in the preset data table is extracted from the target segment text information by using a regular expression.

[0021] In one embodiment, the second field includes a second subfield, and a specific implementation method of extracting field data corresponding to the second field in a preset data table from the target segment text information is as follows:

[0022] The field data corresponding to the second subfield in the preset data table is extracted from the target segment text information through a preset recognition model.

[0023] In one embodiment, after extracting the field data corresponding to each preset field in the preset data table from the target data in the HTML format based on the preset extraction rules, the field data can also be divided into at least one data set based on the business attributes of each field data; generate a query data table corresponding to each data set, and write the field data in each data set into the corresponding query data table; and create a corresponding data access interface for each query data table.

[0024] In a second aspect, an embodiment of the present application provides a data extraction device, which includes a module for executing the method of the first aspect.

[0025] In a third aspect, an embodiment of the present application provides a server, comprising a processor, a network interface and a memory, wherein the processor, network interface and memory are interconnected, wherein the network interface is controlled by the processor for sending and receiving messages, and the memory is used to store a computer program that supports the server to execute the above method, wherein the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method of the above first aspect.

[0026] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method of the first aspect above.

[0027] In the embodiment of the present application, the server crawls the target data from each website based on the data crawling rules pre-configured for each website, verifies the target data according to the preset verification method, and if the verification passes, converts the crawled target data into the hypertext markup language HTML format to obtain the target data in HTML format. Further, the server extracts the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules, and writes each field data into the corresponding preset field column in the preset data table. By adopting the embodiment of the present application, the extraction of target data and the structured storage of target data can be efficiently realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 It is a flowchart of a data extraction method provided in an embodiment of the present application;

[0030] Figure 2 It is a flowchart of another data extraction method provided in an embodiment of the present application;

[0031] Figure 3 This is a page schematic diagram provided by an embodiment of the present application;

[0032] Figure 4 is a schematic block diagram of a data extraction device provided in an embodiment of the present application;

[0033] Figure 5 It is a schematic block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] See also Figure 1 , Figure 1 : is a flow chart of a data extraction method provided in an embodiment of the present application. The method is applied to a server and can be executed by the server. As shown in the figure, the data extraction method may include:

[0036] S101: crawling target data from each website based on the data crawling rules pre-configured for each website.

[0037] The target data may refer to different data in combination with different business scenarios. Taking the procurement winning bid business scenario as an example, the target data may be the winning bid data, such as the winning bid document source URL, procurement location, issuing unit, issuing time, winning bid title, text content, and attachments, etc. Taking the insurance business field as an example, the target data may be the insurance document issuing time, issuing unit, insurance type, number of insured persons, etc., which is not specifically limited in the present application embodiment.

[0038] In one embodiment, based on the structure of different website data sources, different data crawling rules can be set in advance for different websites. After the settings of various data crawling rules are completed, when a crawling request for target data is detected, the target data can be crawled from each website based on the data crawling rules pre-configured for each website.

[0039] S102: Verify the target data according to a preset verification method. If the verification passes, convert the crawled target data into a hypertext markup language HTML format to obtain the target data in HTML format.

[0040] Among them, Html file is Hypertext Markup Language file. The Html file structure includes a header and a body. The header tag contains web page information, and the body tag contains the specific content to be displayed on the web page. HTML tags / labels can have different attribute items to control the content in the tag to display different effects. Html file is a text file, which requires parsing by other programs (such as browsers). Converting the crawled target data into HTML format can make blank lines between paragraphs in the target data (more consistent with the format between paragraphs when the page displays the target data), which is convenient for extracting the field data corresponding to each preset field in the preset data table in step S103.

[0041] Among them, the above-mentioned preset verification methods may include odd verification, even verification, message digest algorithm (Message-Digest Algorithm, MD5) verification, cyclic redundancy code check, LRC check, Gray code check, etc., which are used to verify the integrity and authenticity of the target data to prevent the target data from being tampered with or lost during transmission, and subsequent processing of erroneous data, which affects the accuracy of data extraction.

[0042] In one embodiment, the preset check mode includes odd check or even check, and the specific method of the terminal device checking the target data according to the preset check mode can be: locating the check bit matching the preset check mode in the target data, and when the preset check mode is odd check, detecting whether the check bit is an odd number, and if it is an odd number, determining that the target data has been checked. Alternatively, when the preset check mode is even check, detecting whether the check bit is an even number, and if it is an even number, determining that the target data has been checked.

[0043] The odd check or even check is performed based on whether the number of "1" in a set of binary codes transmitted is odd or even. Specifically, a parity check bit (i.e., the parity check bit) can be pre-set to store whether the number of "1" in the code is odd or even. If odd check is used, the parity check bit is an odd number, indicating that the target data corresponds to the correct data. If even check is used, the parity check bit is an even number, indicating that the target data corresponds to the correct data.

[0044] For example, assuming that the check bit is the last bit in the target data, and the target data is 10001100(1), if the preset check mode is odd check, then the terminal device can determine that the target data check is passed, indicating that the target data is correct and the data transmission is not wrong. If the preset check mode is even check, then the terminal device can determine that the target data check is not passed, indicating that the target data is wrong and the data transmission is wrong.

[0045] In one embodiment, the preset verification method includes information digest algorithm verification, and the specific implementation method of the terminal device verifying the target data according to the preset verification method can be: based on the information digest algorithm, the target data is hashed to obtain the target hash value corresponding to the target data, and the target hash value is compared with the hash value of the target data transmission. If the target hash value obtained by comparison is the same as the hash value of the target data transmission, it is determined that the target data verification has passed. That is, it can be indicated that there is no error in the target data during the transmission process.

[0046] S103: extracting field data corresponding to each preset field in the preset data table from the target data in HTML format based on preset extraction rules.

[0047] Taking the procurement bidding business scenario as an example, the preset fields can be, for example, the primary key, creator, creation date, updater, update date, uniform resource locator (URL), issuing unit, bidding title, project number, bidding unit, bidding unit address, bidding amount, project contact person, project contact person phone number, purchaser, purchaser phone number, purchaser address, agency, agency phone number, attachments and text content.

[0048] In one embodiment, taking the procurement winning business scenario as an example, each preset field may include a first field and / or a second field, and the field data written into the column to which the first field belongs is not associated with the context information, and the field data written into the column to which the second field belongs is associated with the upper and lower information. Among them, the association with the context information can be understood as the field data being interfered with by its own corresponding context information, for example, the second field can be the purchaser, the purchaser's phone number, the name of the procurement project (the winning product), the procurement amount, the agency, the agency's phone number, and the winning unit, etc., which are interfered with by the context; the non-association with the context information can be understood as the field data being not interfered with by its own corresponding context information, for example, the first field can be the URL, publishing unit, winning bid title, and attachments in the target data.

[0049] In one embodiment, the above-mentioned preset fields include a first field. The specific implementation method for the server to extract the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules can be: obtaining a field data crawling template pre-configured for the first field, and extracting the field data corresponding to the first field in the preset data table from the target data in HTML format based on the field data crawling template.

[0050] In another embodiment, each of the preset fields includes a second field, and the specific implementation method of the server extracting the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rule can be: calling the HTML parser to extract text information from the target data in HTML format, and restoring the text information to the web page format, so that there are line breaks between paragraphs in the text information restored to the web page format. Further, based on the line breaks, the text information restored to the web page format is divided into multiple text information segments, and the target segment text information corresponding to the second field is located in the multiple text information segments, and then the field data corresponding to the second field in the preset data table is extracted from the target segment text information.

[0051] Exemplarily, the HTML parser may be BeautifulSoup, which is an HTML / XML parser written in Python. In this case, the server may read the target data in HTML format in sequence, and perform data processing on the target data in HTML format through BeautifulSoup to extract text information corresponding to the target data, and restore the text information to a web page format, so that line breaks '\n' exist between paragraphs in the text information restored to a web page format.

[0052] In one embodiment, the terminal device may perform data processing on the target segment text information through the trained recognition model to extract field data corresponding to the second field in the preset data table from the target segment text information.

[0053] In one embodiment, the second field may include the first subfield and / or the second subfield. Taking the procurement winning business scenario as an example, the second field may include, for example, the purchaser, the purchaser's phone number, the name of the procurement project (winning product), the procurement amount, the agency, the agency's phone number, and the winning bidder, etc., which are interfered with by the context. The procurement amount, the purchaser's phone number, the agency's phone number, the procurement project name, and the winning bidder's address may be the first subfield; the purchaser, the agency, and the winning bidder may be the second subfield.

[0054] In one embodiment, the second field includes the first subfield, and the server can extract the field data corresponding to the first subfield in the preset data table from the target segment text information through a regular expression. The regular expression uses some pre-defined specific characters and the combination of these specific characters to form a regular string, and the regular string is used to express a filtering logic for the string.

[0055] In one embodiment, the second field includes a second subfield, and the server can extract field data corresponding to the second subfield in a preset data table from the target segment text information through a trained recognition model.

[0056] The recognition model is obtained by training the initial recognition model using a large amount of text information and part-of-speech tags including field data under the first subfield. The initial recognition model may be, for example, a conditional random field algorithm (CRF) model.

[0057] The calculation formula corresponding to the CRF algorithm is shown in Formula 1-1:

[0058]

[0059] Where P is the conditional probability, Z(x) is the normalization factor, which is the sum of all possible output sequences, and t k and l is the characteristic function, λ k and μ l Yes k and lThe corresponding weights are, x represents the input observation sequence, y represents the output label sequence corresponding to the input observation sequence, in this scheme, the input observation sequence is the input target data, and the output label sequence corresponding to the input observation sequence is the field data corresponding to the second field in the output target data (for example, the purchaser, agency or successful bidder, etc.).

[0060] Exemplarily, the first subfield is the organization name, and the initial recognition model can be trained using tens of thousands of organization names and the dependency relationship between the organization names and the context as training corpus, so that the trained preset recognition model can quickly recognize the organization name from the new corpus.

[0061] S104: Writing each field data into the corresponding preset field column in the preset data table.

[0062] For example, after extracting each field data from the target data, the extracted field data can be filled into each preset field corresponding to the preset data table in the database (e.g., URL, issuing unit, winning bid title, project number, winning bidder, winning bidder address, winning bid amount, project contact person, project contact phone number, purchaser, purchaser phone number, purchaser address, agency, agency phone number, attachments, text content, etc.).

[0063] In the embodiment of the present application, the server crawls the target data from each website based on the data crawling rules pre-configured for each website, verifies the target data according to the preset verification method, and if the verification passes, converts the crawled target data into the hypertext markup language HTML format to obtain the target data in HTML format. Further, the server extracts the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules, and writes each field data into the corresponding preset field column in the preset data table. By adopting the embodiment of the present application, the extraction of target data and the structured storage of target data can be efficiently realized.

[0064] See also Figure 2 , Figure 2 is a schematic diagram of another data extraction method provided in an embodiment of the present application. The method is applied to a server and can be executed by the server. As shown in the figure, the data extraction method may include:

[0065] S201: crawling target data from each website based on the data crawling rules pre-configured for each website.

[0066] S202: Verify the target data according to a preset verification method. If the verification passes, convert the crawled target data into a hypertext markup language HTML format to obtain the target data in HTML format.

[0067] S203: extracting field data corresponding to each preset field in the preset data table from the target data in HTML format based on preset extraction rules.

[0068] S204: writing each field data into the corresponding preset field column in the preset data table. The specific implementation of steps S201 to S204 can refer to the relevant description of steps S101 to S104 in the above embodiment, which will not be repeated here.

[0069] S205: Divide each field data into at least one data set based on the business attribute of each field data.

[0070] In one embodiment, the preset data table includes field data of multiple business attributes. In order to facilitate users to query field data under a certain business attribute, the business attributes of each field data can be used to divide the field data into at least one data set. Exemplarily, taking the procurement bid winning business scenario as an example, the field data in the preset data table includes the bid winning title, project number, bid winning unit, bid winning unit address, bid winning amount, project contact person, project contact phone number, purchaser, purchaser phone number, purchaser address, agency, agency phone number, attachments and text content. In this case, based on the business attributes, the purchaser phone number, purchaser address, etc. can be divided into the purchaser information set; the agency, agency phone number, etc. can be divided into the agent information set; the bid winning title, project number, bid winning unit, bid winning unit address, bid winning amount, etc. can be divided into the bid winning information set. Among them, the purchaser information set, the agency information set and the bid winning information set can all be understood as the corresponding data sets after the division of each field data.

[0071] S206: Generate a query data table corresponding to each data set, and write the field data in each data set into the corresponding query data table.

[0072] S207: Creating corresponding data access interfaces for each query data table.

[0073] Exemplarily, assuming that the data set includes a purchaser information set, an agent information set and a winning bid information set, a query data table can be created for each of the purchaser information set, the agent information set and the winning bid information set. The purchaser information (e.g., the purchaser, the purchaser's phone number, the purchaser's address, etc.) is written into the first query data table corresponding to the purchaser information set; the agent information (e.g., the agency, the agency's phone number, etc.) is written into the second query data table corresponding to the agent information set; and the winning bid information (e.g., winning bid title, project number, winning bidder, winning bidder address, winning bid amount, etc.) is written into the third query data table corresponding to the winning bid information set.

[0074] Furthermore, corresponding data access interfaces can be created for the first query data table, the second query data table, and the third query data table, and function buttons corresponding to the data access interfaces can be displayed on the client side corresponding to the server, such as Figure 3 As shown, the function button "Purchaser Information Query" is used to query the first query data table, the function button "Agent Information Query" is used to query the second query data table, and the function button "Winning Bidder Information Query" is used to query the third query data table.

[0075] In an embodiment of the present application, the server crawls target data from each website based on the data crawling rules pre-configured for each website, verifies the target data according to a preset verification method, and if the verification passes, converts the crawled target data into a hypertext markup language HTML format to obtain the target data in HTML format. Further, the server extracts the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules, and writes each field data into the corresponding column of the preset field in the preset data table. Further, the server can divide each field data into at least one data set based on the business attributes of each field data, generate a query data table corresponding to each data set, write the field data in each data set into the corresponding query data table, and create a corresponding data access interface for each query data table, so that users can quickly query each query data table according to each data access interface, which is conducive to improving the efficiency of querying data.

[0076] The present application also provides a data extraction device. The device includes a device for performing the above Figure 1 or Figure 2 The module of the method is configured on the server. Figure 4 , is a schematic block diagram of a data extraction device provided in an embodiment of the present application. The data extraction device of this embodiment includes:

[0077] A data crawling module 40, configured to crawl target data from each website based on data crawling rules preconfigured for each website;

[0078] The processing module 41 is used to verify the target data according to a preset verification method. If the verification passes, the crawled target data is converted into a hypertext markup language HTML format to obtain the target data in HTML format;

[0079] The processing module 41 is further used to extract field data corresponding to each preset field in the preset data table from the target data in the HTML format based on a preset extraction rule;

[0080] The reading and writing module 42 is used to write each of the field data into the corresponding column of the preset field in the preset data table.

[0081] In one embodiment, the preset check mode includes odd check or even check, and the processing module 41 is specifically used to locate the check bit that matches the preset check mode in the target data; when the preset check mode is the odd check, detect whether the check bit is an odd number; if it is an odd number, determine that the check of the target data is passed; or, when the preset check mode is the even check, detect whether the check bit is an even number; if it is an even number, determine that the check of the target data is passed.

[0082] In one embodiment, the preset verification method includes information digest algorithm verification, and the processing module 41 is further specifically used to calculate a hash value of the target data based on the information digest algorithm to obtain a target hash value corresponding to the target data; compare the target hash value with the hash value of the target data transmission; if the comparison shows that the target hash value is the same as the hash value of the target data transmission, it is determined that the verification of the target data has passed.

[0083] In one embodiment, each preset field includes a first field and / or a second field, wherein field data written into the column to which the first field belongs is not associated with context information, and field data written into the column to which the second field belongs is associated with context information.

[0084] In one embodiment, the various preset fields include the first field, and the processing module 41 is specifically used to obtain a field data crawling template pre-configured for the first field; based on the field data crawling template, extract the field data corresponding to the first field in the preset data table from the target data in the HTML format.

[0085] In one embodiment, each preset field includes the second field, and the processing module 41 is specifically used to call the HTML parser to extract text information from the target data in the HTML format, and restore the text information to the web page format, so that there are line breaks between paragraphs in the text information restored to the web page format; based on the line breaks, the text information restored to the web page format is divided into multiple segments of text information; the target segment text information corresponding to the second field is located in the multiple segments of text information; and the field data corresponding to the second field in the preset data table is extracted from the target segment text information.

[0086] In one embodiment, the processing module 41 is further specifically used to perform data processing on the target segment text information through the trained recognition model, so as to extract field data corresponding to the second field in the preset data table from the target segment text information.

[0087] In one embodiment, the processing module 41 is also used to divide the various field data into at least one data set based on the business attributes of each field data; generate a query data table corresponding to each data set, and write the field data in each data set into the corresponding query data table; and create a corresponding data access interface for each query data table.

[0088] It should be noted that the functions of the various functional modules of the data extraction device described in the embodiment of the present application can be based on Figure 1 or Figure 2 The specific implementation process of the method in the method embodiment can be referred to Figure 1 or Figure 2 The relevant description of the method embodiment will not be repeated here.

[0089] In the embodiment of the present application, the data crawling module 40 crawls the target data from each website based on the data crawling rules pre-configured for each website, and the processing module 41 verifies the target data according to the preset verification method. If the verification passes, the crawled target data is converted into the hypertext markup language HTML format to obtain the target data in HTML format. Further, the processing module 41 extracts the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules, and the reading and writing module 42 writes each field data into the corresponding preset field column in the preset data table. By adopting the embodiment of the present application, the extraction of target data and the structured storage of target data can be efficiently realized.

[0090] See also Figure 5 , Figure 5 is a schematic block diagram of a server provided in an embodiment of the present application, such as Figure 5 As shown, the server includes a processor 501, a memory 502 and a network interface 503. The processor 501, the memory 502 and the network interface 503 can be connected via a bus or other means. Figure 5Take the connection through the bus as an example. The network interface 503 is controlled by the processor to send and receive messages, the memory 502 is used to store computer programs, the computer programs include program instructions, and the processor 501 is used to execute the program instructions stored in the memory 502. The processor 501 is configured to call the program instructions to execute: crawling target data from each website through the network interface 503 based on the data crawling rules pre-configured for each website; verifying the target data according to a preset verification method; if the verification passes, converting the crawled target data into a hypertext markup language HTML format to obtain target data in HTML format; extracting the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules; writing each of the field data into the corresponding preset field column in the preset data table.

[0091] In one embodiment, the preset check mode includes odd check or even check, and the processor 501 is specifically used to locate the check bit that matches the preset check mode in the target data; when the preset check mode is the odd check, detect whether the check bit is an odd number; if it is an odd number, determine that the check of the target data is passed; or, when the preset check mode is the even check, detect whether the check bit is an even number; if it is an even number, determine that the check of the target data is passed.

[0092] In one embodiment, the preset verification method includes information digest algorithm verification, and the processor 501 is further specifically used to calculate a hash value for the target data based on the information digest algorithm to obtain a target hash value corresponding to the target data; compare the target hash value with the hash value of the target data transmission; if the comparison shows that the target hash value is the same as the hash value of the target data transmission, it is determined that the verification of the target data has passed.

[0093] In one embodiment, each preset field includes a first field and / or a second field, wherein field data written into the column to which the first field belongs is not associated with context information, and field data written into the column to which the second field belongs is associated with context information.

[0094] In one embodiment, the various preset fields include the first field, and the processor 501 is specifically used to obtain a field data crawling template pre-configured for the first field; based on the field data crawling template, extract the field data corresponding to the first field in the preset data table from the target data in the HTML format.

[0095] In one embodiment, each preset field includes the second field, and the processor 501 is specifically used to call the HTML parser to extract text information from the target data in the HTML format, and restore the text information to the web page format, so that there are line breaks between paragraphs in the text information restored to the web page format; divide the text information restored to the web page format into multiple segments of text information based on the line breaks; locate the target segment text information corresponding to the second field in the multiple segments of text information; and extract the field data corresponding to the second field in the preset data table from the target segment text information.

[0096] In one embodiment, the processor 501 is further specifically configured to perform data processing on the target segment text information through a trained recognition model, so as to extract field data corresponding to the second field in a preset data table from the target segment text information.

[0097] In one embodiment, the processor 501 is also used to divide the various field data into at least one data set based on the business attributes of each field data; generate a query data table corresponding to each data set, and write the field data in each data set into the corresponding query data table; and create a corresponding data access interface for each query data table.

[0098] It should be understood that in the embodiment of the present application, the processor 501 may be a central processing unit (CPU), and the processor 501 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0099] The memory 502 may include a read-only memory and a random access memory, and provide instructions and data to the processor 501. A portion of the memory 502 may also include a non-volatile random access memory. For example, the memory 502 may also store information about the device type.

[0100] In a specific implementation, the processor 501, the memory 502 and the network interface 503 described in the embodiment of the present application can execute the Figure 1 or Figure 2The implementation method described in the method embodiment can also execute the implementation method of the data extraction device described in the embodiment of the present application, which will not be repeated here.

[0101] In the embodiment of the present application, the target data is crawled from each website based on the data crawling rules pre-configured for each website through the network interface 503, and the processor 501 verifies the target data according to the preset verification method. If the verification passes, the crawled target data is converted into the hypertext markup language HTML format to obtain the target data in HTML format. Further, the processor 501 extracts the field data corresponding to each preset field in the preset data table from the target data in HTML format based on the preset extraction rules, and writes each field data into the corresponding preset field column in the preset data table. By adopting the embodiment of the present application, the extraction of target data and the structured storage of target data can be efficiently realized.

[0102] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the following are implemented: crawling target data from each website based on data crawling rules preconfigured for each website; converting the crawled target data into a hypertext markup language HTML format to obtain target data in HTML format; extracting field data corresponding to each preset field in a preset data table from the target data in HTML format based on preset extraction rules; and writing each of the field data into a column corresponding to the preset field in the preset data table.

[0103] The computer-readable storage medium may be an internal storage unit of the server described in any of the foregoing embodiments, such as a hard disk or memory of the server. The computer-readable storage medium may also be an external storage device of the server, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the server. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the server and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the server. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0104] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0105] The above disclosure is only part of the embodiments of the present invention, which certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention are still within the scope of the invention.

Claims

1. A data extraction method, characterized in that: The method comprises: crawling target data from each website based on data crawling rules preconfigured for each website; Verifying the target data according to a preset verification method; If the verification is passed, the crawled target data is converted into a hypertext markup language HTML format to obtain the target data in HTML format; Extracting field data corresponding to each preset field in a preset data table from the target data in the HTML format based on preset extraction rules; Writing each of the field data into the corresponding preset field column in the preset data table; Wherein, each preset field includes a first field and a second field, the field data of the column to which the first field belongs is not associated with context information, and the field data of the column to which the second field belongs is associated with context information; extracting the field data corresponding to each preset field in the preset data table from the target data in the HTML format based on the preset extraction rule includes: Acquire a field data crawling template pre-configured for the first field; extract field data corresponding to the first field in a preset data table from the target data in the HTML format based on the field data crawling template; The HTML parser is called to extract text information from the target data in HTML format, and the text information is restored to a web page format so that line breaks exist between paragraphs in the text information restored to a web page format; the text information restored to a web page format is divided into multiple segments of text information based on the line breaks; the target segment text information corresponding to the second field is located in the multiple segments of text information; and the target segment text information is processed by a recognition model obtained through training to extract field data corresponding to the second field in a preset data table from the target segment text information.

2. The method according to claim 1, characterized in that The preset check mode includes odd check or even check, and the checking the target data according to the preset check mode includes: Locating a check bit in the target data that matches the preset check method; When the preset check mode is the odd check, detecting whether the check bit is an odd number; If it is an odd number, it is determined that the target data has passed the verification; or, When the preset check mode is the even check, detecting whether the check bit is an even number, If it is an even number, it is determined that the target data has been verified.

3. The method according to claim 2, characterized in that The preset verification method includes information digest algorithm verification, and the verification of the target data according to the preset verification method includes: Calculating a hash value of the target data based on an information digest algorithm to obtain a target hash value corresponding to the target data; comparing the target hash value with the hash value of the target data transmission; If the target hash value obtained by comparison is the same as the hash value of the target data transmission, it is determined that the target data has passed the verification.

4. The method according to claim 1, characterized in that After extracting the field data corresponding to each preset field in the preset data table from the target data in the HTML format based on the preset extraction rule, the method further includes: Based on the business attributes of each of the field data, each field data is divided into at least one data set; Generate a query data table corresponding to each data set, and write the field data in each data set into the corresponding query data table; A corresponding data access interface is created for each of the query data tables.

5. A data extraction device, characterized in that: The device comprises: A data crawling module, used to crawl target data from each website based on data crawling rules pre-configured for each website; A processing module is used to verify the target data according to a preset verification method, and if the verification passes, the crawled target data is converted into a hypertext markup language HTML format to obtain the target data in HTML format; The processing module is further used to extract field data corresponding to each preset field in the preset data table from the target data in the HTML format based on a preset extraction rule; A read-write module, used for writing each field data into the preset data table into the corresponding preset field column; Wherein, each preset field includes a first field and a second field, the field data of the column to which the first field belongs is not associated with context information, and the field data of the column to which the second field belongs is associated with context information; the processing module extracts the field data corresponding to each preset field in the preset data table from the target data in the HTML format based on a preset extraction rule, specifically for: Acquire a field data crawling template pre-configured for the first field; extract field data corresponding to the first field in a preset data table from the target data in the HTML format based on the field data crawling template; The HTML parser is called to extract text information from the target data in HTML format, and the text information is restored to a web page format so that line breaks exist between paragraphs in the text information restored to a web page format; the text information restored to a web page format is divided into multiple segments of text information based on the line breaks; the target segment text information corresponding to the second field is located in the multiple segments of text information; and the target segment text information is processed by a recognition model obtained through training to extract field data corresponding to the second field in a preset data table from the target segment text information.

6. A server, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are connected to each other, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 4.