Data collection method and device based on named placeholder language and storage medium
By using a named placeholder language to extract fields and format and replace data in the JSON or HTML data returned by the API interface, the problem of headless browsers being unable to collect data from server-side rendered websites is solved, achieving efficient data collection and formatting processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JIANGSU FUTURE NETWORKS INNOVATION
- Filing Date
- 2023-11-24
- Publication Date
- 2026-08-04
AI Technical Summary
Headless browsers cannot effectively collect data from target websites rendered by the server, resulting in low collection efficiency and high costs. Existing technologies cannot meet the needs of comprehensive data collection.
The system uses a named placeholder language to extract and format the fields of the JSON or HTML data returned by the API interface, and uses XPath or regular expressions to parse the fields to generate the final JSON format data.
It enables efficient data collection from server-side rendered websites, improving collection efficiency and usability, and is suitable for various information collection systems.
Smart Images
Figure CN117609657B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of network information acquisition technology, and in particular to a data acquisition method, device and storage medium based on a named placeholder language. Background Technology
[0002] Web scraping often involves simulating a headless browser, especially when the target website uses JavaScript to generate content or requires user interaction. A headless browser lacks a visible graphical user interface (GUI), but can execute JavaScript code to render web pages, thereby automating data collection.
[0003] Headless browsers are suitable for information collection in most situations. However, they still have certain limitations, especially in the following specific cases: the target website is server-side rendered (e.g., using front-end / back-end separation technologies like Vue), page data is returned via API interfaces, and headless browsers cannot render the webpage by executing JavaScript code. In such cases, the usual solutions are to develop a separate browser to simulate headed browsing for continued data collection, or to perform data collection manually. The former increases collection time (headed browsers are less efficient than headless browsers) and development costs, while the latter increases labor costs.
[0004] Reference 1: Chinese invention patent CN202010201225.9 discloses a web page rendering method and device. Although the placeholder string in it also has data replacement characteristics, its function is to modify the collected data to obtain page structure data, thereby realizing the rendering of the web page. However, it does not give the specific data collection process. Summary of the Invention
[0005] The purpose of this invention is to provide a data acquisition method, device, and storage medium based on named placeholder language. By using named placeholder language to extract and format and replace data from websites returned through API interfaces, page data can be obtained, enabling the network information acquisition system to achieve the goal of comprehensive data acquisition.
[0006] The technical solution to achieve the purpose of this invention is as follows:
[0007] A data acquisition method based on a named placeholder language, the data acquisition method comprising:
[0008] Initiate a query request through the API interface to retrieve JSON data;
[0009] Extract fields from JSON data using a pre-defined named placeholder language; format and replace the extracted fields; return the formatted and replaced results to the API interface; data collection is complete.
[0010] The default naming placeholder language is an array structure. Each element in the array structure includes the name of the field to be collected, the collection type, and the parameters corresponding to the collection type.
[0011] Furthermore, the field names are represented by dot symbols, which are used to specify the data level of the field to be collected in the JSON returned by the API interface, and to determine the position where the extracted field needs to be replaced during the formatting replacement process.
[0012] Furthermore, the JSON data is text in string type, and the field extraction process is as follows:
[0013] Iterate through the named placeholder language of the array structure and extract the field names that need to be collected.
[0014] Extract fields from JSON by the data level specified by the field names to be collected, and assemble the extracted fields into JSON format.
[0015] Furthermore, the collection type has two fixed values, "XPath" and "regular", which indicate which type of parsing is performed on the field specified by the field name to be collected; the parameter corresponding to the collection type indicates the specific parameter specified by the collection type, that is, an XPath or a regular expression.
[0016] Furthermore, the JSON data is HTML-type text, and the field extraction process is as follows:
[0017] Iterate through the named placeholder language of the array structure and extract the field names that need to be collected.
[0018] Extract the HTML text fields from the JSON by referring to the data level specified by the field names to be collected.
[0019] If the data collection type is XPath, then the HTML text fields are converted into DOM nodes, and the corresponding XPath is used to extract the fields.
[0020] If the data collection type is regular, use the corresponding regular expression to extract fields from the HTML text fields;
[0021] The extracted fields are assembled into JSON format.
[0022] Furthermore, the JSON format is as follows:
[0023] {"name1":["value11","value12","value13",...],"name2":["value21","value22","value23",...],...}
[0024] Where name i represents the name of the i-th field to be collected, and ["valuei1","valuei2","valuei3",...] represents an array of fields extracted from the JSON based on the data level specified by name i.
[0025] Furthermore, the process for formatting and replacing the extracted fields is as follows:
[0026] Based on the field names to be collected in the pre-defined named placeholder language, a formatted replacement template is determined;
[0027] The extracted field results corresponding to different field names are assigned to the preset formatted replacement template to complete the formatted replacement.
[0028] Furthermore, the template for formatted replacement is shown below:
[0029] https: / / {name1} / {name2} / ...
[0030] Where name i represents the name of the i-th field to be collected;
[0031] The results of the formatted replacement are shown below:
[0032] [https: / / value11 / value21,...https: / / value12 / value22,...https: / / value13 / value23,...]
[0033] Here, "valuei1", "valuei2", "valuei3", ... represent several fields extracted from the JSON based on the data level specified by name i.
[0034] An information acquisition device includes a memory for storing a computer program and a processor for implementing steps such as a data acquisition method based on a named placeholder language when executing the computer program.
[0035] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a data acquisition method based on a named placeholder language.
[0036] The significant advantages of this invention compared to existing technologies are:
[0037] (1) Implement the extraction, formatting and replacement of data from the website returned by the API interface using named placeholder language. This is applicable when the information to be extracted is returned in the form of JSON fields or HTML text.
[0038] (2) It effectively improves the availability and efficiency of network information collection, enabling the network information collection system to meet the needs of comprehensive data collection and to be used in various information collection systems, with broad application prospects. Attached Figure Description
[0039] Figure 1 This is a flowchart illustrating the data acquisition method based on named placeholder language in an embodiment of the present invention.
[0040] Figure 2 This is a schematic diagram illustrating an application scenario of the data acquisition method based on named placeholder language in an embodiment of the present invention. Detailed Implementation
[0041] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0042] like Figure 1 As shown, a data acquisition method based on a named placeholder language is described, which includes:
[0043] Initiate a query request through the API interface to retrieve JSON data;
[0044] Extract fields from JSON data using a pre-defined named placeholder language; format and replace the extracted fields; return the formatted and replaced results to the API interface; data collection is complete.
[0045] The default naming placeholder language is an array structure. Each element in the array structure includes the name of the field to be collected, the collection type, and the parameters corresponding to the collection type.
[0046] Specifically, field names are represented by dot symbols, which are used to specify the data level of the field to be collected in the JSON returned by the API interface, and to determine the position where the extracted field needs to be replaced during the formatting and replacement process.
[0047] Specifically, the JSON data is text in string type, and the field extraction process is as follows:
[0048] Iterate through the named placeholder language of the array structure and extract the field names that need to be collected.
[0049] Extract fields from JSON by the data level specified by the field names to be collected, and assemble the extracted fields into JSON format.
[0050] Specifically, the collection type has two fixed values: "XPath" and "regular," which indicate which type of parsing is performed on the field specified by the field name to be collected. The parameter corresponding to the collection type represents the specific parameter specified for the collection type, namely an XPath or a regular expression.
[0051] Specifically, the JSON data is HTML-type text, and the field extraction process is as follows:
[0052] Iterate through the named placeholder language of the array structure and extract the field names that need to be collected.
[0053] Extract the HTML text fields from the JSON by referring to the data level specified by the field names to be collected.
[0054] If the data collection type is XPath, then the HTML text fields are converted into DOM nodes, and the corresponding XPath is used to extract the fields.
[0055] If the data collection type is regular, use the corresponding regular expression to extract fields from the HTML text fields;
[0056] The extracted fields are assembled into JSON format.
[0057] Specifically, the JSON format is as follows:
[0058] {"name1":["value11","value12","value13",...],"name2":["value21","value22","value23",...],...}
[0059] Where name i represents the name of the i-th field to be collected, and ["valuei1","valuei2","valuei3",...] represents an array of fields extracted from the JSON based on the data level specified by name i.
[0060] Specifically, the process for formatting and replacing the extracted fields is as follows:
[0061] Based on the field names to be collected in the pre-defined named placeholder language, a formatted replacement template is determined;
[0062] The extracted field results corresponding to different field names are assigned to the preset formatted replacement template to complete the formatted replacement.
[0063] Specifically, the template for formatted replacement is shown below:
[0064] https: / / {name1} / {name2} / ...
[0065] Where name i represents the name of the i-th field to be collected;
[0066] The results of the formatted replacement are shown below:
[0067] [https: / / value11 / value21,...https: / / value12 / value22,...https: / / value13 / value23,...]
[0068] Here, "valuei1", "valuei2", "valuei3", ... represent several fields extracted from the JSON based on the data level specified by name i.
[0069] An information acquisition device includes a memory for storing a computer program and a processor for implementing steps such as a data acquisition method based on a named placeholder language when executing the computer program.
[0070] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a data acquisition method based on a named placeholder language.
[0071] like Figure 2 As shown below, the operation process of the data acquisition method based on the named placeholder language will be described in detail in conjunction with the actual application scenario of the present invention.
[0072] S1. Named Placeholder Language (NOL) Syntax and Explanation:
[0073] (1) Named Placeholder Language (NOL) is implemented as an array (List), where each element is a string with a specific syntax, as shown below (Note: ... indicates multiple elements):
[0074] ["{{name1[type1||selector1]}}","{{name2[type2||selector2]}}"...]
[0075] (2) Taking the first element of the array as an example, the specific parameters are explained as follows:
[0076] `name1`: Indicates the name of the field to be collected or formatted for replacement. In data extraction, this field typically uses dot notation to specify the data level of the field to be collected in the JSON returned by the API interface. In formatted replacement, this field indicates the location where the extracted data needs to be replaced.
[0077] type1: Indicates the data collection type. This field has two fixed values, "XPath" (XML Path Language) and "regular" (regular expression), which indicate the type of parsing performed on the field specified by name1.
[0078] selector1: Represents the specific parameters corresponding to the collection type specified by type1. Such as an XPath or a regular expression.
[0079] Except for name1, type1, and selector1, all other symbols have fixed syntax. name1 is required, and [type1||selector1] is optional. If [type1||selector1] is omitted, it can be written in the following format:
[0080] ["{{name1}}","{{name2}}"...]
[0081] The specific circumstances under which [type1||selector1] is required will be explained in detail in the data extraction section below.
[0082] S2, Data Extraction:
[0083] Data extraction falls into two categories: first, the fields in the JSON returned by the API interface are directly usable strings; second, the fields returned by the JSON are HTML-type text, requiring XPath or regular expression parsing to extract the data. The implementation methods for both scenarios are shown below:
[0084] (1) When the fields in the JSON can be used directly, the implementation method is as follows:
[0085] In this case, [type||selector] can be left blank. Iterate through the Named Placeholder Language (NOL) array and loop through the field names (name) named using dot notation.
[0086] The values are retrieved from the JSON by the hierarchy represented by the name. The retrieved values may be one or multiple (if there are multiple values, the specified name is in an array). Finally, they are assembled into the following JSON format:
[0087] {"name1":["value11","value12","value13",...],"name2":["value21","value22","value23",...],...}
[0088] (2) When the field returned in the JSON is HTML type text, the implementation is as follows:
[0089] In this case, [type||selector] is a required field. Iterate through the Named Placeholder Language (NOL) array and loop through the field names (name) named using dot notation.
[0090] The HTML text field is retrieved from the JSON by using the level indicated by the name.
[0091] If the type is XPath, the HTML string is converted into a DOM node, and the data is extracted according to the XPath corresponding to the selector;
[0092] If the type is regular (regular expression), the regular expression corresponding to the selector is used to perform regular expression matching to extract values from the HTML text;
[0093] Finally, they are all assembled into the following JSON format:
[0094] {"name1":["value11","value12","value13",...],"name2":["value21","value22","value23",...],...}
[0095] (3) The data format of the final assembly in the two cases above is consistent, so that it can be used for formatting replacement.
[0096] S3, Format Replacement:
[0097] Formatted replacement involves organizing and piecing together the extracted JSON data into the desired information to achieve the purpose of data collection. The following example demonstrates the functionality of formatted replacement by organizing the extracted JSON into the required domain name (Note: formatted replacement is not limited to extracting and piecing together domain names; it can be customized according to business needs).
[0098] (1) Determine a template for format replacement, as shown below:
[0099] https: / / {name1} / {name2}
[0100] Here, name1 and name2 correspond to the name1 and name2 configured in the Named Placeholder Language (NOL).
[0101] (2) Parse the extracted JSON data, assign the data from the arrays corresponding to different names to the formatted replacement template, and obtain the following data after replacement:
[0102] [https: / / value11 / value21,...https: / / value12 / value22,...https: / / value13 / value23,...]
[0103] The final array obtained is the list of domain names to be collected, which completes the data extraction and URL formatting replacement of the website returned by the API interface using Named Placeholder Language (NOL).
[0104] Specifically, let's assume the API interface returns the following format:
[0105] {code:200,data:[{url:. / 2023 / zwgk / ,id:12},{url:. / 2023 / zwgk / ,id:13},{url:. / 2023 / zwgk / ,id:14},{url:. / 2023 / zwgk / ,id:15}],message:success}
[0106] Data collection requires extracting the corresponding URL and ID from the data. Assuming the current website domain is https: / / example.com, the fields in the JSON can be directly used during data extraction.
[0107] The design is as follows, using the Named Placeholder Language (NOL):
[0108] ["{{data[].url}}","{{data[].id}}"]
[0109] The formatted replacement template is designed as follows:
[0110] https: / / example.com / {data[].url} / {data[].id}
[0111] As shown in the flowchart, the data is first extracted to obtain the following JSON:
[0112] {"url":[". / 2023 / zwgk / ",". / 2023 / zwgk / ",". / 2023 / zwgk / ",". / 2023 / zwgk / "],"id":[12,13,14,15]}
[0113] The obtained JSON was formatted and replaced to obtain the following data:
[0114] [https: / / example.com / 2023 / zwgk / 12,https: / / example.com / 2023 / zwgk / 13,https: / / example.com / 2023 / zwgk / 14,https: / / example.com / 2023 / zwgk / 15]
[0115] This completes the function of extracting data and formatting and replacing URLs from the website returned by the API interface using a named placeholder language.
[0116] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.
[0117] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0118] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.
[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A data acquisition method based on named placeholder language, characterized in that: The data acquisition method includes: Initiate a query request through the API interface to retrieve JSON data; The JSON data is processed by extracting fields according to a pre-defined named placeholder language; the extracted fields are then formatted and replaced, and the formatted and replaced results are returned to the API interface, thus completing the data collection. The preset naming placeholder language is an array structure, and each element in the array structure includes a field name to be collected, a collection type, and parameters corresponding to the collection type. The JSON format is as follows: {"name1": ["value11", "value12", "value13", ...], "name2": ["value21", "value22", "value23", ...], ...} Where name i represents the name of the i-th field to be collected, and ["valuei1", "valuei2", "valuei3",...] represents an array of fields extracted from the JSON based on the data level specified by name i. The process for formatting and replacing the extracted fields is as follows: Based on the field names to be collected in the pre-defined named placeholder language, a formatted replacement template is determined; The extracted field results corresponding to different field names are assigned to the preset formatted replacement template to complete the formatted replacement; The template for the formatted replacement is shown below: https: / / {name1} / {name2} / ... Where name i represents the name of the i-th field to be collected; The result of the formatted replacement is shown below: [https: / / value11 / value21,... https: / / value12 / value22,... https: / / value13 / value23,... ...] Here, "valuei1", "valuei2", "valuei3", ... represent several fields extracted from the JSON based on the data level specified by name i.
2. The data acquisition method based on named placeholder language according to claim 1, characterized in that: The field names are represented by dot symbols and are used to specify the data level of the field to be collected in the JSON returned by the API interface, as well as to determine the position where the extracted field needs to be replaced during the formatting and replacement process.
3. The data acquisition method based on named placeholder language according to claim 2, characterized in that: The JSON data is a string of text, and the field extraction process is as follows: Iterate through the named placeholder language of the array structure and extract the field names that need to be collected. Based on the data level specified by the field name to be collected, the fields are extracted from the JSON, and the extracted fields are assembled into JSON format.
4. The data acquisition method based on named placeholder language according to claim 2, characterized in that: The collection type has two fixed values, "XPath" and "regular", which indicate which type of parsing is performed on the field specified by the field name to be collected; the parameter corresponding to the collection type represents the specific parameter specified by the collection type, that is, an XPath or a regular expression.
5. The data acquisition method based on named placeholder language according to claim 4, characterized in that: The JSON data is HTML-type text, and the field extraction process is as follows: Iterate through the named placeholder language of the array structure and extract the field names that need to be collected. Based on the data level specified by the field name to be collected, extract the fields from the JSON to obtain the HTML text fields; If the data collection type is XPath, then the HTML text fields are converted into DOM nodes, and the corresponding XPath is used to extract the fields. If the data collection type is regular, use the corresponding regular expression to extract fields from the HTML text fields; The extracted fields are assembled into JSON format.
6. An information acquisition device, characterized in that: include: Memory, used to store computer programs; A processor, configured to implement the steps of the data acquisition method based on a named placeholder language as described in any one of claims 1 to 5 when executing the computer program.
7. A computer-readable storage medium, characterized in that: The computationally readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data acquisition method based on a named placeholder language as described in any one of claims 1 to 5.