Data analysis mapping method and system based on context awareness

Abstract

The invention provides a data analysis mapping method and system based on context awareness, and the method comprises the steps: carrying out the directional crawling of a target website through a crawler instance, so as to obtain multi-format webpage source data, then extracting the structural features, semantic features and source features of the multi-format webpage source data, and constructing a source data context feature set; dynamically matching an adaptive analysis strategy based on the source data context feature set to accurately extract target field data, performing semantic similarity matching on the target field data and a preset service scene standard field library, and performing data format unified conversion in combination with a context format rule to obtain standardized field data; and finally, performing multi-dimensional verification on the standardized field data, and generating an SQL statement after the verification is passed, so as to solve the problems that an existing data analysis mapping scheme depends on a fixed template and cannot dynamically adapt to page structure change of a target website, manual auxiliary field mapping consumes time, and errors are easy to introduce.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crawler data processing, and in particular to a data analysis mapping method and system based on context awareness. BACKGROUND

[0002] With the deep application of big data technology in the fields of finance, e-commerce, government affairs, etc., the demand for directional analysis of multi-source heterogeneous web page data and mapping of business standard field library is increasingly urgent, and efficient implementation of standardized processing of such data has become the core prerequisite for supporting subsequent data analysis and storage.

[0003] In the prior art, the data analysis mapping scheme adopts a fixed template configuration and manual assistance implementation method. For the target website to be crawled, the technical personnel need to analyze its fixed page structure in advance, write exclusive field positioning rules through the template configuration link, such as setting a specific tag path for an HTML (HyperText Markup Language, HyperText Markup Language) page, defining a target key name for JSON (JavaScript Object Notation, Object Notation) data, and pre-setting corresponding format conversion rules to form an analysis template that only adapts to the current structure of the website. After completing the template configuration, the crawler module will start the directional crawling task according to the pre-set URL (Uniform Resource Locator, Uniform Resource Locator) list, obtain multi-format web page source data including HTML static pages, JSON interface data, XML (Extensible Markup Language, Extensible Markup Language) configuration data, etc., and directly transmit the multi-format web page source data to the fixed analysis module. The fixed analysis module can only execute data extraction according to the fixed template configured in advance: filtering target field data from multi-format web page source data according to the pre-set field positioning rules. If the page structure of the target website changes, the fixed template will be invalid, and the technical personnel must re-analyze the page structure and modify the template configuration, otherwise the effective data cannot be continuously extracted.

[0004] The existing scheme relies on a fixed analysis template and can only adapt to the static page structure of the target website, and cannot dynamically adjust the analysis strategy according to the structural features, semantic associations and source rules of the web page source data. When the page structure of the target website changes, the template needs to be manually reconfigured, and the adaptability and flexibility are poor. SUMMARY

[0005] The present application provides a data analysis mapping method and system based on context awareness to solve the problem that the existing data analysis mapping scheme relies on a fixed template and cannot dynamically adapt to changes in the page structure of the target website, and manual assistance field mapping is time-consuming and prone to errors.

[0006] On one hand, the present invention provides a context-aware data parsing and mapping method, comprising: By using a web crawler instance to perform targeted crawling of the target website, multi-format webpage source data can be obtained; The structural features, semantic features, and source of the multi-format webpage source data are extracted to obtain the source data context feature set; Based on the source data context feature set, a parsing strategy is matched, and based on the parsing strategy, target field data is extracted from the multi-format webpage source data; The target field data is semantically similar to a preset standard field library for business scenarios. Combined with the format rules of the source data context feature set, a unified data format conversion is performed to obtain the mapped standardized field data. Perform multi-dimensional validation on the standardized field data. If the validation passes, generate an SQL statement based on the standardized field data.

[0007] Optionally, structural features, semantic features, and source extraction are performed on the multi-format webpage source data to obtain a source data context feature set, including: The inherent hierarchical structure, inter-element associations, and element attribute information of the multi-format webpage source data are extracted using parsing tools to form structured structural features. Load a domain dictionary that matches the target website scenario, perform dictionary matching on the multi-format webpage source data, mark domain terms and their positions in the multi-format webpage source data, and identify the business logic relationships of the domain terms through field association analysis to form semantic features; Read the request logs of the multi-format webpage source data and extract the target website domain name and the basic characteristics of the multi-format webpage source data; The historical crawling database is accessed to analyze the naming patterns and format characteristics of the fields corresponding to the basic features, thus forming source features; The structural features, semantic features, and source features are associated and mapped to form a source data context feature set.

[0008] Optionally, the inherent hierarchical structure, inter-element associations, and element attribute information of the multi-format webpage source data are extracted using a parsing tool to form structured structural features, including: The multi-format webpage source data is structured by parsing tools to identify the nesting depth and hierarchical correspondence of each component in the data, thereby obtaining the inherent hierarchical structure relationship. Based on the inherent hierarchical structure, the inclusion relationship, mapping relationship and dependency relationship between different components in the data are analyzed, and the association features between elements are extracted. Extract the identification information, data type, content length, and additional attributes of each component in the data to obtain element attribute information; The inherent hierarchical structure, the inter-element associations, and the element attribute information are structurally integrated to form structural features.

[0009] Optionally, based on the source data context feature set, a parsing strategy is matched, and based on the parsing strategy, target field data is extracted from the multi-format webpage source data, including: Based on the structural features in the source data context feature set, the data format type and hierarchical parsing rules are determined; combined with the semantic features and the source features, a parsing strategy that is compatible with the structural features, the semantic features and the source features is matched from a preset parsing strategy library, wherein the parsing strategy includes parsing tools, field location methods and data extraction rules; Based on the matching parsing strategy, the parsing tool is invoked to traverse the multi-format webpage source data according to the hierarchical parsing rules, and the data unit where the target field data is located is locked according to the field positioning method. The original field value is extracted from the data unit through the data extraction rules. Redundant information is removed and integrity is verified from the original field values ​​to obtain the target field data.

[0010] Optionally, based on the matching parsing strategy, the parsing tool is invoked to traverse the multi-format webpage source data according to the hierarchical parsing rules, locate the data unit containing the target field data according to the field positioning method, and extract the original field value from the data unit according to the data extraction rules, including: The parsing tool that matches the parsing strategy is invoked to perform structured traversal of multi-format webpage source data according to the hierarchical parsing rules. For HTML format data, the DOM tree node level is traversed; for JSON format data, the key-value nesting level is traversed; and for XML format data, the tag nesting level is traversed. The element identifiers and relationships of each level are recorded synchronously. Based on the field location method, and combined with the structural and semantic features in the context feature set of the source data, candidate data units including the target field are locked during the traversal process; Data extraction rules are applied to candidate data units. For HTML format data, BeautifulSoup is used to obtain the text or attribute values ​​within tags. For JSON / XML format data, the corresponding values ​​are accurately located and extracted by key name / tag name. For units containing complex content, regular expressions are used to match and extract the field content that conforms to the business rules, thus obtaining the original field values.

[0011] Optionally, the target field data is semantically similar to a preset standard field library for business scenarios, and a unified data format conversion is performed based on the format rules of the source data context feature set to obtain the mapped standardized field data, including: Extract the field name, field value type, and semantic description features of the target field data to generate a target field feature vector; Extract the name, type constraints, and semantic specification features of standard fields from the preset business scenario standard field library, and generate standard field feature vectors; The similarity value between the feature vector of the target field and the feature vector of the standard field is calculated using a semantic similarity algorithm; When the similarity value reaches a preset threshold, a mapping relationship between the target field data and the standard field is established; Obtain the format pattern of the target field data from the source data context feature set; Based on the mapping relationship and the format rules, the field values ​​of the target field data are converted to be consistent with the format of the standard field. The target field data is linked and integrated to form standardized field data whose field names, semantics, and formats conform to the requirements of the standard field library for business scenarios.

[0012] Optionally, when the similarity value reaches a preset threshold, establishing a mapping relationship between the target field data and the standard field further includes: By combining the semantic features and the source features, a context weight factor is generated, and the similarity value is adjusted by weighting to obtain a weighted similarity value; When the weighted similarity value exceeds a preset threshold, an initial mapping relationship between the target field and the corresponding standard field is established; Based on the structural features in the source data context feature set, the initial mapping relationship is validated for consistency, and contradictory mappings are removed to form a new mapping relationship.

[0013] Optionally, multi-dimensional validation is performed on the standardized field data. If the validation passes, an SQL statement is generated based on the standardized field data, including: Perform integrity checks, format checks, logical checks, and uniqueness checks on the standardized field data; If the integrity check, the format check, and the logic check all pass, then the data type and length constraints in the standardized field data that passed the checks are parsed, and the SQL generation engine is called to obtain an executable SQL stored statement.

[0014] Optionally, it also includes: Historical multi-format webpage source data, source data context feature sets, and corresponding mapping relationship records are collected according to a preset time period. The evolution parameters of the structural features, the temporal correlation degree of the semantic features, and the stability index of the source features within the preset time period are extracted to construct a time series context dataset. A time series context model is generated by training the time series context dataset using a time series modeling algorithm. The deviation value between the source data context feature set and the historical time series features is calculated based on the time series context model. When the deviation value continuously exceeds a preset threshold or a preset amount of update is detected in the target website data structure, the parsing strategy is updated.

[0015] On the other hand, the present invention also provides a context-aware data parsing and mapping system, comprising: The targeted crawling module is used to perform targeted crawling on target websites using crawler instances to obtain multi-format webpage source data; The feature extraction module is used to extract structural features, semantic features, and source information from the multi-format webpage source data to obtain a source data context feature set. The parsing and extraction module is used to match a parsing strategy based on the source data context feature set, and extract target field data from the multi-format webpage source data based on the parsing strategy. The field mapping module is used to perform semantic similarity matching between the target field data and the preset business scenario standard field library, and to perform unified data format conversion in combination with the format rules of the source data context feature set to obtain the mapped standardized field data. The validation module is used to perform multi-dimensional validation on the standardized field data. If the validation passes, an SQL statement is generated based on the standardized field data.

[0016] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the context-aware data parsing and mapping method as described above.

[0017] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the context-aware data parsing and mapping method as described above.

[0018] On the other hand, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the context-aware data parsing and mapping method as described above.

[0019] This invention provides a context-aware data parsing and mapping method and system. The method uses a web crawler instance to perform targeted crawling on a target website to obtain multi-format webpage source data. Subsequently, it extracts structural features, semantic features, and source features from the multi-format webpage source data and constructs a source data context feature set. Then, it dynamically matches and adapts a parsing strategy based on the source data context feature set to accurately extract target field data. Next, it performs semantic similarity matching with a preset business scenario standard field library and performs a unified data format conversion based on context format rules to obtain standardized field data. Finally, it performs multi-dimensional verification on the standardized field data and generates SQL statements after the verification passes. This solves the problems of existing data parsing and mapping schemes that rely on fixed templates and cannot dynamically adapt to changes in the target website page structure, and that manual field mapping is time-consuming and prone to errors. It improves the automation level of data parsing and mapping, the flexibility of adapting to changes in website structure, and significantly reduces the cost of manual intervention. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the context-aware data parsing and mapping method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the context-aware data parsing and mapping system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0023] Figure 1 This is a flowchart illustrating the context-aware data parsing and mapping method provided in an embodiment of the present invention.

[0024] like Figure 1As shown, the context-aware data parsing and mapping method provided in this embodiment of the invention mainly includes the following steps: 101. Perform targeted crawling on the target website using a crawler example to obtain multi-format webpage source data.

[0025] When performing targeted crawling on target websites, the crawler's crawling strategy is flexibly adjusted based on the website's characteristics, such as webpage structure and data storage methods. For example, for webpages using dynamic loading technology, the crawler accurately obtains complete source data by simulating user operations and analyzing webpage scripts. Simultaneously, to ensure the legality and stability of the crawling process, the crawler strictly adheres to the target website's robots.txt protocol and sets reasonable crawling intervals to avoid placing excessive access pressure on the target website. Targeted crawling enables the efficient collection of multi-format webpage source data.

[0026] 102. Extract structural features, semantic features, and source information from multi-format webpage source data to obtain a source data context feature set.

[0027] After obtaining multi-format webpage source data, feature extraction is required. For structural feature extraction, the HTML or XML tag structure of the webpage source data is parsed to identify key data blocks and layout information, such as tables, lists, and paragraphs. Structural features help in understanding the organization and presentation of the data. Secondly, for semantic feature extraction, natural language processing techniques are used to segment, tag, and name entity recognition of the text content in the webpage, extracting meaningful words and phrases, as well as semantic relationships between text content, thereby capturing the core content and theme of the data. Finally, source extraction focuses on the origin and link information of the webpage source data. By tracing the original publisher and dissemination path of the data, credibility assessment and source tracing support are provided for the data.

[0028] Specifically, structural features, semantic features, and source extraction are performed on multi-format webpage source data to obtain a source data context feature set, including: The inherent hierarchical structure, inter-element associations, and element attribute information of multi-format webpage source data are extracted using parsing tools to form structured structural features. Load a domain dictionary that matches the target website scenario, perform dictionary matching on multi-format webpage source data, mark domain terms and their positions in the multi-format webpage source data, and identify the business logic relationships of domain terms through field association analysis to form semantic features.

[0029] Read request logs of multi-format webpage source data to extract the basic characteristics of the target website domain and multi-format webpage source data.

[0030] By accessing historical crawled databases, the naming patterns and format characteristics of fields corresponding to basic features are statistically analyzed to form source features.

[0031] By associating and mapping structural features, semantic features, and source features, a source data context feature set is formed.

[0032] The process involves using parsing tools to perform structured parsing on multi-format webpage source data, identifying the nesting depth and hierarchical correspondence of each component in the data to obtain the inherent hierarchical structure relationship, and then analyzing the inclusion, mapping, and dependency relationships between different components based on the inherent hierarchical structure relationship to extract the association features between elements. In addition, the identification information, data type, content length, and additional attributes of each component are extracted to obtain element attribute information. After the information is structured and integrated, a structured structural feature is formed.

[0033] Subsequently, a domain dictionary matching the target website scenario is loaded. The domain dictionary is then used to perform dictionary matching on multi-format webpage source data, marking domain terms and their specific locations in the data. Then, the business logic relationships between these domain terms are identified through field association analysis, thereby forming semantic features.

[0034] Simultaneously, request logs of multi-format webpage source data are read to extract the target website domain name and basic characteristics of multi-format webpage source data. Historical crawling databases are also called to perform statistical analysis on the naming rules and format characteristics of the fields corresponding to the basic characteristics, thereby forming source characteristics.

[0035] Finally, the structural features, semantic features, and source features constructed above are associated and mapped to form a source data context feature set that includes the data's inherent structure, domain semantic associations, and source scenario information.

[0036] Furthermore, parsing tools are used to extract the inherent hierarchical structure, inter-element relationships, and element attribute information from multi-format webpage source data, forming structured structural features, including: By using parsing tools to perform structured parsing of multi-format webpage source data, the nesting depth and hierarchical correspondence of each component unit in the data are identified, and the inherent hierarchical structure is obtained. Based on the inherent hierarchical structure, we analyze the inclusion, mapping, and dependency relationships between different components in the data and extract the association features between elements. Extract the identification information, data type, content length, and additional attributes of each component in the data to obtain element attribute information; The inherent hierarchical structure, the interrelationships between elements, and the attribute information of elements are structurally integrated to form structural features.

[0037] This involves using parsing tools to perform structured parsing operations on multi-format webpage source data. Different parsing tools can be used for different data types. For example, for HTML data, the DOM (Document Object Model) tree structure can be parsed; for JSON data, the key-value nesting logic can be parsed; and for XML data, the tag hierarchy can be parsed. This allows for the accurate identification of the nesting depth and hierarchical correspondence of each component in the data, revealing the inherent hierarchical structure.

[0038] Next, based on the inherent hierarchical structure, we further analyze the various association patterns between different components in the data. These association patterns include inclusion relationships, such as the wrapping and mapping relationship between a list unit and multiple data item units, the mapping and dependency relationship between field units corresponding to functions in different modules, and the data dependency of computational field units on basic field units. We extract the association features between elements from these association patterns.

[0039] Subsequently, the specific attributes of each component in the data are extracted to obtain element attribute information, including identification information, data type, content length, and additional attributes.

[0040] Finally, the inherent hierarchical structure, the interrelationships between elements, and the element attribute information are integrated according to the preset structured framework to form a complete structured feature.

[0041] 103. Based on the source data context feature set, match the parsing strategy, and extract the target field data from the multi-format webpage source data based on the parsing strategy.

[0042] The matching and parsing strategy comprehensively considers structural features, semantic features, and source features from the source data's contextual feature set. For example, if structural features indicate that the webpage data is presented in tabular form, semantic features indicate that the table content is related to a specific business domain, and source features indicate the data's publishing channel, then a parsing strategy that is specific to the tabular data and adapted to that business domain will be prioritized. Based on the matched parsing strategy, target field data is accurately extracted from multi-format webpage source data. This target field data contains key information required for subsequent business processing.

[0043] Specifically, based on the source data context feature set, a parsing strategy is matched, and based on the parsing strategy, target field data is extracted from multi-format webpage source data, including: Based on the structural features in the source data context feature set, the data format type and hierarchical parsing rules are determined; combined with semantic features and source features, parsing strategies that are compatible with the structural features, semantic features and source features are matched from the preset parsing strategy library. The parsing strategies include parsing tools, field location methods and data extraction rules.

[0044] Based on the matching parsing strategy, the parsing tool is invoked to traverse the multi-format webpage source data according to the hierarchical parsing rules. The data unit containing the target field data is located according to the field positioning method, and the original field value is extracted from the data unit through the data extraction rules.

[0045] Redundant information is removed and integrity is verified from the original field values ​​to obtain the target field data.

[0046] Among them, based on the structural features in the context feature set of source data, the data format type and corresponding hierarchical parsing rules of multi-format web page source data are clarified. For example, the data format type of multi-format web page source data is any one of HTML, JSON, and XML, and the hierarchical parsing rules are DOM tree traversal path, key-value nesting hierarchical order, and tag nesting parsing logic.

[0047] Then, combining semantic and source features, parsing strategies that are compatible with all three features are selected from a pre-defined parsing strategy library. These strategies specifically include suitable data parsing tools, field location methods, and targeted data extraction rules. For example, BeautifulSoup is used for HTML data, and a JSON parsing library is used for JSON data. Field location methods can utilize node locking based on domain terminology and unit filtering based on hierarchical relationships. Extraction rules can include rules for extracting text within tags and regular expression matching rules.

[0048] Next, based on the matching parsing strategy, the corresponding parsing tool is called to perform a structured traversal of the multi-format webpage source data according to the determined hierarchical parsing rules. For example, HTML format data is traversed by traversing DOM tree nodes, and JSON data format is traversed by traversing key-value nesting levels. During the traversal, the specific data unit containing the target field data is locked according to the field positioning method, and the original field value is extracted from the data unit according to the data extraction rules.

[0049] Finally, redundant information is removed and integrity is verified from the extracted original field values ​​to obtain the target field data. For example, extra spaces and irrelevant comment text are removed, and the presence of missing key information is checked to obtain the target field data.

[0050] The parsing strategy, based on matching, invokes parsing tools to traverse multi-format webpage source data according to hierarchical parsing rules. It then locates the data unit containing the target field data using field positioning methods and extracts the original field values ​​from the data unit according to data extraction rules. Specifically, this includes: The parsing tool that matches the parsing strategy is invoked to perform structured traversal of multi-format webpage source data according to the hierarchical parsing rules. For HTML format data, the DOM tree node level is traversed; for JSON format data, the key-value nesting level is traversed; and for XML format data, the tag nesting level is traversed. The element identifiers and relationships of each level are recorded synchronously. Based on the field location method, and combined with the structural and semantic features in the context feature set of the source data, candidate data units including the target field are locked during the traversal process; Data extraction rules are applied to candidate data units. For HTML format data, BeautifulSoup is used to obtain the text or attribute values ​​within tags. For JSON / XML format data, the corresponding values ​​are accurately located and extracted by key name / tag name. For units containing complex content, regular expressions are used to match and extract the field content that conforms to the business rules, thus obtaining the original field values.

[0051] Specifically, the parsing tool matching the parsing strategy is first invoked to perform a structured traversal of the multi-format webpage source data according to the hierarchical parsing rules. For example, for HTML format data, it traverses sequentially along the DOM tree node hierarchy; for JSON format data, it traverses step by step along the key-value nesting hierarchy; and for XML format data, it traverses each element along the tag nesting hierarchy. At the same time, the element identifiers at each level are recorded, such as the id / class of HTML tags, the key names of JSON, the tag names of XML, as well as the relationships between elements, the ownership of each parent and child node, and key-value dependencies.

[0052] Subsequently, based on the field location method and combined with the structural and semantic features of the source data context feature set, candidate data units including the target field are locked during the above traversal process. Then, data extraction rules are applied to the candidate data units. For example, for HTML format data, the BeautifulSoup parser is used to obtain the text content or specified attribute value within the target tag. For JSON / XML format data, the corresponding data item is located by key name / tag name and the attribute value is extracted. For units containing complex content such as messy characters and nested text, a preset regular expression is called to match and filter, extracting the field content that conforms to the business rules, and finally obtaining the original field value.

[0053] 104. Perform semantic similarity matching between the target field data and the preset standard field library for business scenarios, and perform unified data format conversion based on the format rules of the source data context feature set to obtain the mapped standardized field data.

[0054] In particular, uniformly converting the format of target field data ensures consistency across different business scenarios, eliminating data processing obstacles caused by format differences and improving the compatibility and accuracy of data in subsequent analysis, storage, and application stages. For example, uniformly converting different date formats to the standard YYYY-MM-DD format ensures that data can be accurately processed in different business systems.

[0055] Specifically, the target field data is semantically matched with a pre-defined standard field library for business scenarios. Then, a unified data format conversion is performed based on the format rules of the source data's contextual feature set to obtain the mapped, standardized field data, including: Extract the field name, field value type, and semantic description features of the target field data to generate the target field feature vector.

[0056] The process involves extracting the identifying names of the target fields from the extracted target field data, determining the data type of the field values ​​(e.g., character, integer, date), and then mining the semantic descriptive features of the fields, i.e., the specific meanings represented by the fields in the business scenario, such as the order date corresponding to the time when the transaction occurred, and the product price corresponding to the selling price of the goods.

[0057] Subsequently, natural language processing techniques are used to quantize the features into numerical vectors, forming target field feature vectors that can be used for calculation. The dimensions of the vectors correspond one-to-one with the feature dimensions, ensuring the comparability of the features.

[0058] Extract the name, type constraints, and semantic specification features of standard fields from the preset business scenario standard field library, and generate standard field feature vectors.

[0059] Specifically, for a pre-defined standard field library for business scenarios, features of each standard field are extracted. These features include the official name of the standard field, the type constraints of the field value, and the semantic specification features of the field, i.e., the meaning of the standard field as explicitly defined in the business rules, such as the transaction date being specified as the date of successful order payment. The same quantization method used to generate the target field feature vector is employed to convert the features into standard field feature vectors, ensuring that the calculation dimensions match those of the target field feature vectors.

[0060] A semantic similarity algorithm is used to calculate the similarity value between the feature vector of the target field and the feature vector of the standard field.

[0061] When calculating similarity values, semantic similarity algorithms suitable for text and feature vectors can be selected, such as the word vector cosine distance algorithm or edit distance algorithm. These algorithms numerically calculate the similarity between the target field's feature vector and the feature vectors of each standard field. The cosine distance algorithm characterizes similarity by calculating the cosine of the angle between two vectors, while the edit distance algorithm measures similarity by calculating the number of character modifications made to the names or semantic descriptions of two fields. The final result is a specific similarity value between the target field and each standard field, used to determine the degree of field matching.

[0062] When the similarity value reaches a preset threshold, a mapping relationship is established between the target field data and the standard field.

[0063] The process begins by obtaining a preset threshold and comparing the calculated similarity value with that threshold. If the similarity value between a standard field and the target field exceeds the threshold, it indicates a high degree of match in name, type, and semantics. In this case, a mapping relationship is automatically established between the target field data and the standard field. For example, the target field "order date" is mapped to the standard field "transaction date".

[0064] Extract the format pattern of the target field data from the source data context feature set.

[0065] Specifically, the format patterns related to the target field data are extracted from the source features of the source data context feature set. These patterns include historical format characteristics of fields within the same target website, format habits of similar data sources, and common format variations of field values. These format patterns provide a basis for subsequent format conversion, ensuring that the conversion conforms to the actual characteristics of the data source.

[0066] Based on the mapping relationship and format rules, the field values ​​of the target field data are converted to match the format of the standard field data.

[0067] Based on the established mapping relationship, the format requirements of the corresponding standard fields are clarified; simultaneously, combined with the acquired format patterns, the original format characteristics of the target field data are determined. Then, targeted format conversion operations are performed. For example, for date fields, MM / DD / YYYY are converted to YYYY-MM-DD using date parsing functions; for numeric fields, signs are removed and two decimal places are uniformly retained; for text fields, case is standardized or special characters are removed. Ultimately, it is ensured that the format of the converted target field data is completely consistent with the standard field.

[0068] The target field data is linked and integrated to form standardized field data whose field names, semantics, and formats conform to the requirements of the standard field library for business scenarios.

[0069] During the integration process, it is ensured that the name of each field is consistent with the standard field library, the semantics match the business definition of the standard field, and the format strictly conforms to the constraints of the standard field, ultimately forming complete standardized field data that can be directly used for subsequent data storage and business analysis.

[0070] Specifically, when the similarity value reaches a preset threshold, establishing a mapping relationship between the target field data and the standard field also includes: By combining semantic features and source features, a context weight factor is generated, and the similarity value is adjusted by weighting to obtain a weighted similarity value; When the weighted similarity value exceeds a preset threshold, an initial mapping relationship is established between the target field and the corresponding standard field; Based on the structural features in the source data context feature set, the initial mapping relationship is checked for consistency, and the mapping relationship is formed after eliminating contradictory mappings.

[0071] Specifically, context weight factors are generated by combining semantic features and source features from the source data's context feature set using a feature importance evaluation algorithm. For example, context weight factors can be generated using random forest feature weights.

[0072] The context weight factor is weighted by the basic similarity value to obtain a weighted similarity value that better reflects the actual scenario. When the weighted similarity value exceeds a preset threshold, it indicates that the target field and the standard field meet the requirements in terms of semantic matching and scenario adaptability, thereby establishing an initial mapping relationship between the target field and the corresponding standard field.

[0073] Finally, based on the structural features in the source data context feature set, the initial mapping relationship is validated for consistency. For example, it checks whether there are logical conflicts in the mapping of related fields in the same level, and whether the mapping of dependent fields matches the main field. After eliminating contradictory mapping relationships, an accurate final mapping relationship that conforms to the data structure logic is formed.

[0074] 105. Perform multi-dimensional validation on standardized field data. If the validation passes, generate SQL statements based on the standardized field data.

[0075] Once the standardized field data passes multi-dimensional validation, a predefined SQL (Structured Query Language) template can be used, combined with information from the standardized field data such as field names, data types, and constraints, to dynamically populate placeholders in the SQL template. This generates SQL statements that conform to database syntax and accurately reflect data mapping relationships. The SQL statements can then be used for subsequent data insertion, update, or query operations.

[0076] Specifically, multi-dimensional validation is performed on the standardized field data. If the validation passes, an SQL statement is generated based on the standardized field data, including: Perform integrity checks, format checks, logical checks, and uniqueness checks on standardized field data; If the integrity check, format check, and logical check all pass, the data type and length constraints in the standardized field data that passed the checks are parsed, and the SQL generation engine is called to obtain an executable SQL stored statement.

[0077] When performing integrity checks, format checks, logical checks, and uniqueness checks on standardized field data, the integrity check compares the standardized field data one by one with the core field list of the preset business scenario standard field library to check for any missing core fields. The format check checks whether the field value format is compliant based on the format specifications of each field in the standard field library through format template matching or regular expression verification. The logical check verifies the numerical matching, business rule compliance, and reasonableness of dependency relationships between fields with business relationships. The uniqueness check extracts the unique identifier features from the standardized field data and compares them with historical stored data to check for duplicate data.

[0078] If the above integrity, format, logic, and uniqueness checks all pass, the data type and length constraints of each field in the standardized field data that has passed the checks are further parsed. The SQL generation engine is then called to generate an insert statement that matches the field order and data type of the standardized field data, ultimately resulting in a stored SQL statement that can be directly executed in the target database.

[0079] In some embodiments, the context-aware data parsing and mapping method provided by the present invention further includes: Historical multi-format webpage source data, source data context feature sets, and corresponding mapping relationship records are collected according to a preset time period. Evolution parameters of structural features, temporal correlation of semantic features, and stability indicators of source features are extracted within the preset time period to construct a time series context dataset. A time series context model is generated by training a time series context dataset using a time series modeling algorithm. The deviation between the source data context feature set and the historical time series features is calculated based on the time series context model. When the deviation value continuously exceeds the preset threshold or when the target website data structure is updated by a preset amount, the parsing strategy is updated.

[0080] Specifically, according to a pre-set time period, historical multi-format webpage source data, corresponding source data context feature sets, and past field mapping relationship records are continuously collected. Based on this, the evolution parameters of structural features, the temporal correlation of semantic features, and the stability indicators of source features within the pre-set time period are further extracted. For example, evolution parameters include the rate of change of data unit nesting levels and the frequency of changes in element association relationships; temporal correlation includes the co-occurrence pattern of domain terms in different time periods and the temporal stability of business logic relationships; and stability indicators include the fluctuation frequency of naming formats of fields in the same source data and the success rate of adaptation of historical mapping relationships.

[0081] After integrating the feature data of the time series dimension, a time series context dataset is constructed. Then, a modeling algorithm suitable for time series data is adopted, such as a long short-term memory network. The constructed time series context dataset is used as a training sample. Through iterative training, the data features are learned to change over time, and a time series context model that can characterize the time series evolution trend of data features is generated.

[0082] Finally, based on the time series context model, the deviation between the currently acquired source data context feature set and the historical time series features is calculated in real time. When the deviation exceeds the preset threshold for multiple consecutive periods, or when the target website data structure is found to have undergone a preset level of update, the parsing strategy update mechanism is automatically triggered. The latest time series context features are called to rematch the appropriate parsing tools, field positioning methods and data extraction rules, and the parsing strategy is dynamically updated to adapt to the time series changes of data features.

[0083] Based on the same general inventive concept, this invention also protects a context-aware data parsing and mapping system. The context-aware data parsing and mapping system provided by this invention will be described below. The context-aware data parsing and mapping system described below and the context-aware data parsing and mapping method described above can be referred to each other.

[0084] like Figure 2 As shown, embodiments of the present invention also provide a context-aware data parsing and mapping system, comprising: The targeted crawling module 210 is used to perform targeted crawling on the target website through a crawler instance to obtain multi-format webpage source data; The feature extraction module 220 is used to extract structural features, semantic features, and source information from multi-format webpage source data to obtain a source data context feature set. The parsing and extraction module 230 is used to match the parsing strategy based on the context feature set of the source data, and extract the target field data from the multi-format web page source data based on the parsing strategy. The field mapping module 240 is used to perform semantic similarity matching between the target field data and the preset business scenario standard field library, and to perform unified data format conversion in combination with the format rules of the source data context feature set to obtain the mapped standardized field data. The validation module 250 is used to perform multi-dimensional validation on the standardized field data. If the validation passes, an SQL statement is generated based on the standardized field data.

[0085] Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention.

[0086] like Figure 3 As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340. The processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions from the memory 330 to execute a context-aware data parsing and mapping method.

[0087] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the context-aware data parsing and mapping method provided by the above methods.

[0089] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the context-aware data parsing and mapping methods provided by the above methods.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A context-aware data parsing and mapping method, characterized in that, include: By using a web crawler instance to perform targeted crawling of the target website, multi-format webpage source data can be obtained; The structural features, semantic features, and source of the multi-format webpage source data are extracted to obtain the source data context feature set; Based on the source data context feature set, a parsing strategy is matched, and based on the parsing strategy, target field data is extracted from the multi-format webpage source data; The target field data is semantically similar to a preset standard field library for business scenarios. Combined with the format rules of the source data context feature set, a unified data format conversion is performed to obtain the mapped standardized field data. Perform multi-dimensional validation on the standardized field data. If the validation passes, generate an SQL statement based on the standardized field data.

2. The context-aware data parsing and mapping method according to claim 1, characterized in that, The source data of the multi-format web pages is subjected to structural features, semantic features, and source extraction to obtain a source data context feature set, including: The inherent hierarchical structure, inter-element associations, and element attribute information of the multi-format webpage source data are extracted using parsing tools to form structured structural features. Load a domain dictionary that matches the target website scenario, perform dictionary matching on the multi-format webpage source data, mark domain terms and their positions in the multi-format webpage source data, and identify the business logic relationships of the domain terms through field association analysis to form semantic features; Read the request logs of the multi-format webpage source data and extract the target website domain name and the basic characteristics of the multi-format webpage source data; The historical crawling database is accessed to analyze the naming patterns and format characteristics of the fields corresponding to the basic features, thus forming source features; The structural features, semantic features, and source features are associated and mapped to form a source data context feature set.

3. The context-aware data parsing and mapping method according to claim 2, characterized in that, The inherent hierarchical structure, inter-element relationships, and element attribute information of the multi-format webpage source data are extracted using parsing tools to form structured structural features, including: The multi-format webpage source data is structured by parsing tools to identify the nesting depth and hierarchical correspondence of each component in the data, thereby obtaining the inherent hierarchical structure relationship. Based on the inherent hierarchical structure, the inclusion relationship, mapping relationship and dependency relationship between different components in the data are analyzed, and the association features between elements are extracted. Extract the identification information, data type, content length, and additional attributes of each component in the data to obtain element attribute information; The inherent hierarchical structure, the inter-element associations, and the element attribute information are structurally integrated to form structural features.

4. The context-aware data parsing and mapping method according to claim 2, characterized in that, Based on the source data context feature set, a parsing strategy is matched, and based on the parsing strategy, target field data is extracted from the multi-format webpage source data, including: Based on the structural features in the source data context feature set, the data format type and hierarchical parsing rules are determined; combined with the semantic features and the source features, a parsing strategy that is compatible with the structural features, the semantic features and the source features is matched from a preset parsing strategy library, wherein the parsing strategy includes parsing tools, field location methods and data extraction rules; Based on the matching parsing strategy, the parsing tool is invoked to traverse the multi-format webpage source data according to the hierarchical parsing rules, and the data unit where the target field data is located is locked according to the field positioning method. The original field value is extracted from the data unit through the data extraction rules. Redundant information is removed and integrity is verified from the original field values ​​to obtain the target field data.

5. The context-aware data parsing and mapping method according to claim 4, characterized in that, Based on the matching parsing strategy, the parsing tool is invoked to traverse the multi-format webpage source data according to the hierarchical parsing rules. The data unit containing the target field data is located according to the field positioning method. The original field values ​​are extracted from the data unit using the data extraction rules, including: The parsing tool that matches the parsing strategy is invoked to perform structured traversal of multi-format webpage source data according to the hierarchical parsing rules. For HTML format data, the DOM tree node level is traversed; for JSON format data, the key-value nesting level is traversed; and for XML format data, the tag nesting level is traversed. The element identifiers and relationships of each level are recorded synchronously. Based on the field location method, and combined with the structural and semantic features in the context feature set of the source data, candidate data units including the target field are locked during the traversal process; Data extraction rules are applied to candidate data units. For HTML format data, BeautifulSoup is used to obtain the text or attribute values ​​within tags. For JSON / XML format data, the corresponding values ​​are accurately located and extracted by key name / tag name. For units containing complex content, regular expressions are used to match and extract the field content that conforms to the business rules, thus obtaining the original field values.

6. The context-aware data parsing and mapping method according to claim 1, characterized in that, The target field data is semantically similar to a pre-defined standard field library for business scenarios. Then, a unified data format conversion is performed based on the format rules of the source data context feature set to obtain the mapped standardized field data, including: Extract the field name, field value type, and semantic description features of the target field data to generate a target field feature vector; Extract the name, type constraints, and semantic specification features of standard fields from the preset business scenario standard field library, and generate standard field feature vectors; The similarity value between the feature vector of the target field and the feature vector of the standard field is calculated using a semantic similarity algorithm; When the similarity value reaches a preset threshold, a mapping relationship between the target field data and the standard field is established; Obtain the format pattern of the target field data from the source data context feature set; Based on the mapping relationship and the format rules, the field values ​​of the target field data are converted to be consistent with the format of the standard field. The target field data is linked and integrated to form standardized field data whose field names, semantics, and formats conform to the requirements of the standard field library for business scenarios.

7. The context-aware data parsing and mapping method according to claim 6, characterized in that, When the similarity value reaches a preset threshold, a mapping relationship is established between the target field data and the standard field, which further includes: By combining the semantic features and the source features, a context weight factor is generated, and the similarity value is adjusted by weighting to obtain a weighted similarity value; When the weighted similarity value exceeds a preset threshold, an initial mapping relationship between the target field and the corresponding standard field is established; Based on the structural features in the source data context feature set, the initial mapping relationship is validated for consistency, and contradictory mappings are removed to form a new mapping relationship.

8. The context-aware data parsing and mapping method according to claim 1, characterized in that, Perform multi-dimensional validation on the standardized field data. If the validation passes, generate an SQL statement based on the standardized field data, including: Perform integrity checks, format checks, logical checks, and uniqueness checks on the standardized field data; If the integrity check, the format check, and the logic check all pass, then the data type and length constraints in the standardized field data that passed the checks are parsed, and the SQL generation engine is called to obtain an executable SQL stored statement.

9. The context-aware data parsing and mapping method according to claim 2, characterized in that, Also includes: Historical multi-format webpage source data, source data context feature sets, and corresponding mapping relationship records are collected according to a preset time period. The evolution parameters of the structural features, the temporal correlation degree of the semantic features, and the stability index of the source features within the preset time period are extracted to construct a time series context dataset. A time series context model is generated by training the time series context dataset using a time series modeling algorithm. The deviation value between the source data context feature set and the historical time series features is calculated based on the time series context model. When the deviation value continuously exceeds a preset threshold or a preset amount of update is detected in the target website data structure, the parsing strategy is updated.

10. A context-aware data parsing and mapping system, characterized in that, include: The targeted crawling module is used to perform targeted crawling on target websites using crawler instances to obtain multi-format webpage source data; The feature extraction module is used to extract structural features, semantic features, and source information from the multi-format webpage source data to obtain a source data context feature set. The parsing and extraction module is used to match a parsing strategy based on the source data context feature set, and extract target field data from the multi-format webpage source data based on the parsing strategy. The field mapping module is used to perform semantic similarity matching between the target field data and the preset business scenario standard field library, and to perform unified data format conversion in combination with the format rules of the source data context feature set to obtain the mapped standardized field data. The validation module is used to perform multi-dimensional validation on the standardized field data. If the validation passes, an SQL statement is generated based on the standardized field data.

Images