An alignment method based on overseas open source information structured analysis field

Abstract

This invention discloses a method for aligning fields based on structured parsing of overseas open-source information, relating to the field of data fusion and parsing. The method includes: acquiring raw data packets from overseas open-source platforms; performing structured parsing and field extraction on the raw data packets to generate a candidate field set; performing semantic recognition and field mapping on the fields in the candidate field set based on a dynamic cognitive network to generate a field mapping decision result; resolving and correcting the consistency of conflicting field values ​​mapped to the same standard field based on the field mapping decision result to obtain an aligned field set; and organizing the aligned field set into a unified structured record and outputting it according to the standard field model. This method achieves intelligent mapping and conflict resolution of cross-platform fields through an evolvable semantic network, possessing the ability to adaptively learn unknown fields, achieve deep semantic understanding, and output interpretable decisions, significantly improving the intelligence level and accuracy of multi-source information fusion.

Description

Technical Field

[0001] This invention relates to the field of data fusion and parsing, specifically to a method for aligning fields in structured parsing of information based on overseas open-source sources. Background Technology

[0002] With the development of the globalized digital economy and the deepening of cyberspace governance, overseas open-source platforms (such as social media, news websites, and public databases) have become important channels for obtaining and analyzing public information. This type of information comes from a wide range of sources, and its data formats and field representations vary considerably, exhibiting typical characteristics of multi-source heterogeneity.

[0003] Against this backdrop, when comprehensively utilizing overseas open-source information, it is usually necessary to integrate the raw data from different platforms into unified structured data to support subsequent analysis and applications. However, inconsistencies in field names and diverse semantic expressions between different data sources present challenges in field mapping and unified organization.

[0004] Currently, in existing technologies, field alignment processing mostly adopts manual rule configuration or static mapping table methods. However, the maintenance cost is high when facing changes in platform fields or the continuous emergence of new fields. In addition, although some supervised learning-based methods can improve the degree of automation, they often rely on labeled data, and their adaptability to dynamic semantic changes still needs to be improved.

[0005] Therefore, there is a pressing need for an intelligent field alignment method that can automatically understand field semantics, adapt to dynamic changes, and possess continuous learning capabilities, to efficiently, accurately, and adaptively handle the complex and ever-changing field semantic alignment issues in multi-source heterogeneous open-source information environments. Summary of the Invention

[0006] Based on the shortcomings of the existing technology described above, the purpose of this invention is to provide a method for aligning fields in structured parsing of information based on overseas open source, so as to solve the above-mentioned technical problems.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for aligning fields in structured parsing of information based on overseas open-source software, comprising: S1: Obtain the raw data packets from overseas open-source platforms, perform structured parsing and field extraction on the raw data packets, and generate a set of candidate fields; S2: Based on a dynamic cognitive network, semantic recognition and field mapping are performed on the fields in the candidate field set to generate field mapping decision results; S3: Based on the field mapping decision results, resolve and correct the consistency of conflicting field values ​​mapped to the same standard field to obtain an aligned field set; S4: Based on the standard field model, organize the aligned field set to generate a unified structured record and output it.

[0008] The present invention is further configured such that the original data packet includes: platform identifier, original field name set and corresponding original values, collection timestamp and heterogeneous data content.

[0009] The present invention is further configured such that S1 includes: For different types of heterogeneous data content in the original data packet, separate parsing and extraction are performed: For documents defined by markup language, locate and extract predefined page elements to obtain the first type of candidate fields and their corresponding values; The key-value pairs or nested hierarchical data returned by the application programming interface are parsed in a structured manner to obtain the second type of candidate fields and their corresponding values. For unstructured text content, named entity recognition and / or keyword extraction are used to obtain the third type of candidate fields and their corresponding values. Extract key descriptive information from the metadata of multimedia objects to obtain the fourth type of candidate fields and their corresponding values; Integrate the candidate fields and their corresponding values ​​from the first, second, third, and fourth categories to generate a candidate field set containing field names and their corresponding values.

[0010] The present invention is further configured such that S2 includes: Based on the candidate field set, generate corresponding semantic feature representations for each field in the candidate field set; The semantic feature representation is input into a dynamic cognitive network for semantic reasoning and mapping; Based on the output of the dynamic cognitive network, a field mapping decision result containing mapping confidence and semantic matching basis is generated.

[0011] The present invention is further configured such that generating the corresponding semantic feature representation includes: For each candidate field in the candidate field set, a multidimensional semantic perception vector is constructed using a preset multidimensional semantic representation generation method. The multidimensional semantic perception vector includes: lexical feature sub-vector, context feature sub-vector, value range feature sub-vector, platform feature sub-vector, and external knowledge feature sub-vector. By fusing multidimensional semantic perception vectors, semantic feature representations corresponding to candidate fields are generated.

[0012] The present invention is further configured such that the dynamic cognitive network is a graph structure network, wherein its nodes include: standard field nodes and semantic concept nodes, and the edges represent the semantic association strength between nodes.

[0013] The present invention is further configured such that inputting the semantic feature representation into a dynamic cognitive network for semantic reasoning and mapping includes: The semantic feature representation of the candidate fields is input into the dynamic cognitive network; When the semantic feature representation meets the preset matching conditions, a mapping relationship is established between candidate fields and standard field nodes; When the semantic feature representation does not meet the preset matching conditions, a temporary semantic concept node is created in the dynamic cognitive network, and the semantic association between it and the existing node is established and adjusted according to the preset update rules. Based on the adjusted semantic association, a network structure update suggestion is generated to merge fields or add standard field nodes. Based on the established mapping relationship or the generated network structure update suggestions, the output field mapping decision results are determined.

[0014] The present invention is further configured such that the output field mapping decision result includes: The field mapping decision results include: mapping confidence and semantic matching basis; The mapping confidence level includes: Using a pre-defined similarity evaluation method, the semantic feature representation of the candidate field and the semantic association strength between the mapped standard field node in the dynamic cognitive network are quantified, and the quantified semantic association strength is set as the mapping confidence. The semantic matching criteria include: By extracting the semantic association paths and corresponding association feature information between candidate field and standard field nodes in the dynamic cognitive network, the semantic association paths and corresponding association features are set as the basis for semantic matching.

[0015] The present invention is further configured such that S3 includes: Based on the field mapping decision results, identify multiple different field values ​​that are mapped to the same standard field; Based on preset priority rules, multiple different field values ​​are comprehensively evaluated and sorted to determine the preferred field value. The calculation basis of the priority rules includes at least two of the following: mapping confidence in the field mapping decision result, credibility weight of the data source platform, data freshness calculated based on the collection timestamp in the original data, and data completeness. Perform at least one standardization process based on the preferred field value to obtain a corrected set of aligned fields. The standardization process includes: converting time information into a standard time zone and format, converting numerical information into a unified unit of measurement, converting currency information into a base currency, or converting text information into a target language.

[0016] The present invention is further configured such that S4 includes: Based on a predefined standard field model, the values ​​of each field in the alignment field set are mapped to the corresponding standard fields in the model. Based on the mapping relationship in the standard field model, organize and generate a unified structured record containing standard field names and standard field values.

[0017] This invention provides a method for aligning fields based on structured parsing of overseas open-source information. The method comprises the following steps: S1: Obtaining the original data packet from an overseas open-source platform, performing structured parsing and field extraction to generate a candidate field set; S2: Based on a dynamic cognitive network, performing semantic recognition and field mapping on the fields in the candidate field set to generate a field mapping decision result; S3: Based on the field mapping decision result, resolving and correcting the consistency of conflicting field values ​​mapped to the same standard field to obtain an aligned field set; S4: Organizing the aligned field set into a unified structured record and outputting it according to the standard field model. The beneficial effects include: Achieving evolutionary semantic understanding: By introducing a dynamic cognitive network, the system can automatically process unknown or emerging field terms and continuously optimize its semantic knowledge base during operation, thereby improving the long-term adaptability and automation level of ever-changing open source information and overcoming the shortcomings of traditional methods that rely on manual rule updates and cannot adapt to new concepts.

[0018] Achieving deep cross-source field matching: Employing a semantic awareness method that integrates multi-dimensional features (lexical, contextual, value domain, platform, and external knowledge), it achieves a deep understanding and accurate mapping of field meanings, resolving semantic ambiguity issues caused by platform cultural differences and diverse expressions, resulting in higher alignment accuracy.

[0019] Provides interpretable and reliable intelligent decision-making results: The field mapping and conflict resolution results output throughout the process are accompanied by confidence levels and specific evidence, making the automated decision-making process transparent and auditable, and enhancing the reliability of downstream analysis applications and user trust.

[0020] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating an exemplary embodiment of the present invention, showing a method for aligning fields in structured parsing of information from overseas open-source sources. Detailed Implementation

[0022] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0023] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0024] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0025] Example: A method for aligning structured parsing fields based on overseas open-source information, such as... Figure 1 As shown, it includes: S1: Obtain the raw data packets from overseas open-source platforms, perform structured parsing and field extraction on the raw data packets, and generate a set of candidate fields; S2: Based on a dynamic cognitive network, semantic recognition and field mapping are performed on the fields in the candidate field set to generate field mapping decision results; S3: Based on the field mapping decision results, resolve and correct the consistency of conflicting field values ​​mapped to the same standard field to obtain an aligned field set; S4: Based on the standard field model, organize the aligned field set to generate a unified structured record and output it.

[0026] The present invention is further configured such that the original data packet includes: a platform identifier, a set of original field names and their corresponding original values, a collection timestamp, and heterogeneous data content. Specifically, the original data packet is a data set containing a platform identifier, a set of original field names and their corresponding original values, a collection timestamp, and heterogeneous data content. During construction, a Hypertext Transfer Protocol (HTTP) request is first initiated to the target Uniform Resource Locator (URL). The system configuration file has preset mapping rules between platforms and identifiers, for example: a social platform A, platform B; the system matches the URL of the requested URL with this mapping rule to determine the platform identifier. After receiving the response, for documents based on Hypertext Markup Language (HTML), a Document Object Model (DOM) parser is used for parsing. Elements containing specific HTML attributes or text patterns are found through predefined parsing rules, for example: finding page elements with the attribute `data-testid` value of `UserName`, combining the attribute name and value as the original field name, and the text within the element as the corresponding original value; for JSON format data returned by the application programming interface (API), a JSON parser is used for parsing, using JSON keys as original field names and JSON values ​​as the corresponding original values; all extracted field names and values ​​form the set of original field names and their corresponding original values. Upon successfully receiving the response, a system function is immediately invoked to generate the current Coordinated Universal Time (UTC) timestamp. The timestamp format follows the ISO 8601 standard, such as 2023-10-27T08:30:00Z. The heterogeneous data content is the received complete raw response body string, such as a complete Hypertext Markup Language (HTML) document string or a JSON response string. Finally, the platform identifier, the set of raw field names and their corresponding raw values, the collection timestamp, and the heterogeneous data content are encapsulated into a raw data packet.

[0027] The present invention is further configured such that S1 includes: For different types of heterogeneous data content in the original data packet, separate parsing and extraction are performed: For documents defined by markup language, locate and extract predefined page elements to obtain the first type of candidate fields and their corresponding values; The key-value pairs or nested hierarchical data returned by the application programming interface are parsed in a structured manner to obtain the second type of candidate fields and their corresponding values. For unstructured text content, named entity recognition and / or keyword extraction are used to obtain the third type of candidate fields and their corresponding values. Extract key descriptive information from the metadata of multimedia objects to obtain the fourth type of candidate fields and their corresponding values; The first, second, third, and fourth categories of candidate fields and their corresponding values ​​are integrated to generate a candidate field set containing field names and their corresponding values. Specifically, firstly, the original data packet is received; secondly, different types of heterogeneous data content in the original data packet are parsed and extracted separately. For documents based on markup language definitions, such as Hypertext Markup Language documents, the Document Object Model parser is used to load the document string and locate page elements according to pre-configured parsing rules. The parsing rules are described using Extensible Style Sheets Language (ESL) transformation path expressions or Cascading Style Sheets (CSS) selectors. For example, the rule: Use the ESL transformation path expression / / div[@class='profile'] / span[@class='name'] / text() to extract "user name". The field names defined in the rule and the extracted text are combined to form key-value pairs to obtain the first category of candidate fields and their corresponding values. For key-value pairs or nested hierarchical data returned by the application programming interface (API), such as JavaScript object representation data, the JavaScript object representation parser loads the data string and accesses the value of the specified path in the data structure according to a predefined field mapping table. The field mapping table defines the correspondence between standard field names and data paths. For example, the field "followers count" is mapped to the path ['public_metrics']['followers_count']. The parser retrieves the value according to this path and generates key-value pairs to obtain the second type of candidate fields and their corresponding values. For unstructured text content, such as user profile text, the first step is to perform word segmentation and stop word removal preprocessing. The stop word list used is derived from the English stop word list of the Natural Language Toolkit library. Then, named entity recognition or keyword extraction is performed. Named entity recognition is completed using a pre-trained natural language processing model, such as the en_core_web_trf model from the spaCy library. This model is based on the Transformer architecture and can identify entity types such as people, organizations, and locations in the text. The identified entity types and content are used as field names and values. Keyword extraction uses the graph-based TextRank algorithm. The default size of the co-occurrence window used in the algorithm is set to 5 words. The top 5 words with the highest weights are extracted as field values, with the field name preset as "text keywords". This step obtains the third type of candidate fields and their corresponding values. For multimedia object metadata, such as Joint Image Experts Group format images, the image processing library's Exchange Image File Format Tag module is used to read the metadata and extract key descriptive information based on a predefined tag name dictionary. The tag name dictionary defines the mapping between standard field names and exchange image file format tag keys. For example, the field "capture time" is mapped to the tag key DateTimeOriginal, and the field "capture latitude" is mapped to the tag key GPSLatitude, thus obtaining the fourth category of candidate fields and their corresponding values.Finally, the candidate fields and their corresponding values ​​from the first, second, third, and fourth categories are integrated. During integration, if duplicate field names are encountered, a source type suffix is ​​added for differentiation, such as "Username_html" and "Username_api". All key-value pairs are stored in a dictionary data structure to generate a candidate field set containing field names and their corresponding values. For example, after processing a raw data package containing a user homepage and profile text in Hypertext Markup Language, the final candidate field set might be {"Username_html": "Zhang San", "Followers_api": "1500", "Location_ner": "Beijing", "Text Keywords": "Researcher Artificial Intelligence"}.

[0028] The present invention is further configured such that S2 includes: Based on the candidate field set, generate corresponding semantic feature representations for each field in the candidate field set; The semantic feature representation is input into a dynamic cognitive network for semantic reasoning and mapping; Based on the output of the dynamic cognitive network, a field mapping decision result containing mapping confidence and semantic matching criteria is generated. Specifically, firstly, based on the candidate field set, a corresponding semantic feature representation is generated for each field in the candidate field set. In this step, the user converts the text name of the field into a multi-dimensional digital vector that integrates lexical, contextual, value range, platform features, and external knowledge, providing a unified and computable foundation for the machine to understand the deep semantics of the field. Next, the semantic feature representation is input into the dynamic cognitive network for semantic reasoning and mapping. In this step, the dynamic cognitive network acts as a graph-structured, evolvable knowledge base. By calculating the semantic association strength, it intelligently determines whether a field should be mapped to an existing standard field node or triggers an update process to create a new node, thereby achieving accurate semantic alignment and adaptive learning of cross-platform fields. Finally, based on the output of the dynamic cognitive network, a field mapping decision result containing mapping confidence and semantic matching criteria is generated. The mapping confidence quantifies the reliability of the semantic association, while the semantic matching criteria record the association path and feature contribution on which the decision is based, together constituting an interpretable and trustworthy final output.

[0029] The present invention is further configured such that generating the corresponding semantic feature representation includes: For each candidate field in the candidate field set, a multidimensional semantic perception vector is constructed using a preset multidimensional semantic representation generation method. The multidimensional semantic perception vector includes: lexical feature sub-vector, context feature sub-vector, value range feature sub-vector, platform feature sub-vector, and external knowledge feature sub-vector. By fusing multidimensional semantic-aware vectors, semantic feature representations corresponding to candidate fields are generated. Specifically, firstly, the candidate field set generated in step S1 is received. For each candidate field in the candidate field set, a pre-defined multidimensional semantic representation generation method is executed to construct a multidimensional semantic-aware vector. This multidimensional semantic-aware vector is composed of five predetermined feature sub-vectors concatenated in sequence: lexical feature sub-vector, context feature sub-vector, value range feature sub-vector, platform feature sub-vector, and external knowledge feature sub-vector. The lexical feature sub-vector encodes the textual features of the field name. The construction process is as follows: the field name string is lowercase and segmented using underscores; each segmented word is lemmatized using WordNetLemmatizer from the Natural Language Toolkit library; then, a pre-trained global vector word representation model is queried to obtain a 300-dimensional word vector for each lemmatized word; finally, average pooling is performed on all word vectors to output a 300-dimensional lexical feature sub-vector. The context feature sub-vector encoding field's co-occurrence relationship features are constructed as follows: Analyze the co-occurrence of the target field with other fields in the original data record, calculate the co-occurrence frequency, mark co-occurrence relationships with frequencies exceeding a default threshold of 50% as 1, otherwise as 0, forming a binary vector. At the same time, calculate the average cosine similarity between the target field name and other field names in the word vector space as a supplementary feature. After concatenating the binary vector with the similarity scalar, input it into a fully connected neural network containing a 64-dimensional hidden layer for processing, and output a 64-dimensional context feature sub-vector. The value range feature sub-vector encodes the distribution characteristics of the field values. The construction process is as follows: Analyze all sample values ​​of the field in the current batch of data to determine its dominant data type. If more than 95% of the samples can be parsed as integers or floating-point numbers, it is determined to be a numeric type; otherwise, it is determined to be a text type. This determination result is encoded as a 2D one-hot vector. For numeric types, calculate the minimum, maximum, average, and standard deviation to form a 4D vector. For text types, calculate the average character length and the ratio of unique values ​​to form a 2D vector. After concatenating the type one-hot vector with the statistical vector, map it to 32 dimensions through a linear transformation layer to output the value range feature sub-vector. The platform feature sub-vector encodes the characteristics of the data source platform. The construction process is as follows: Based on the platform identifier associated with the candidate field, query a pre-trained platform embedding lookup table. This table assigns a fixed 16-dimensional dense vector to each known platform. For example, assigning a 16-dimensional vector representing the characteristics of platform A to the platform identifier "A" directly obtains this vector as the platform feature sub-vector.The external knowledge feature sub-vector encoding field represents the conceptual features of a general knowledge context. The construction process is as follows: Using the field name as the query term, the ConceptNet knowledge graph's API is called to obtain a list of directly connected parent concepts. Then, the pre-trained concept embedding model built into the knowledge graph is used to convert each concept into a 100-dimensional vector. These concept vectors are then simply averaged to output a 100-dimensional external knowledge feature sub-vector. After generating these five sub-vectors sequentially, a fusion step is performed: they are concatenated into a 512-dimensional intermediate vector in a fixed order: lexical (300-dimensional), context (64-dimensional), value range (32-dimensional), platform (16-dimensional), and external knowledge (100-dimensional). This 512-dimensional intermediate vector is then input into a multilayer perceptron for fusion and dimensionality reduction. This multilayer perceptron has a 256-dimensional hidden layer and uses a linear rectified function as the activation function, ultimately outputting a 256-dimensional dense vector. This 256-dimensional dense vector is the final semantic feature representation corresponding to the candidate field. For example, for the candidate field "Follower_Num", after the above complete process, a 256-dimensional numeric vector representing its comprehensive semantics will be obtained.

[0030] The present invention further specifies that the dynamic cognitive network is a graph structure network, whose nodes include standard field nodes and semantic concept nodes, and edges represent the semantic association strength between nodes. Specifically, the dynamic cognitive network adopts a graph structure network as its basic framework to carry and organize semantic knowledge. The nodes of the dynamic cognitive network include two types: standard field nodes and semantic concept nodes. Standard field nodes represent unified target fields defined and maintained by the system; semantic concept nodes serve as an intermediate semantic layer, used to abstract and aggregate original fields with similar semantics from different data sources. The edges of the dynamic cognitive network are used to connect nodes, and each edge represents the semantic association strength between the two connected nodes with a quantified value. The graph structure framework makes semantic relationships computable and inferable. The division between standard field nodes and semantic concept nodes realizes the hierarchical representation of knowledge, and the semantic association strength parameter of the edges enables the entire network to adjust weights and evolve knowledge through data feedback, jointly supporting deep semantic understanding and adaptive alignment across source fields.

[0031] The present invention is further configured such that inputting the semantic feature representation into a dynamic cognitive network for semantic reasoning and mapping includes: The semantic feature representation of the candidate fields is input into the dynamic cognitive network; When the semantic feature representation meets the preset matching conditions, a mapping relationship is established between candidate fields and standard field nodes; When the semantic feature representation does not meet the preset matching conditions, a temporary semantic concept node is created in the dynamic cognitive network, and the semantic association between it and the existing node is established and adjusted according to the preset update rules. Based on the adjusted semantic association, a network structure update suggestion is generated to merge fields or add standard field nodes. Based on the established mapping relationship or the generated network structure update suggestions, the field mapping decision result is output. Specifically, firstly, the semantic feature representation vector of the candidate field is input into the dynamic cognitive network. The dynamic cognitive network accesses all standard field node vectors stored internally and uses the cosine similarity algorithm to calculate the similarity score between the input vector and each standard field node vector, finding the highest similarity score and its corresponding standard field node. The preset matching condition is set as follows: the highest similarity score must be greater than or equal to 0.85. If the highest similarity score meets the preset matching condition, a direct mapping relationship is established between the candidate field and this standard field node; simultaneously, this highest similarity score is recorded as the base value of the mapping confidence, and the composition of the input vector is analyzed to identify the feature sub-vector category with the highest contribution as part of the semantic matching basis. Then, the process jumps to the output step. If the highest similarity score does not meet the preset matching condition, i.e., the score is less than 0.85, the process enters the temporary node processing flow. The dynamic cognitive network creates a new temporary semantic concept node, which has a unique identifier and stores the semantic feature representation vector of the current candidate field in this node. Next, the semantic associations between temporary semantic concept nodes and existing nodes are initialized according to preset update rules. The preset update rules stipulate that newly created semantic association edges have an initial strength value of 0.3. During initialization, the system selects the top three existing nodes with the highest similarity scores to the input vector. These nodes can be standard field nodes or existing semantic concept nodes. Directed edges are created from the newly created temporary semantic concept nodes to these three nodes, and the semantic association strength of each edge is set to 0.3. After the temporary semantic concept nodes are created, the system enters a long-term observation and adjustment phase. When processing other batches of data, if the same original field name from the same platform identifier is encountered again, and the semantic feature representation vector of that field is also routed to the same temporary semantic concept node, the association strength adjustment is triggered. The preset update rules stipulate that each time it recurs, the semantic association strength of the edge between the temporary semantic concept node and the existing node with the highest similarity in this calculation increases by 0.1. Furthermore, if the field value represented by this temporary node is adopted as the final preferred value in subsequent conflict resolution steps, the semantic association strength of the relevant edge increases by an additional 0.2. The system continuously monitors the strength of all associated edges of temporary semantic concept nodes. When the semantic association strength value of an associated edge reaches or exceeds the preset stability threshold of 0.7, the system automatically generates a network structure update suggestion. If this strong association edge points to an existing standard field node, a "field merging suggestion" is generated, recommending that such original fields be formally and permanently mapped to the standard field node.If a temporary semantic concept node exhibits a unique pattern, with none of its associated edges reaching 0.7, but its association strength with multiple existing nodes continues to grow slowly and continuously, and it is not strongly bound to any existing standard field node, the system may generate a "suggestion for adding a new standard field node" after administrator review. Finally, based on the above process, the field mapping decision result is output. For direct mapping, the field mapping decision result includes the determined target standard field name, mapping confidence, and semantic matching basis. For cases processed by temporary nodes, the field mapping decision result includes the temporary semantic concept node identifier, the generated network structure update suggestion, the mapping confidence based on the current strongest associated edge strength, and the associated path as the semantic matching basis. An exemplary complete calculation process is as follows: Assume the input field is the original field "User_Tagline" of the platform identifier "B", and its semantic feature representation vector has been calculated. Step 1: Calculate the cosine similarity between this vector and all standard field node vectors, obtaining a highest score of 0.75 (corresponding to the "account_description" node), which is lower than 0.85. Step 2: Create a temporary semantic concept node "Temp_Node_B_Tagline" and store the vector. Step 3: Initialize the edges from this node to the three nodes "account_description", "signature", and "bio", all with a strength of 0.3. Step 4: In subsequent data, the "User_Tagline" field is processed four more times from platform "B". Three of these processing times show the highest similarity to the "account_description" node, and one time shows the highest similarity to the "bio" node. According to the rules, the edge strength to "account_description" increases three times (3 * 0.1), from 0.3 to 0.6; the edge strength to "bio" increases once (0.1), to 0.4. Assuming that the field value is adopted in conflict resolution on the fifth occurrence, the edge strength to "account_description" increases by another 0.2, reaching 0.8, exceeding the stable threshold of 0.7. Step 5: The system automatically generates a "field merging suggestion": It is recommended to permanently map the "User_Tagline" field from platform "B" to the standard field "account_description". Step 6: For this input, output the field mapping decision results, including: the original field name "User_Tagline", the suggested mapping to "account_description", the mapping confidence score of 0.8, the semantic matching basis is the association path (from the ephemeral node to the "account_description" node), and the network structure update suggestion.

[0032] The present invention is further configured such that the output field mapping decision result includes: The field mapping decision results include: mapping confidence and semantic matching basis; The mapping confidence level includes: Using a pre-defined similarity evaluation method, the semantic feature representation of the candidate field and the semantic association strength between the mapped standard field node in the dynamic cognitive network are quantified, and the quantified semantic association strength is set as the mapping confidence. The semantic matching criteria include: By extracting the semantic association paths and corresponding association features between candidate fields and standard field nodes in the dynamic cognitive network, these semantic association paths and corresponding association features are set as the semantic matching basis. Specifically, firstly, intermediate results from the "semantic reasoning and mapping" step are received, with the goal of assembling and outputting a structured field mapping decision result. This result contains two core parts: mapping confidence and semantic matching basis. Mapping confidence is a value between 0 and 1, used to quantify the degree of certainty in mapping a candidate field to a specific standard field; a higher value indicates stronger certainty. Semantic matching basis is a structured explanatory information section used to explain the reasoning process and key evidence leading to the current mapping conclusion. The process of generating the mapping confidence score is as follows: The input consists of the semantic feature representation vector of the candidate field and the vector of the target standard field node. Both vectors have the same dimension, such as 256 dimensions. The similarity between the two vectors is calculated using a preset similarity evaluation method. In this embodiment, cosine similarity is specified as the preset similarity evaluation method. The cosine similarity calculation produces an original score between -1 and 1. Subsequently, this original score is normalized and linearly transformed to the interval between 0 and 1. The transformation formula is: Mapping confidence score = (cosine similarity original score + 1) / 2. The calculated value is the final mapping confidence score. The process of generating semantic matching criteria consists of two sub-steps. The first sub-step is to extract semantic association paths. A semantic association path refers to a sequence of nodes and edges connecting the source node and the target standard field node in the graph structure of the dynamic cognitive network. The source node may be a standard field node with a direct mapping relationship or a temporary semantic concept node created for an unknown field. During extraction, a graph traversal algorithm, such as depth-first search or breadth-first search, is used to find all reachable paths from the source node to the target node in the network. To avoid invalid searches, the system presets a maximum search depth of 5 hops, meaning the number of nodes on the path does not exceed 6. From all the found paths, one is selected as the optimal semantic association path according to preset rules. The preset rules may be to select the path with the fewest edges, or, when the edge weights are known, to select the path with the largest product of the semantic association strengths of all edges on the path. Then, the sequence of node identifiers on this optimal path is recorded. The second sub-step is to analyze the association feature information. The association feature information aims to reveal which specific dimensions of the candidate field semantic features play a key role in the final similarity matching.In implementation, the complete semantic feature representation vector is first decomposed into five feature sub-vector segments according to its composition: lexical feature sub-vector segment, context feature sub-vector segment, value range feature sub-vector segment, platform feature sub-vector segment, and external knowledge feature sub-vector segment. Then, the similarity between the candidate field and the target standard field node on these five sub-vector segments is calculated. This can be done by calculating the reciprocal of the Euclidean distance between the corresponding dimensional slice vectors or by directly calculating the dot product. Next, the one or two feature sub-vector categories with the highest contribution are identified and output. The criteria for judging the contribution are: the independent similarity value of the sub-vector segment is significantly higher than the average similarity of the overall vector, or the value change of the sub-vector segment has the greatest impact on the overall similarity calculation result. The identified high-contribution feature categories (such as "external knowledge features") are recorded. Finally, assemble the field mapping decision results: Create a structured data object, such as one conforming to JavaScript object notation. Within this object, establish a field named "confidence," whose value is the previously calculated mapping confidence score. Establish a field named "rationale" to store the semantic matching criteria. This field is itself an object containing two subfields: "matching_path" and "key_features." The "matching_path" subfield contains a list of node identifiers for the extracted optimal semantic association path; the "key_features" subfield contains textual descriptions of the key association features obtained from the analysis. This complete data object constitutes the final output field mapping decision result.

[0033] The present invention is further configured such that S3 includes: Based on the field mapping decision results, identify multiple different field values ​​that are mapped to the same standard field; Based on preset priority rules, multiple different field values ​​are comprehensively evaluated and sorted to determine the preferred field value. The calculation basis of the priority rules includes at least two of the following: mapping confidence in the field mapping decision result, credibility weight of the data source platform, data freshness calculated based on the collection timestamp in the original data, and data completeness. Based on the preferred field values, at least one standardization process is performed to obtain a corrected set of aligned fields. The standardization process includes: converting time information to a standard time zone and format, converting numerical information to a unified unit of measurement, converting currency information to a base currency, or converting text information to a target language. Specifically, firstly, the system identifies conflicts based on the field mapping decision results: it iterates through all field mapping decision results, groups them according to standard field names, and for the same standard field name, if it finds that it is associated with original field values ​​from different data source platforms or multiple collection time points with different specific content, these values ​​are identified as a set of conflicting field values; for example, the standard field "follower_count" may collect values ​​"1.2K", "1243", and "1200". Next, the conflicting field values ​​are comprehensively evaluated and sorted based on the preset priority rules to determine the preferred field value. The preset priority rules require the calculation of a comprehensive priority score for each conflicting field value. The calculation basis of this score must include at least two of the following four items: the mapping confidence in the field mapping decision result, the credibility weight of the data source platform, the data freshness calculated based on the collection timestamp in the original data, and the data completeness. In this embodiment, all four items are used to calculate the comprehensive priority score by weighted summation. The mapping confidence score is directly derived from the confidence value in the field mapping decision result. The credibility weight of the data source platform is obtained by querying a preset mapping table, which assigns a static weight value to each known platform identifier. For example, the weight of platform "A" is 0.9, and the weight of platform "B" is 0.8. Data freshness is calculated by calculating the difference between the collection timestamp and the current system time (converted to days) and applying an exponential decay function. The preset decay coefficient is 0.05, and the freshness score is calculated by multiplying the negative decay coefficient of the natural constant e by the difference in days raised to the power of the difference. Data completeness is evaluated based on the field value type: for text types, the ratio of the string length to a preset typical length threshold (upper limit is 1.0) is calculated; for numeric or date types, the format of the numbers is checked for standardization or the date portion is complete. After normalizing each score to the range of 0 to 1, a weighted sum is performed according to preset weights. In this embodiment, the default weight for each item is 0.25. After calculating the comprehensive priority score of all conflicting field values, they are sorted in descending order of score, and the field value corresponding to the highest score is determined as the preferred field value.Then, at least one standardization process is performed based on the preferred field values. This standardization process includes the following types: converting time information to a standard time zone and format; using a date and time parsing library to parse various input formats and convert them to Coordinated Universal Time (UTC) and format them as ISO 8601 standard strings; converting numerical information to a unified unit of measurement; identifying unit symbols (such as K and M) in the string and querying a preset unit conversion dictionary for conversion (e.g., "K" corresponds to 1000); converting currency information to a base currency; identifying currency codes and querying an exchange rate table to convert to a preset base currency (such as the US dollar); and converting text information to the target language; using a language detection library to detect the language and calling a machine translation application programming interface to translate it to the target language (such as English). The standardized value is the correction value. Finally, an alignment field set is generated: after completing the above evaluation and correction for each conflicting standard field, a key-value pair is formed by combining a standard field name with its corresponding correction value. All such key-value pairs are integrated into a data structure (e.g., a dictionary), which is the final output alignment field set. For example, after all conflicts have been resolved, the alignment field set may contain entries such as {"follower_count":"1200","location":"Beijing,China"}.

[0034] The present invention is further configured such that S4 includes: Based on a predefined standard field model, the values ​​of each field in the alignment field set are mapped to the corresponding standard fields in the model. Based on the mapping relationships in the standard field model, a unified structured record containing standard field names and values ​​is organized and generated. Specifically, first, the system loads the predefined standard field model from persistent storage. The standard field model is a structured specification that defines the attributes of all standard fields ultimately output by the system. The model content is represented as a JSON array, where each element is an object describing a standard field. Each object must contain two keys: "field_name" and "data_type". For example, a standard field model definition could be [{"field_name":"account_name","data_type":"STRING"},{"field_name":"follower_count","data_type":"INTEGER"},{"field_name":"account_created_at","data_type":"DATETIME"}]. The system uses a JSON parsing library to read and parse this model data into memory. Next, mapping and organization operations are performed. The system iterates through each field definition object in the standard field model array in memory. For the currently traversed field definition object, the system extracts the string value corresponding to the "field_name" key as the current standard field name. Then, it uses this current standard field name as the lookup key to search for the corresponding field value in the input aligned field set dictionary. The aligned field set is a data structure output from step S3 above, with standard field names as keys and corrected strings as values. If a key matching the current standard field name is found in the aligned field set, the corresponding field value string is retrieved. Based on the data type requirements specified by the "data_type" key in the current field definition object, a forced type conversion is performed on this field value string: if "data_type" is "STRING", the string remains unchanged; if it is "INTEGER", the built-in integer conversion function of the programming language is called to convert the string to an integer; if it is "FLOAT", it is converted to a floating-point number; if it is "DATETIME", it remains an ISO 8601 format string or is converted to a specific date and time object. If the type conversion fails, the system records the error and sets the field value to null. If no matching key is found in the alignment field set, a null value is assigned to the current standard field name according to preset rules. After transformation or assignment, the system combines the current standard field name and the processed field value into a key-value pair and stores it in a newly created empty dictionary, called the record container. After traversing all field definition objects in the standard field model, the record container contains a complete mapping of all standard field names and their final values. Finally, a unified structured record is generated and output.The system converts the record container dictionary into a standardized JSON string using a JSON serialization library; this process is called generating a unified structured record. For example, a possible output is {"account_name":"TechExplorer","follower_count":1250,"account_created_at":"2020-05-21T08:30:00Z"}. The generated JSON string is output as a unified structured record, which can be written to a text file, sent to a message queue, or stored in a database.

[0035] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for aligning fields in structured parsing of overseas open-source information, characterized in that, include: S1: Obtain the raw data packets from overseas open-source platforms, perform structured parsing and field extraction on the raw data packets, and generate a set of candidate fields; S2: Based on a dynamic cognitive network, semantic recognition and field mapping are performed on the fields in the candidate field set to generate field mapping decision results; S3: Based on the field mapping decision results, resolve and correct the consistency of conflicting field values ​​mapped to the same standard field to obtain an aligned field set; S4: Based on the standard field model, organize the aligned field set to generate a unified structured record and output it.

2. The method for aligning fields in structured parsing of overseas open-source information according to claim 1, characterized in that, The original data packet includes: platform identifier, original field name set and corresponding original values, collection timestamp, and heterogeneous data content.

3. The method for aligning fields in structured parsing of overseas open-source information according to claim 2, characterized in that, S1 includes: For different types of heterogeneous data content in the original data packet, separate parsing and extraction are performed: For documents defined by markup language, locate and extract predefined page elements to obtain the first type of candidate fields and their corresponding values; The key-value pairs or nested hierarchical data returned by the application programming interface are parsed in a structured manner to obtain the second type of candidate fields and their corresponding values. For unstructured text content, named entity recognition and / or keyword extraction are used to obtain the third type of candidate fields and their corresponding values. Extract key descriptive information from the metadata of multimedia objects to obtain the fourth type of candidate fields and their corresponding values; Integrate the candidate fields and their corresponding values ​​from the first, second, third, and fourth categories to generate a candidate field set containing field names and their corresponding values.

4. The method for aligning fields in structured parsing of overseas open-source information according to claim 1, characterized in that, S2 includes: Based on the candidate field set, generate corresponding semantic feature representations for each field in the candidate field set; The semantic feature representation is input into a dynamic cognitive network for semantic reasoning and mapping; Based on the output of the dynamic cognitive network, a field mapping decision result containing mapping confidence and semantic matching basis is generated.

5. The method for aligning fields in structured parsing of overseas open-source information according to claim 1, characterized in that, The generation of the corresponding semantic feature representation includes: For each candidate field in the candidate field set, a multidimensional semantic perception vector is constructed using a preset multidimensional semantic representation generation method. The multidimensional semantic perception vector includes: lexical feature sub-vector, context feature sub-vector, value range feature sub-vector, platform feature sub-vector, and external knowledge feature sub-vector. By fusing multidimensional semantic perception vectors, semantic feature representations corresponding to candidate fields are generated.

6. The method for aligning fields in structured parsing of overseas open-source information according to claim 5, characterized in that, The dynamic cognitive network is a graph structure network, whose nodes include standard field nodes and semantic concept nodes, and edges represent the semantic association strength between nodes.

7. A method for aligning fields in structured parsing of overseas open-source information according to claim 6, characterized in that, The step of inputting semantic feature representations into a dynamic cognitive network for semantic reasoning and mapping includes: The semantic feature representation of the candidate fields is input into the dynamic cognitive network; When the semantic feature representation meets the preset matching conditions, a mapping relationship is established between candidate fields and standard field nodes; When the semantic feature representation does not meet the preset matching conditions, a temporary semantic concept node is created in the dynamic cognitive network, and the semantic association between it and the existing node is established and adjusted according to the preset update rules. Based on the adjusted semantic association, a network structure update suggestion is generated to merge fields or add standard field nodes. Based on the established mapping relationship or the generated network structure update suggestions, the output field mapping decision results are determined.

8. A method for aligning fields in structured parsing of overseas open-source information according to claim 7, characterized in that, The output field mapping decision results include: The field mapping decision results include: mapping confidence and semantic matching basis; The mapping confidence level includes: Using a pre-defined similarity evaluation method, the semantic feature representation of the candidate field and the semantic association strength between the mapped standard field node in the dynamic cognitive network are quantified, and the quantified semantic association strength is set as the mapping confidence. The semantic matching criteria include: By extracting the semantic association paths and corresponding association feature information between candidate field and standard field nodes in the dynamic cognitive network, the semantic association paths and corresponding association features are set as the basis for semantic matching.

9. A method for aligning fields in structured parsing of overseas open-source information according to claim 1, characterized in that, S3 includes: Based on the field mapping decision results, identify multiple different field values ​​that are mapped to the same standard field; Based on preset priority rules, multiple different field values ​​are comprehensively evaluated and sorted to determine the preferred field value. The calculation basis of the priority rules includes at least two of the following: mapping confidence in the field mapping decision result, credibility weight of the data source platform, data freshness calculated based on the collection timestamp in the original data, and data completeness. Perform at least one standardization process based on the preferred field value to obtain a corrected set of aligned fields. The standardization process includes: converting time information into a standard time zone and format, converting numerical information into a unified unit of measurement, converting currency information into a base currency, or converting text information into a target language.

10. A method for aligning fields in structured parsing of overseas open-source information according to claim 1, characterized in that, S4 includes: Based on a predefined standard field model, the values ​​of each field in the alignment field set are mapped to the corresponding standard fields in the model. Based on the mapping relationship in the standard field model, organize and generate a unified structured record containing standard field names and standard field values.

Images