Semantic alignment method and device for heterogeneous data and computer program product

Abstract

The invention discloses a semantic alignment method and device for heterogeneous data and a computer program product, and the method comprises the steps: firstly carrying out semantic enhancement processing on a heterogeneous data field, and obtaining enhanced field representation information; and then coding the first heterogeneous data field and the second heterogeneous data field which belong to different data sources respectively by utilizing a pre-constructed domain language coding model with the capability of accurately understanding specific domain specific terms and business contexts to obtain semantic vectors of the first heterogeneous data field and the second heterogeneous data field so as to capture subtle semantic differences between the first heterogeneous data field and the second heterogeneous data field. Then, after the semantic similarity of the first heterogeneous data field and the second heterogeneous data field is calculated according to the semantic vectors of the first heterogeneous data field and the second heterogeneous data field, a target interpretation template is obtained through matching from an interpretation template library in combination with enhanced field representation information; the method and the device are used for generating a semantic alignment result and an interpretation of a first heterogeneous data field and a second heterogeneous data field, so that the semantic alignment efficiency and accuracy of the heterogeneous data fields can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a semantic alignment method, device, and computer program product for heterogeneous data. Background Technology

[0002] With the advent of the information age, enterprises and research institutions have accumulated massive amounts of data resources in their daily operations and decision-making processes. These data are not only vast in quantity but also diverse in form, including structured data (such as database records), semi-structured data (such as log files and XML documents), and unstructured data (such as text, images, and audio). Furthermore, this heterogeneous data is often distributed across multiple heterogeneous data systems, which differ significantly in data models, formats, and semantics, leading to a severe "data silo" phenomenon. Therefore, achieving semantic alignment of heterogeneous data to solve the "data silo" problem caused by differences in structure, format, and terminology among multi-source heterogeneous data is particularly important, and it lays the foundation for effective data integration and deep knowledge discovery.

[0003] Currently, existing methods for achieving semantic alignment of heterogeneous data typically fall into three categories: The first is rule-based mapping, but this method relies on predefined mapping rules (e.g., "customer name" corresponds to "customer_name"), requiring experts to write numerous transformation rules. This results in poor flexibility and high maintenance costs, making it difficult to adapt to new data sources or heterogeneous data with semantic changes. The second is independent machine learning-based alignment, but this method uses traditional natural language processing models (e.g., Word2Vec, BERT) to calculate the semantic similarity of field names, without considering the influence of domain-specific knowledge, leading to low alignment accuracy. The third is knowledge graph-based alignment, but this method requires building a semantic network to support alignment processing, resulting in high construction costs and reliance on prior knowledge. It is evident that all three existing methods for achieving semantic alignment of heterogeneous data have significant problems and struggle to achieve ideal alignment results. Summary of the Invention

[0004] The main objective of this application is to provide a semantic alignment method, device, and computer program product for heterogeneous data, which can improve the semantic alignment efficiency and accuracy of multi-source heterogeneous data to achieve the desired alignment effect.

[0005] This application provides a semantic alignment method for heterogeneous data, including:

[0006] Acquire multi-source heterogeneous data, and perform semantic enhancement processing on the heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information;

[0007] A first heterogeneous data field and a second heterogeneous data field are selected from the multi-source heterogeneous data; the first heterogeneous data field and its business definition and the second heterogeneous data field and their business definition are input into a pre-built domain language encoding model for encoding processing to obtain the semantic vector of the first heterogeneous data field and the semantic vector of the second heterogeneous data field; the first heterogeneous data field and the second heterogeneous data field belong to heterogeneous data from different data sources.

[0008] Based on the semantic vector of the first heterogeneous data field and the semantic vector of the second heterogeneous data field, calculate the semantic similarity between the first heterogeneous data field and the second heterogeneous data field;

[0009] Based on the semantic similarity between the first heterogeneous data field and the second heterogeneous data field, as well as the enhanced field representation information, a target explanation template is obtained from a pre-built explanation template library; and using the target explanation template, the semantic alignment results of the first heterogeneous data field and the second heterogeneous data field and their explanations are generated.

[0010] In one possible implementation, the semantic enhancement processing of heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information includes:

[0011] Perform upstream and downstream process field correlation analysis on the heterogeneous data fields in the multi-source heterogeneous data to obtain the first analysis result;

[0012] Based on the data dictionary associated with the multi-source heterogeneous data, the association information of the heterogeneous data fields in the multi-source heterogeneous data is extracted to obtain the extraction result;

[0013] The heterogeneous data fields in the multi-source heterogeneous data are analyzed using usage logs to obtain a second analysis result.

[0014] Using the first analysis result, the extraction result, and the second analysis result, semantic enhancement processing is performed on the heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information.

[0015] In one possible implementation, the enhanced field characterization information includes at least one of a field co-occurrence frequency statistics table and a list of strongly correlated field pairs.

[0016] In one possible implementation, the domain language encoding model is constructed as follows:

[0017] Construct heterogeneous training sample data for the target domain, wherein the heterogeneous training sample data includes positive sample heterogeneous data pairs and negative sample heterogeneous data pairs;

[0018] The heterogeneous data fields and their business definitions in the positive sample heterogeneous data pairs are input into the initial domain language encoding model, and the heterogeneous data fields and their business definitions in the negative sample heterogeneous data pairs are input into the initial domain language encoding model. By adjusting the loss function, the domain language encoding model is trained to obtain the model.

[0019] In one possible implementation, the loss function is a contrastive loss function; the loss function is used to narrow the distance between positive sample heterogeneous data pairs in the vector space and to widen the distance between negative sample heterogeneous data pairs in the vector space.

[0020] In one possible implementation, the construction of heterogeneous training sample data for the target domain includes:

[0021] Obtain unlabeled metadata of the target domain; and based on at least one preset rule among text similarity rules, data type consistency rules, and business rules, filter out field pairs that meet preset conditions from the unlabeled metadata of the target domain to form positive sample heterogeneous data pairs and negative sample heterogeneous data pairs.

[0022] In one possible implementation, the construction of heterogeneous training sample data for the target domain includes:

[0023] Semantic enhancement processing is performed on the unlabeled metadata of the target domain to obtain sample enhancement field representation information;

[0024] The field pairs that meet the preset conditions and are selected from the unlabeled metadata of the target domain are fused with the sample enhancement field representation information to form enhanced positive sample heterogeneous data pairs and negative sample heterogeneous data pairs.

[0025] In one possible implementation, the interpretation template library includes a correspondence between text semantic rules and interpretation templates, a correspondence between business logic rules and interpretation templates, and a correspondence between context association rules and interpretation templates; the step of matching the target interpretation template from the pre-built interpretation template library based on the semantic similarity of the first heterogeneous data field and the second heterogeneous data field and the enhanced field representation information includes:

[0026] Determine whether the semantic similarity between the first heterogeneous data field and the second heterogeneous data field, as well as the enhanced field representation information, satisfy the triggering condition of at least one of the text semantic rules, business logic rules, and context association rules;

[0027] If so, the interpretation templates corresponding to at least one rule to which the triggering condition is satisfied will be combined to obtain the target interpretation template.

[0028] This application embodiment also provides a semantic alignment device for heterogeneous data, including: a processor, a memory, and a system bus;

[0029] The processor and the memory are connected via the system bus;

[0030] The memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the above-described implementations of the semantic alignment methods for heterogeneous data.

[0031] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute any one of the above-described methods for semantic alignment of heterogeneous data.

[0032] This application provides a method, device, and computer program product for semantic alignment of heterogeneous data. First, multi-source heterogeneous data is acquired, and semantic enhancement processing is performed on the heterogeneous data fields within the multi-source heterogeneous data to obtain enhanced field representation information. Then, a first heterogeneous data field and a second heterogeneous data field, belonging to different data sources, are selected from the multi-source heterogeneous data. The first heterogeneous data field and its business definition, and the second heterogeneous data field and its business definition, are input into a pre-built domain language encoding model for encoding processing to obtain semantic vectors for the first and second heterogeneous data fields. Next, based on the semantic vectors of the first and second heterogeneous data fields, the semantic similarity between the first and second heterogeneous data fields is calculated. Then, based on the semantic similarity between the first and second heterogeneous data fields and the enhanced field representation information, a target interpretation template is matched from a pre-built interpretation template library. Finally, using this target interpretation template, the semantic alignment result of the first and second heterogeneous data fields and its explanation are generated.

[0033] As can be seen, this application first performs semantic enhancement processing on the heterogeneous data fields to obtain enhanced field representation information when performing semantic alignment of heterogeneous data. This constructs rich field semantic representations for subsequent alignment processing, which is more conducive to improving alignment accuracy compared to existing plain text matching methods. Based on this, this application further utilizes a pre-built domain language encoding model with the ability to accurately understand specific domain-specific terms and business contexts to encode the first and second heterogeneous data fields, selected from multi-source heterogeneous data and belonging to different data sources, to obtain their respective semantic vectors. This allows for faster and more accurate capture of subtle semantic differences between the two. Next, after calculating the semantic similarity between the first and second heterogeneous data fields based on their semantic vectors, the enhanced field representation information can be combined with the target interpretation template from the interpretation template library to generate the semantic alignment results and explanations for the first and second heterogeneous data fields. Compared with existing heterogeneous data semantic alignment methods, this method can effectively improve the efficiency and accuracy of semantic alignment between every two heterogeneous data fields in multi-source heterogeneous data, and adds explanations of the alignment results, making the alignment process more transparent and facilitating subsequent manual review or integrated retrieval, thereby achieving the ideal alignment effect. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating a semantic alignment method for heterogeneous data provided in an embodiment of this application;

[0036] Figure 2 A schematic diagram illustrating the overall process of semantic alignment of heterogeneous data provided in the embodiments of this application;

[0037] Figure 3 This is a schematic diagram of the structure of a semantic alignment device for heterogeneous data provided in an embodiment of this application. Detailed Implementation

[0038] As enterprises accelerate their digital transformation, organizations often have multiple heterogeneous data systems with significant differences in data models, formats, and semantics, leading to severe "data silos." This heterogeneity exists not only in structured data (such as database records) but also widely in semi-structured data (such as JSON / XML logs) and unstructured data (such as text reports, images, and videos), making cross-system data fusion and joint analysis exceptionally complex and difficult. Therefore, achieving semantic alignment of heterogeneous data to address the "data silo" problem caused by differences in structure, format, and terminology among multi-source heterogeneous data is crucial, laying the foundation for effective data integration and deep knowledge discovery.

[0039] Currently, existing methods for achieving semantic alignment of heterogeneous data can generally be categorized into the following three types:

[0040] (1) Alignment method based on rule mapping.

[0041] This method heavily relies on manually predefined mapping rules (i.e., the process of establishing the correspondence between data fields between the source and target data systems, including industry-specific mappings and semantic unification of multiple fields) and cumbersome ETL processes (i.e., a standardized process of extracting data from various data sources, cleaning and transforming it, and finally loading it into the target system). It requires extensive manual configuration by domain experts, resulting in low efficiency and difficulty adapting to dynamically changing data environments. When faced with new data sources or changes in business semantics, this method often requires rewriting the mapping rules, lacking adaptability and scalability.

[0042] (2) Alignment method based on independent machine learning.

[0043] This method uses traditional natural language processing models (such as Word2Vec, BERT, etc.) to calculate the semantic similarity of field names, without considering the influence of domain-specific knowledge, which leads to low alignment accuracy.

[0044] (3) Alignment method based on knowledge graph.

[0045] This method requires the construction of a semantic network to support subsequent semantic alignment processing, which is costly and relies on prior knowledge, making it difficult to guarantee the accuracy of the alignment results.

[0046] As can be seen, the existing three methods for semantic alignment of heterogeneous data can be summarized into at least four main drawbacks: First, weak domain adaptability, specifically manifested in the inability of general models to capture the semantics of specific terms, leading to incorrect field mapping. Second, lack of context awareness, specifically manifested in the failure to consider the upstream and downstream relationships and usage scenarios of fields in the data flow, matching only based on the textual similarity of field names, ignoring business logic relevance. Third, limited automation, specifically manifested in the need for significant manual intervention in configuring mapping rules or labeling training data, making it difficult to support the needs of large-scale data platforms. Fourth, insufficient interpretability, specifically manifested in the lack of explanations for the matching rationale behind the semantic alignment results of heterogeneous data. Therefore, it is difficult to achieve the desired alignment effect.

[0047] To address the aforementioned shortcomings, this application provides a semantic alignment method for heterogeneous data. First, multi-source heterogeneous data is acquired, and semantic enhancement processing is performed on the heterogeneous data fields within this data to obtain enhanced field representation information. Then, a first heterogeneous data field and a second heterogeneous data field, belonging to different data sources, are selected from the multi-source heterogeneous data. The first heterogeneous data field and its business definition, along with the second heterogeneous data field and its business definition, are input into a pre-constructed domain language encoding model for encoding processing, resulting in semantic vectors for the first and second heterogeneous data fields. Next, based on the semantic vectors of the first and second heterogeneous data fields, their semantic similarity is calculated. Then, based on the semantic similarity and enhanced field representation information, a target interpretation template is matched from a pre-constructed interpretation template library. Finally, using this target interpretation template, the semantic alignment result of the first and second heterogeneous data fields and its explanation are generated.

[0048] As can be seen, this application first performs semantic enhancement processing on the heterogeneous data fields to obtain enhanced field representation information when performing semantic alignment of heterogeneous data. This constructs rich field semantic representations for subsequent alignment processing, which is more conducive to improving alignment accuracy compared to existing plain text matching methods. Based on this, this application further utilizes a pre-built domain language encoding model with the ability to accurately understand specific domain-specific terms and business contexts to encode the first and second heterogeneous data fields, selected from multi-source heterogeneous data and belonging to different data sources, to obtain their respective semantic vectors. This allows for faster and more accurate capture of subtle semantic differences between the two. Next, after calculating the semantic similarity between the first and second heterogeneous data fields based on their semantic vectors, the enhanced field representation information can be combined with the target interpretation template from the interpretation template library to generate the semantic alignment results and explanations for the first and second heterogeneous data fields. Compared with existing heterogeneous data semantic alignment methods, this method can effectively improve the efficiency and accuracy of semantic alignment between every two heterogeneous data fields in multi-source heterogeneous data, and adds explanations of the alignment results, making the alignment process more transparent and facilitating subsequent manual review or integrated retrieval, thereby achieving the ideal alignment effect.

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] See Figure 1 This is a flowchart illustrating a semantic alignment method for heterogeneous data provided in this embodiment. The method includes the following steps:

[0051] S101: Obtain multi-source heterogeneous data and perform semantic enhancement processing on the heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information.

[0052] In this embodiment, in order to improve the semantic alignment efficiency and accuracy of multi-source heterogeneous data and achieve the ideal alignment effect, such as... Figure 2As shown, after obtaining multi-source heterogeneous data, semantic enhancement methods, whether existing or future, are used to perform semantic enhancement processing on each heterogeneous data field in the multi-source heterogeneous data to obtain enhanced field representation information. This constructs a rich contextual representation (i.e., field semantic representation) for each heterogeneous data field, which is then used to execute the subsequent step S104 to achieve alignment processing of heterogeneous data fields. Compared with the existing plain text matching method, this method is more helpful in improving the accuracy of alignment.

[0053] It should be noted that this application does not limit the source, domain, content, or acquisition method of multi-source heterogeneous data, which can be determined based on actual circumstances and experience. For example, multi-source heterogeneous data can come from different data sources such as internal enterprise business systems (e.g., e-commerce platforms, membership systems, inventory systems, financial systems, OA office systems, etc.), the Internet, and user-generated content platforms (e.g., social media, news websites, etc.). The domain of multi-source heterogeneous data can be, but is not limited to, e-commerce retail, finance, healthcare, education, and manufacturing. The content of multi-source heterogeneous data can include, but is not limited to, structured data (e.g., tabular data in MySQL / Oracle databases), semi-structured data (e.g., log files, XML documents), and unstructured data (e.g., text, images, audio). Multi-source heterogeneous data can be acquired through existing or future data acquisition methods such as API calls, web crawling, and direct database connections. For example, JSON data can be acquired through API calls combined with format conversion.

[0054] Specifically, one possible implementation is that after obtaining multi-source heterogeneous data, the process of semantically enhancing the fields of each heterogeneous data contained therein to obtain enhanced field representation information may include the following steps (1)-(4):

[0055] Step (1): Perform upstream and downstream process field correlation analysis on the heterogeneous data fields in the multi-source heterogeneous data to obtain the first analysis result.

[0056] In this implementation, after obtaining multi-source heterogeneous data, the first step is to analyze data pipeline information such as SQL query logs and ETL scripts to identify the set of heterogeneous data fields with potential semantic relationships in the multi-source heterogeneous data, which will be used as the first analysis result to execute subsequent steps (4).

[0057] For example, consider the sales data (specifically, metadata from the two databases) from two different data sources—the online store (a MySQL database) and the offline store system (a PostgreSQL database) of a large retail enterprise—as multi-source heterogeneous data. The online system uses the field "online_sales_amount" to store transaction amounts, while the offline system uses "store_revenue". Member identifiers are "vip_id" in the online system and "member_card_no" in the offline system. Existing integration methods based on simple field name matching have a high failure rate. To address this, this application provides an automated solution based on deep semantic understanding. The process of performing upstream and downstream process field correlation analysis on the heterogeneous data fields contained in this multi-source heterogeneous data can be as follows: First, use database log analysis tools (such as Oracle's LogMiner) or monitor database system tables (such as MySQL's information_schema) to collect SQL query logs and ETL script execution records. Then, use an SQL parser to extract the table names, field names, and operation types (such as SELECT, JOIN, WHERE, GROUP BY) appearing in all queries. Next, based on the parsed field information, association rule analysis (such as the Apriori algorithm) is applied to count the frequency of fields appearing together in the same query or ETL task. For example, through analysis, it is found that the fields "online_sales_amount", "discount_amount", "coupon_amount", and the "total_price" field in the "orders" table appear together in more than 70% of sales analysis-related queries. Furthermore, these fields frequently participate in aggregation calculations such as SUM and AVG, as well as GROUP BY operations. Therefore, it can be preliminarily determined that they belong to the core semantic cluster of "sales transactions".

[0058] Step (2): Based on the data dictionary related to the multi-source heterogeneous data, extract the association information of the heterogeneous data fields in the multi-source heterogeneous data to obtain the extraction results.

[0059] In this implementation, after obtaining multi-source heterogeneous data, in order to improve the accuracy of semantic representation, not only are context features extracted from the upstream and downstream process field association analysis dimension mentioned in step (1), but also the association information of heterogeneous data fields in multi-source heterogeneous data (such as the complete business semantics, business definition, calculation formula, value range and business rule constraints of heterogeneous data fields) can be extracted according to the data dictionary related to multi-source heterogeneous data, and the extraction results are obtained to execute the subsequent step (4).

[0060] In this embodiment, the specific content of the data dictionary related to multi-source heterogeneous data is not limited. By integrating multi-source metadata information, semantic enhancement based on the data dictionary can be achieved. The data dictionary related to multi-source heterogeneous data may include, but is not limited to, database system table comments (referring to brief textual descriptions of tables, columns, indexes, etc., with no limitation on the specific construction method, which can be added when creating or modifying database objects using standard DDL statements, etc.), manually maintained data dictionary documents (referring to detailed business documents independent of the database, containing rich business semantics such as business definitions, calculation formulas, value ranges, business rules, and dependencies of data elements, with no limitation on the specific construction method, which can be manually created and maintained by data administrators, business analysts, etc. through documents or professional data management platforms, etc.), and metadata repositories (referring to a centralized metadata management platform that automatically collects and integrates metadata from multiple sources such as database systems, ETL tools, and BI platforms to form a unified view, with no limitation on the specific construction method, which can automatically crawl and integrate metadata from multiple source systems through specific metadata management tools and establish relationships, etc.). By utilizing the data dictionary content related to multi-source heterogeneous data, we can extract the complete business semantics, business definitions, calculation formulas, value ranges, and business rule constraints of each heterogeneous data field in the multi-source heterogeneous data, and use these as the extraction results.

[0061] For example, based on the above example, let's consider sales data from two different data sources—an online store (specifically a MySQL database) and an offline store system (specifically a PostgreSQL database)—of a large retail enterprise. Using a manually maintained data dictionary document as an example, since this document typically defines the business meaning, calculation formulas, and value ranges of data elements in detail, accessing and parsing it might reveal that the "store_revenue" field in the offline store system (specifically the PostgreSQL database) is explicitly defined as "the final amount received for a single transaction at the store, minus returns," with the calculation formula being sales_amount - return_amount. Similarly, the "online_sales_amount" field in the online store system (specifically the MySQL database) is defined in the data dictionary as "the actual amount received for an order, after deducting discounts and refunds." This high degree of consistency between the calculation logic and the essence of the business provides crucial evidence for determining the semantic equivalence of these two fields.

[0062] Step (3): Perform log analysis on the heterogeneous data fields in the multi-source heterogeneous data to obtain the second analysis result.

[0063] In this implementation, after obtaining multi-source heterogeneous data, in order to improve the accuracy of semantic representation, not only are context features extracted from the upstream and downstream process field association analysis dimension mentioned in step (1), but also the association information of heterogeneous data fields in the multi-source heterogeneous data is extracted based on the data dictionary in step (2). Furthermore, the usage log analysis of heterogeneous data fields in the multi-source heterogeneous data can be performed to obtain the second analysis result, which is used to execute the subsequent step (4). Specifically, the application patterns of each field in various systems (including but not limited to query logs, report generation records, statistical field usage frequency and co-occurrence patterns, etc.) can be collected and mined. Association analysis algorithms (such as the Apriori algorithm) can be used to discover frequently co-occurring field combinations, and the semantic association between fields can be strengthened based on these patterns.

[0064] For example, based on the above example, let's take sales data from two different data sources—the online mall (specifically a MySQL database) and the offline store system (specifically a PostgreSQL database) of a large retail enterprise—as the example of multi-source heterogeneous data. We can apply preset association rules (such as the Apriori algorithm) to analyze and calculate the co-occurrence support of each heterogeneous data field in reports or queries. For instance, suppose the analysis finds that "vip_id" has a co-occurrence support of 82% with "registration_date" and "last_login_time" in user profile analysis-related queries and reports. Similarly, "member_card_no" also has a co-occurrence support of 78% with "registration_date" and "last_login_time." This indicates that although "vip_id" and "member_card_no" have different names, they play highly similar semantic roles in real business scenarios; that is, they both serve as core identifiers of user membership information and are often associated with the same attribute fields.

[0065] Step (4): Using the first analysis result, the extraction result and the second analysis result, perform semantic enhancement processing on the heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information.

[0066] In this implementation, after obtaining the first analysis result, extraction result and second analysis result through steps (1), (2) and (3) respectively, these result data can be used to perform semantic enhancement processing on each heterogeneous data field in the multi-source heterogeneous data from three dimensions, and obtain enhanced field representation information composed of a field co-occurrence frequency statistics table and at least one data in the strongly correlated field pair list, providing a direct basis for subsequent steps to achieve semantic alignment.

[0067] For example: Based on the above example, taking the sales data from two different data sources—the online mall (specifically a MySQL database) and the offline store system (specifically a PostgreSQL database) of a large retail enterprise—as the multi-source heterogeneous data obtained, after obtaining the first analysis result, extraction result, and second analysis result through the above steps (1), (2), and (3), the contents of the field co-occurrence frequency statistics table and the contents of the strongly correlated field pair list formed by using these results data can be shown in Tables 1 and 2 below:

[0068] Heterogeneous data field 1 Field 2 Co-occurrence frequency online_sales_amount discount_amount 0.72 vip_id registration_date 0.82 member_card_no last_login_time 0.78

[0069] Table 1

[0070] Heterogeneous data field 1 Field 2 Related basis store_revenue online_sales_amount The business definitions and computational logic in the data dictionary are highly consistent. vip_id member_card_no The co-occurrence patterns of the fields used in the logs are highly similar to those of the key attribute fields.

[0071] Table 2

[0072] S102: Select the first heterogeneous data field and the second heterogeneous data field from the multi-source heterogeneous data; and input the first heterogeneous data field and its business definition and the second heterogeneous data field and its business definition into the pre-built domain language encoding model for encoding processing to obtain the semantic vector of the first heterogeneous data field and the semantic vector of the second heterogeneous data field; wherein the first heterogeneous data field and the second heterogeneous data field belong to heterogeneous data from different data sources.

[0073] In this embodiment, after obtaining multi-source heterogeneous data in step S101, a first heterogeneous data field and a second heterogeneous data field belonging to different data sources can be selected. The first heterogeneous data field and its business definition, and the second heterogeneous data field and its business definition, are then input into a pre-built domain language encoding model for encoding processing to obtain the semantic vectors of the first and second heterogeneous data fields, such as... Figure 2 As shown, this is used to execute the subsequent step S103.

[0074] For example, based on the above example, let's take sales data from two different data sources—the online store (specifically a MySQL database) and the offline store system (specifically a PostgreSQL database) of a large retail enterprise—as the multi-source heterogeneous data obtained. Assuming the first and second heterogeneous data fields are selected as "vip_id" and "member_card_no" respectively, we can construct corresponding rich text sequences based on the business definitions of these two fields. Specifically, the "vip_id" field can be constructed as a rich text sequence: [CLS] vip_id [SEP] Unique identifier for premium customers [SEP]. Similarly, the "member_card_no" field can be constructed as another rich text sequence: [CLS] member_card_no [SEP] Unique number assigned to membership card [SEP]. Then, a trained domain language encoding model (encoder) is used to encode these two enhanced text sequences to generate their respective semantic vectors, which can be defined as V. vip_id and V member_card_no .

[0075] It should be noted that this application does not limit the composition structure and construction method of the domain language coding model. One optional implementation method is to achieve domain-specific optimization of the domain language coding model to improve its coding accuracy and robustness. The domain language coding model can be obtained by comparative training using a loss function on positive and negative heterogeneous data pairs from heterogeneous training sample data of a specific domain (defined here as the target domain, the specific content of which is not limited and can be set according to actual conditions, such as the financial domain, the education domain, etc.). The specific construction process includes the following steps A and B:

[0076] Step A: Construct heterogeneous training sample data for the target domain, which includes positive sample heterogeneous data pairs and negative sample heterogeneous data pairs.

[0077] It's important to note that the domain language encoding model is trained by comparing positive and negative heterogeneous data pairs within the target domain's training sample heterogeneous data. Through the constraints of this model and the loss function, the distance between positive and negative heterogeneous data pairs in the vector space can be continuously reduced, while simultaneously increasing the distance between negative heterogeneous data pairs. This ultimately yields a model capable of accurately understanding enterprise-specific terminology and business context. Therefore, constructing heterogeneous training sample data from the target domain, consisting of both positive and negative heterogeneous data pairs, is crucial for training a model with stronger encoding capabilities.

[0078] In this embodiment, in order to build a model with greater accuracy and robustness, this application adopts a weakly supervised training strategy that combines a rule engine to generate pseudo-labels and injects human adversarial examples to generate heterogeneous training sample data, thereby reducing the reliance on manual annotation.

[0079] Specifically, when constructing heterogeneous training sample data for the target domain, one possible approach is to first obtain the unlabeled metadata of the target domain, and then, based on at least one of the preset rules among text similarity rules, data type consistency rules, and business rules, select field pairs that meet the preset conditions (the specific content is not limited and can be set according to actual conditions and experience) from the unlabeled metadata of the target domain to form positive sample heterogeneous data pairs and negative sample heterogeneous data pairs, which are used to form the heterogeneous training sample data for the target domain.

[0080] This application does not limit the content of the text similarity rules and can set them according to actual conditions and empirical values. For example, the text similarity rules can be set as follows: if the normalized edit distance of the names of two heterogeneous data fields is less than a set threshold (the specific value is not limited, such as 3), and the vector cosine similarity calculated by the basic semantic model (such as Sentence-BERT) is higher than the threshold (the specific value is not limited, such as 0.8), then the two can be marked as a positive sample heterogeneous data pair <anchor point, positive sample>, such as <"cust_id", "customer_id">.

[0081] Furthermore, this application does not limit the content of the data type consistency rules, and they can be set according to actual conditions and empirical values. For example, the data type consistency rule can be set as follows: if there is a fundamental data type conflict between two heterogeneous data fields (such as string type versus numeric type), they can be marked as a negative sample heterogeneous data pair to effectively prevent misalignment.

[0082] Furthermore, this application does not limit the content of the business rules, which can be set according to actual circumstances and experience. For example, the business rule can be set as follows: if the similarity of the business definition descriptions of two heterogeneous data fields exceeds a threshold, or if the calculation formulas are consistent (such as "sales revenue" and "net_sales" both being defined as "revenue - returns"), then the two can be marked as a positive sample heterogeneous data pair.

[0083] Another alternative approach is to introduce an active learning scheme when constructing heterogeneous training sample data in the target domain to improve the model's robustness to fuzzy cases. This involves having domain experts review sample pairs with low prediction confidence and manually injecting clearly labeled adversarial samples (including positive and negative heterogeneous data pairs). For example, experts can explicitly label <"account balance", "current balance"> as a positive heterogeneous data pair and <"account balance", "credit limit"> as a negative heterogeneous data pair. These high-quality heterogeneous data pairs can then be used as part of the core training sample heterogeneous data.

[0084] Another optional implementation is to further improve the model's encoding capability. When constructing heterogeneous training sample data for the target domain, semantic enhancement processing can be performed on the unlabeled metadata of the target domain to obtain sample enhancement field representation information. For specific enhancement methods, please refer to the above steps (1)-(4) for the process of speech enhancement processing of heterogeneous data fields in multi-source heterogeneous data. Simply replace "heterogeneous data fields in multi-source heterogeneous data" with "heterogeneous data fields in unlabeled metadata of the target domain". This will not be elaborated here. Then, the field pairs that meet the preset conditions selected from the unlabeled metadata of the target domain and the sample enhancement field representation information are fused to form enhanced positive sample heterogeneous data pairs and negative sample heterogeneous data pairs.

[0085] In this implementation, for a sample field pair <anchor field, strongly related field>, the input is not only the field name text, but can also be concatenated with related information such as business definitions and calculation formulas from the data dictionary, thus forming a text fragment rich in context, which serves as the final input data when training the model to execute the subsequent step B.

[0086] In this way, through the various implementation methods described above, large-scale, high-quality heterogeneous training sample data (including positive sample heterogeneous data pairs and negative sample heterogeneous data pairs) of the target domain can be constructed, thereby significantly reducing the dependence on manual annotation, which not only improves the accuracy and robustness of the homogeneous model, but also reduces the construction cost.

[0087] Step B: Input the heterogeneous data fields and their business definitions from the positive sample heterogeneous data pairs into the initial domain language encoding model, and input the heterogeneous data fields and their business definitions from the negative sample heterogeneous data pairs into the initial domain language encoding model. By adjusting the loss function, the domain language encoding model is trained to obtain the domain language encoding model.

[0088] In this embodiment, the initial domain language encoding model can be a classic Siamese Network structure (used for contrastive learning fine-tuning). This structure can contain two (or three) sub-models (encoders) sharing weights, each built based on a selected pre-trained language model (such as BERT-base, RoBERTa-large, etc.). The encoder is responsible for mapping the input text sequence into dense vectors in a high-dimensional semantic space.

[0089] Based on this, after constructing the target domain training sample heterogeneous data consisting of positive sample heterogeneous data pairs and negative sample heterogeneous data pairs through step A, each sample heterogeneous data pair (including positive sample heterogeneous data pairs and negative sample heterogeneous data pairs) and its business definition can be input into the initial domain language encoding model for training. The model outputs the similarity between the semantic vectors of the two fields in each sample heterogeneous data pair and the label of the corresponding sample heterogeneous data pair. Through multiple rounds of model training, until the training termination condition is met, the domain language encoding model is obtained.

[0090] The input data for each heterogeneous data pair (including positive and negative sample heterogeneous data pairs) and its label can be in the form of (Field_A, Field_B, Label). A Label of 1 indicates that the two fields in the heterogeneous data pair are semantically equivalent (i.e., a positive sample heterogeneous data pair), while a Label of 0 indicates that the two fields in the heterogeneous data pair are not semantically equivalent (i.e., a negative sample heterogeneous data pair). A typical format for the input training data is: [CLS] Field Name [SEP] Business Definition [SEP] Key Related Field [SEP]. For example, a sample input for a positive sample heterogeneous data pair could be: "[CLS] online_sales_amount [SEP] Order Amount, after deducting discounts and refunds [SEP] discount_amount [SEP], 1", etc.

[0091] Correspondingly, for the model output, each encoder with shared weights can output a fixed-size semantic vector after processing the input sequence of one field. This can be expressed as follows: for a pair of heterogeneous data pairs as input (Field_A, Field_B), the encoder can output vector V respectively.A and V B Based on this, by calculating two output vectors V A and V B The cosine similarity can be used as the semantic similarity score between the two fields, with the score ranging from 0 to 1.

[0092] Furthermore, during each round of model training, a pre-defined loss function is used. This loss function is designed to shorten the distance between heterogeneous data pairs of positive samples in the vector space and widen the distance between heterogeneous data pairs of negative samples in the vector space. By adjusting this loss function, the correlation between each heterogeneous data pair (including both positive and negative samples) can be constrained, resulting in increasingly higher vector similarity between two fields in positive heterogeneous data pairs and increasingly lower vector similarity between two fields in negative heterogeneous data pairs.

[0093] Specifically, in order to train a domain-specific language encoding model with better encoding performance, this embodiment uses the contrastive loss function during training, and the specific calculation formula is as follows:

[0094]

[0095] Where L represents the total loss of a batch of heterogeneous input data; N represents the batch size; (Anchor point) and (Positive sample) represents the semantic vector of a strongly correlated field pair (i.e., a positive sample heterogeneous data pair); (Anchor point) and (Negative sample) represents a heterogeneous data pair consisting of an anchor field and randomly sampled or constructed semantically irrelevant fields; d(x, y) represents the Euclidean distance between vectors x and y; margin is a hyperparameter that defines the minimum margin that should be maintained between negative sample pairs, and its specific value is not limited and can be set according to actual conditions and empirical values. It can be seen that only when the heterogeneous data pairs of positive samples become increasingly similar (i.e., the vector similarity between the two fields in the heterogeneous data pair of positive samples is higher), and the vector similarity between the two fields of negative samples becomes increasingly higher and increasingly dissimilar (i.e., the vector similarity between the two fields in the heterogeneous data pairs of positive and negative samples is lower), will the value of the loss function L decrease, thus achieving the constraint objective of the loss function.

[0096] In this way, by continuously adjusting the value of the loss function L through multiple rounds of training, training can be stopped when the value of the loss function L converges, for example, when the value of the loss function L reaches its minimum and remains basically unchanged. The domain language coding model obtained at this time can then be used as the trained domain language coding model.

[0097] S103: Calculate the semantic similarity between the first heterogeneous data field and the second heterogeneous data field based on the semantic vector of the first heterogeneous data field and the semantic vector of the second heterogeneous data field.

[0098] In this embodiment, after obtaining the semantic vectors of the first heterogeneous data field and the second heterogeneous data field through step S102, existing or future vector similarity calculation methods can be used to calculate the semantic similarity between the first heterogeneous data field and the second heterogeneous data field based on their semantic vectors. For example, the cosine similarity between the semantic vectors of the first heterogeneous data field and the second heterogeneous data field can be calculated as their semantic similarity score (ranging from 0 to 1) to execute the subsequent step S104.

[0099] For example, based on the above example, let's take sales data from two different data sources—the online store (specifically a MySQL database) and the offline store system (specifically a PostgreSQL database) of a large retail enterprise—as the example of multi-source heterogeneous data. We then use a trained domain language encoding model to encode the first heterogeneous data field "vip_id" and its business definition, and the second heterogeneous data field "member_card_no" and its business definition, generating their respective semantic vectors V. vip_id and V member_card_no Then, the cosine similarity between the two vectors can be calculated, yielding a score between 0 and 1. Because the domain language encoding model is trained on a large amount of heterogeneous data (containing similar correspondences) of training samples from the target domain, it can deeply understand that although "vip_id" and "member_card_no" are literally different, they are semantically highly related in a given business context (both are unique member identifiers and are often associated with the same attributes). Therefore, it may obtain a high semantic similarity score, such as 0.89.

[0100] S104: Based on the semantic similarity between the first heterogeneous data field and the second heterogeneous data field, as well as the enhanced field representation information, a target interpretation template is obtained from the pre-built interpretation template library; and using the target interpretation template, the semantic alignment result of the first heterogeneous data field and the second heterogeneous data field and its interpretation description are generated.

[0101] In this embodiment, it should be noted that, in order to enhance the explanation of the alignment results, make the alignment process more transparent, and facilitate subsequent manual review or integrated retrieval operations, thereby achieving the desired alignment effect, this embodiment pre-constructs an explanation template library. This explanation template library can be a structured knowledge base, and it can be used to associate different types of matching evidence with corresponding natural language explanation templates. The specific content included in this explanation template library is not limited, and may include, but is not limited to, the correspondence between text semantic rules and explanation templates, the correspondence between business logic rules and explanation templates, and the correspondence between context association rules and explanation templates. Examples of the content included in this explanation template library are shown in Table 3 below:

[0102] Rule Categories Triggering conditions (example) Corresponding Explanation Template (Example) Text semantic rules The edit distance between field names is less than the threshold, and the semantic similarity score between the semantic vectors output by the model is greater than 0.85. "The field names are highly similar, and the meanings are close." Business logic rules If the business definitions of two fields in the data dictionary have an overlap of >90%, or their calculation formulas are completely identical, then... "The business definition and calculation logic of the fields are completely consistent." Contextual association rules Log analysis revealed that the two fields frequently co-occurred with the same set of other fields (co-occurrence frequency > 0.75). "In the business context, this field often appears together with XXX, and the usage pattern is highly consistent." Mixed triggering rules When multiple rules above are met Combine multiple templates to form a more comprehensive explanation.

[0103] Table 3

[0104] Based on this, after obtaining the semantic similarity (specifically, a semantic similarity score) between the first heterogeneous data field and the second heterogeneous data field through steps S103 and S101, and enhancing the field representation information, as follows: Figure 2 As shown, it can be further determined whether the semantic similarity of the first heterogeneous data field and the second heterogeneous data field and the enhanced field representation information meet the triggering conditions of at least one of the text semantic rules, business logic rules and context association rules shown in Table 3 above. If so, the interpretation templates corresponding to at least one rule to which the triggering conditions are met are combined to obtain the target interpretation template, which is used to generate the semantic alignment results of the first heterogeneous data field and the second heterogeneous data field and their interpretations, so as to facilitate subsequent manual review or integrated retrieval operations.

[0105] Furthermore, semantic similarity scores below a preset threshold (the specific value is not limited and can be set according to the situation and experience, such as 0.85) can be manually reviewed and labeled. The alignment results confirmed / corrected by the manual reviewer are then used as the final alignment results and their explanations, and sent to the downstream BI platform (which can be understood as a system at the end of the data process in the business intelligence ecosystem. It transforms the processed data into understandable business insights in a visual and interactive way to assist enterprises in making various decisions) as the data basis for subsequent operations.

[0106] Specifically, for high semantic similarity scores (e.g., ≥0.95), the alignment results can be automatically applied to the data model; for medium semantic similarity scores (e.g., 0.85 ~ 0.94), a suggested label can be attached and pushed to the user for reference and confirmation; while low semantic similarity scores (e.g., <0.85) can be automatically marked as high-priority items "awaiting manual review" and directly included in the manual review queue to ensure that potential risks are intervened in a timely manner.

[0107] For example, taking the sales data from two different data sources—the online mall (specifically a MySQL database) and the offline store system (specifically a PostgreSQL database) of a large retail enterprise—as the multi-source heterogeneous data obtained, assuming that the semantic similarity score of the first heterogeneous data field "online_sales_amount" and the second heterogeneous data field "store_revenue" is 0.96, by searching the pre-built interpretation template library shown in Table 3, it is found that the field pair composed of these two fields simultaneously triggered the "text semantic rule" (the edit distance of the field name is < the threshold, and the semantic similarity score between the semantic vectors output by the model is > 0.85), the "business logic rule" (the calculation formula is consistent in the data dictionary), and the "context association rule" (all of which frequently co-occur with fields such as discounts and coupons, with a co-occurrence frequency > 0.75). Therefore, various explanation templates can be automatically combined as the target explanation template, and the semantic alignment result and its explanation can be generated as follows: "The semantic similarity score is 0.96; the field business definitions are highly consistent, both representing the actual transaction amount after deducting discounts and returns, and the calculation logic is the same; they often appear in conjunction with similar field sets (such as discounts and coupons) in the data analysis process." Furthermore, based on the preset semantic similarity score judgment strategy, this alignment result can automatically take effect and generate standardized JSON output for integration by downstream BI platforms.

[0108] Thus, the solution for semantic alignment of multi-source heterogeneous data achieved by executing steps S101-S104 above achieves at least four technological advancements compared to existing methods for semantic alignment of heterogeneous data: First, improved alignment accuracy, specifically manifested in the significant improvement in field alignment accuracy through context enhancement and training of a domain language encoding model using domain knowledge. Second, stronger domain adaptability, specifically manifested in the domain language encoding model trained using domain knowledge being able to quickly adapt to different industry terms, exhibiting strong versatility and applicability to multiple industries such as finance, healthcare, and retail. Third, higher degree of automation, specifically manifested in the use of a time-weakly supervised training strategy to generate heterogeneous training sample data when training the domain language encoding model, thereby reducing reliance on manual annotation and supporting automated deployment. Compared to existing methods for semantic alignment of heterogeneous data requiring weeks of manual configuration, deployment time can be shortened to hours. Fourth, enhanced interpretability, specifically manifested in the final generation of an explanation containing information such as alignment reasons and reference indicators, thereby enhancing the interpretability of the alignment results.

[0109] In summary, this embodiment provides a semantic alignment method for heterogeneous data. First, it acquires multi-source heterogeneous data and performs semantic enhancement processing on the heterogeneous data fields to obtain enhanced field representation information. Then, it selects a first heterogeneous data field and a second heterogeneous data field belonging to different data sources from the multi-source heterogeneous data. The first heterogeneous data field and its business definition, and the second heterogeneous data field and its business definition, are input into a pre-built domain language encoding model for encoding processing to obtain semantic vectors for the first and second heterogeneous data fields. Next, based on the semantic vectors of the first and second heterogeneous data fields, the semantic similarity between the two fields is calculated. Then, based on the semantic similarity and enhanced field representation information, a target interpretation template is matched from a pre-built interpretation template library. Finally, using this target interpretation template, the semantic alignment result of the first and second heterogeneous data fields and its explanation are generated.

[0110] As can be seen, this application first performs semantic enhancement processing on the heterogeneous data fields to obtain enhanced field representation information when performing semantic alignment of heterogeneous data. This constructs rich field semantic representations for subsequent alignment processing, which is more conducive to improving alignment accuracy compared to existing plain text matching methods. Based on this, this application further utilizes a pre-built domain language encoding model with the ability to accurately understand specific domain-specific terms and business contexts to encode the first and second heterogeneous data fields, selected from multi-source heterogeneous data and belonging to different data sources, to obtain their respective semantic vectors. This allows for faster and more accurate capture of subtle semantic differences between the two. Next, after calculating the semantic similarity between the first and second heterogeneous data fields based on their semantic vectors, the enhanced field representation information can be combined with the target interpretation template from the interpretation template library to generate the semantic alignment results and explanations for the first and second heterogeneous data fields. Compared with existing heterogeneous data semantic alignment methods, this method can effectively improve the efficiency and accuracy of semantic alignment between every two heterogeneous data fields in multi-source heterogeneous data, and adds explanations of the alignment results, making the alignment process more transparent and facilitating subsequent manual review or integrated retrieval, thereby achieving the ideal alignment effect.

[0111] Furthermore, embodiments of this application also provide a semantic alignment device for heterogeneous data, including: a processor, a memory, and a system bus;

[0112] The processor and the memory are connected via the system bus;

[0113] The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform any of the above-described implementations of the semantic alignment method for heterogeneous data.

[0114] refer to Figure 3 The diagram illustrates a structural schematic suitable for implementing the semantic alignment device for heterogeneous data in the embodiments of this application. The semantic alignment device for heterogeneous data in the embodiments of this application may include, but is not limited to, terminals such as mobile phones, tablets, and smartwatches, as well as servers. Figure 3 The semantic alignment device for heterogeneous data shown is merely an example of an electronic device and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0115] like Figure 3As shown, the semantic alignment device for heterogeneous data may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 302 or a program loaded from storage device 308 into random access memory (RAM) 303. When the semantic alignment device for heterogeneous data is powered on, RAM 303 also stores various programs and data required for device operation. The processor 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0116] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, memory cards, hard drives, etc.; and communication devices 309. Communication device 309 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 A semantic alignment device for heterogeneous data with various means is shown; however, it should be understood that implementation or possession of all the means shown is not required. More or fewer means may be implemented alternatively.

[0117] Furthermore, embodiments of this application also provide a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to execute any of the above-described implementations of the semantic alignment method for heterogeneous data.

[0118] Furthermore, this application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute any of the above-mentioned implementation methods of the semantic alignment method for heterogeneous data.

[0119] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0120] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0121] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0122] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A semantic alignment method for heterogeneous data, characterized in that, include: Acquire multi-source heterogeneous data, and perform semantic enhancement processing on the heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information; Select a first heterogeneous data field and a second heterogeneous data field from the multi-source heterogeneous data; The first heterogeneous data field and its business definition, and the second heterogeneous data field and its business definition, are input into a pre-built domain language encoding model for encoding processing to obtain the semantic vectors of the first heterogeneous data field and the second heterogeneous data field; the first heterogeneous data field and the second heterogeneous data field belong to heterogeneous data from different data sources. Based on the semantic vector of the first heterogeneous data field and the semantic vector of the second heterogeneous data field, calculate the semantic similarity between the first heterogeneous data field and the second heterogeneous data field; Based on the semantic similarity between the first heterogeneous data field and the second heterogeneous data field, as well as the enhanced field representation information, the target interpretation template is obtained by matching from the pre-built interpretation template library. Then, using the target interpretation template, semantic alignment results and their interpretations are generated for the first heterogeneous data field and the second heterogeneous data field.

2. The method according to claim 1, characterized in that, The semantic enhancement processing of heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information includes: Perform upstream and downstream process field correlation analysis on the heterogeneous data fields in the multi-source heterogeneous data to obtain the first analysis result; Based on the data dictionary associated with the multi-source heterogeneous data, the association information of the heterogeneous data fields in the multi-source heterogeneous data is extracted to obtain the extraction result; The heterogeneous data fields in the multi-source heterogeneous data are analyzed using usage logs to obtain a second analysis result. Using the first analysis result, the extraction result, and the second analysis result, semantic enhancement processing is performed on the heterogeneous data fields in the multi-source heterogeneous data to obtain enhanced field representation information.

3. The method according to claim 2, characterized in that, The enhanced field characterization information includes at least one of the following: a field co-occurrence frequency statistics table and a list of strongly correlated field pairs.

4. The method according to claim 1, characterized in that, The domain language encoding model is constructed as follows: Construct heterogeneous training sample data for the target domain, wherein the heterogeneous training sample data includes positive sample heterogeneous data pairs and negative sample heterogeneous data pairs; The heterogeneous data fields and their business definitions in the positive sample heterogeneous data pairs are input into the initial domain language encoding model, and the heterogeneous data fields and their business definitions in the negative sample heterogeneous data pairs are input into the initial domain language encoding model. By adjusting the loss function, the domain language encoding model is trained to obtain the model.

5. The method according to claim 4, characterized in that, The loss function is a contrastive loss function; the loss function is used to narrow the distance between positive sample heterogeneous data pairs in the vector space and to widen the distance between negative sample heterogeneous data pairs in the vector space.

6. The method according to claim 4, characterized in that, The heterogeneous training sample data for the target domain includes: Obtain unlabeled metadata of the target domain; and based on at least one preset rule among text similarity rules, data type consistency rules, and business rules, filter out field pairs that meet preset conditions from the unlabeled metadata of the target domain to form positive sample heterogeneous data pairs and negative sample heterogeneous data pairs.

7. The method according to claim 6, characterized in that, The heterogeneous training sample data for the target domain includes: Semantic enhancement processing is performed on the unlabeled metadata of the target domain to obtain sample enhancement field representation information; The field pairs that meet the preset conditions and are selected from the unlabeled metadata of the target domain are fused with the sample enhancement field representation information to form enhanced positive sample heterogeneous data pairs and negative sample heterogeneous data pairs.

8. The method according to any one of claims 1-7, characterized in that, The interpretation template library includes the correspondence between text semantic rules and interpretation templates, the correspondence between business logic rules and interpretation templates, and the correspondence between context association rules and interpretation templates; the step of matching the target interpretation template from the pre-built interpretation template library based on the semantic similarity of the first heterogeneous data field and the second heterogeneous data field and the enhanced field representation information includes: Determine whether the semantic similarity between the first heterogeneous data field and the second heterogeneous data field, as well as the enhanced field representation information, satisfy the triggering condition of at least one of the text semantic rules, business logic rules, and context association rules; If so, the interpretation templates corresponding to at least one rule to which the triggering condition is satisfied will be combined to obtain the target interpretation template.

9. A semantic alignment device for heterogeneous data, characterized in that, include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, the one or more programs including instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on a terminal device, cause the terminal device to perform the method described in any one of claims 1-8.

11. A computer program product, characterized in that, The program product includes a computer program that, when executed by an electronic product, can implement the method described in any one of claims 1-8.

Images