A cross-domain cross-source data alignment method, system and electronic device
By employing a cross-domain and cross-source data alignment method, and utilizing a data multimodal representation model to generate vector representations of keys, values, and visual positions, the problem of aligning data tables from different domains and sources is solved, improving the accuracy and efficiency of data alignment and enhancing the robustness and generalization ability of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- WUHAN UNIV
- Filing Date
- 2023-01-03
- Publication Date
- 2026-04-10
AI Technical Summary
Existing technologies struggle to efficiently align data when dealing with tables from different fields and sources, especially when handling irregularly named fields, resulting in low matching accuracy. Furthermore, deep learning models suffer from textual ambiguity and distributional differences during training, leading to insufficient generalization capabilities and requiring repeated construction and adjustment of matching strategies.
A cross-domain, cross-source data alignment method is adopted. A data multimodal representation model is used to generate vector representations of keys, values, and visual positions. Combined with textual and tabular visual information, semantic distance is calculated using multi-head self-attention and cross-attention mechanisms to determine the data alignment result.
It improves the accuracy and efficiency of data alignment, enhances the robustness and generalization ability of the model, reduces human intervention, and is applicable to data alignment from different fields and sources.
Smart Images
Figure CN116050374B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of computers, and relates to a data alignment method, system and electronic device, in particular to a data alignment method, system and electronic device for a table containing multiple fields or multiple cells. BACKGROUND
[0002] Enterprises, government agencies and scientific research institutions produce a large amount of data in their daily production, operation, management and research. These data are stored in a large number of, scattered, self-defined databases or data tables. These databases or tables lack unified data division standards and field naming specifications, resulting in the use of multiple expression methods for data with the same or similar content in different fields and different sources, which causes difficulty in alignment. Because of the above problems, when data statistics, data mining and analysis applications are involved, such as cross-department, cross-unit data aggregation, personnel information collection, data auditing and other needs, manual item-by-item comparison is often used to align the data, which requires a large amount of manpower and financial resources, resulting in a large amount of waste.
[0003] The existing machine automatic data alignment or matching method mainly calculates the similarity of field names or keys in key-value pairs, and selects data pairs with high similarity as matching prediction results. The similarity calculation methods include traditional character matching, keyword matching, TF-IDF based on statistical methods, and similarity calculation based on pre-trained Word2Vec and WordEmbedding of deep learning.
[0004] The manual rule method and the statistical method have low matching accuracy when facing irregularly named field names. As for the deep learning method, because the number of keys is relatively small, the text length is short, and the context is lacking, the problem of polysemy in short text is difficult to solve. For example, the content of "remarks" in different data sources is very different and cannot be simply classified into one category. Moreover, the training data used by the existing pre-trained language model during training are mainly sentences, while the keys of data are usually words or phrases, which have a distribution gap, making it difficult to widely use the trained model. It is necessary to repeatedly build and adjust the matching strategy for different field sources, which is inefficient. SUMMARY
[0005] In view of the data alignment needs and the deficiencies of the prior art, the present application provides a cross-domain and cross-source data alignment method, system and electronic device, which takes multiple sets of key-value pair data to be matched as input and outputs the matching results therebetween.
[0006] The technical scheme adopted by the method of the present application is: a cross-domain and cross-source data alignment method, comprising the following steps:
[0007] Step 1: input multiple sets of table data to be aligned;
[0008] Step 2: extract key-value pairs in the data and their positions in the table;
[0009] Step 3: use a data multi-modal representation model to generate vector representations of keys, values, and visual positions;
[0010] The data multi-modal representation model is used to vectorize the keys and values in the data from text and data in the table visually;
[0011] The multi-modal representation model includes an input module, a data content embedding module, a data position embedding module, an encoding module, a decoding module, and a feature output module;
[0012] The input module is used for input of two modalities of text and table pictures, both in sequence form; and a sequence containing only a [start] tag is inputted at the same time;
[0013] The data content embedding module includes a text embedding module and a picture embedding module; the text embedding module is in the form of an Embedding embedding layer; the picture embedding module is a convolutional neural network layer, the first layer is a convolutional layer with a convolution kernel size of 3 and a step size of 1; the second layer is a convolutional layer with a convolution kernel size of 5 and a step size of 1; the third and fourth layers are convolutional layers with a convolution kernel size of 7 and a step size of 2; a residual block is added after the first and second convolutional layers, and a normalization layer, an activation layer, and a residual block are sequentially added after the third and fourth convolutional layers;
[0014] The data position embedding module is used to represent the relative positions of the text sequence and the picture sequence, wherein the text sequence adopts one-dimensional position, and the picture sequence arranges the original two-dimensional coordinates in the order of from left to right and from top to bottom into one-dimensional position;
[0015] The encoding module includes a plurality of calculation units with the same structure, each calculation unit sequentially includes a multi-head self-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer, and a normalization layer; each addition operation layer and normalization layer is connected with a residual block;
[0016] The multi-head self-attention module is composed of a plurality of self-attention modules, each of which independently extracts information from different representation subspaces; the self-attention module is an attention module with the same input vector, which is used to calculate the weighting between different feature maps; take different feature maps a and b, perform matrix multiplication and Softmax operation on them to get the local similarity between them, and then multiply the result with b to get a new feature map c with attention, which is the output of the attention module;
[0017] The decoding module sequentially comprises a self-attention module, an addition operation layer, a normalization layer, a cross-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer and a normalization layer; each addition operation layer and normalization layer is connected with a residual block; the cross-attention module is an attention module with two different vectors as inputs, one of which is from the output of the encoding module, and the other is from the output of the previous decoding module;
[0018] The feature output module is the output of the last decoding module;
[0019] Step 4: Calculate the semantic distance of the vector representation of different data;
[0020] Step 5: Evaluate the semantic distance between different data to determine the alignment result.
[0021] The technical scheme of the system of the application is: a cross-domain and cross-source data alignment system comprising the following modules:
[0022] Module 1 is used for inputting multiple sets of table data to be aligned;
[0023] Module 2 is used for extracting key-value pairs in the data and their positions in the table;
[0024] Module 3 is used for generating vector representations of keys, values and visual positions using a data multi-modal representation model;
[0025] The data multi-modal representation model is used for vector representation of keys and values in the data from text and data in the table visually;
[0026] The multi-modal representation model comprises an input module, a data content embedding module, a data position embedding module, an encoding module, a decoding module and a feature output module;
[0027] The input module is used for input of two modalities of text and table picture, and both are in sequence form; and a sequence containing only a [start] tag is inputted at the same time;
[0028] The data content embedding module comprises a text embedding module and a picture embedding module; the text embedding module is in the form of an Embedding embedding layer; the picture embedding module is a convolutional neural network layer, the first layer is a convolutional layer with a convolution kernel size of 3 and a step size of 1; the second layer is a convolutional layer with a convolution kernel size of 5 and a step size of 1; the third and fourth layers are convolutional layers with a convolution kernel size of 7 and a step size of 2; a residual block is added after each of the first and second convolutional layers; a normalization layer, an activation layer and a residual block are sequentially added after each of the third and fourth convolutional layers;
[0029] The data position embedding module is used for representing relative positions of the text sequence and the picture sequence, wherein the text sequence adopts one-dimensional positions, and the picture sequence arranges original two-dimensional coordinates into one-dimensional positions in sequence from left to right and from top to bottom.
[0030] The encoding module comprises a plurality of calculation units with the same structure, each calculation unit sequentially comprises a multi-head self-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer and a normalization layer; each addition operation layer and normalization layer is connected with a residual block;
[0031] The multi-head self-attention module is composed of a plurality of self-attention modules, each self-attention module independently extracts information from different representation subspaces; the self-attention module is an attention module with the same input vector, which is used for calculating the weighting between different feature maps; the local similarity between different feature maps a and b is obtained by matrix multiplication and Softmax operation, and the result is multiplied by b to obtain a new attention feature map c as the output of the attention module;
[0032] The decoding module sequentially comprises a self-attention module, an addition operation layer, a normalization layer, a cross-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer and a normalization layer; each addition operation layer and normalization layer is connected with a residual block; the cross-attention module is an attention module with two different input vectors, one of which is from the output of the encoding module, and the other is from the output of the previous decoding module;
[0033] The feature output module is the output of the last decoding module;
[0034] Module 4 is used for calculating the semantic distance of the vector representation of different data;
[0035] Module 5 is used for evaluating the semantic distance between different data to determine the alignment result.
[0036] The technical scheme adopted by the electronic device of the present application is: an electronic device comprising:
[0037] One or more processors;
[0038] A storage device is used for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors realize the cross-domain cross-source data alignment method.
[0039] Compared with the prior art, the present application has the following advantages:
[0040] (1) In addition to the use of the pairing between the keys, the matching of the value is considered to enhance the matching of the keys in the prior art. The key text content in the key-value pair is usually short and is easily affected by polysemy. The text content of the value is relatively longer and can effectively overcome this problem, so adding the value as the basis for comparison can improve the robustness of the model.
[0041] (2) In addition to the representation in this paper, the table visual structure is fused as part of the semantic representation of the key-value pair, breaking the limitation of the prior art that only uses single-modal information for matching. Different fields of data have great differences in textual expression, and the expression generalization ability of the text model is required to be higher. In the tabulation process, similar contents often have similar table structures. Adding the local data features of the table structure can supplement the text information to a certain extent. BRIEF DESCRIPTION OF DRAWINGS
[0042] Figure 1 The method flowchart of the embodiment of the present application is shown in the figure.
[0043] Figure 2 The data multi-modal representation model structure diagram of the embodiment of the present application is shown in the figure.
[0044] Figure 3 The table data diagram of the embodiment of the present application is shown in the figure.
[0045] Figure 4 The key-value pair and content position extraction diagram in the MySQL table and the Word table in the embodiment of the present application is shown in the figure.
[0046] Figure 5 The vector table diagram of the key, value and visual position generated by using the data multi-modal representation model in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION
[0047] In order to facilitate those skilled in the art to understand and implement the present application, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application.
[0048] See Figure 1 The present application provides a cross-domain and cross-source data alignment method, which comprises the following steps:
[0049] Step 1: input a plurality of sets of table data to be aligned;
[0050] In this embodiment, the table data comes from different fields or different sources in the same field, and contains multiple different fields. Table data refers to data with "key-value pair" relationship, the value corresponding to the same key can be unique or not unique, and the storage format of table data is not limited, including Word, Excel, PDF, HTML, and data stored in a database, etc.
[0051] Step 2: Extract the key-value pairs in the data and their positions in the table;
[0052] Step 3: Use the data multi-modal representation model to generate vector representations of keys, values, and visual positions;
[0053] See Figure 2 The data multi-modal representation model of the embodiment is used to perform vector representation of keys and values in the data from text and data in the table visually. The input of the data multi-modal representation model is a combination of three types of multi-modal data, including the key of the data (text data), the value of the data (text data), and the local screenshot of the table (picture data). The text data and the picture data are input into the representation model based on the Transformer. The difference from similar multi-modal representation models is that the input data in this method has a natural alignment relationship between the text data and the picture data because the position of the data in the table is known, so there is no need to perform alignment operation between the two modalities.
[0054] The multi-modal representation model of the embodiment includes an input module, a data content embedding module, a data position embedding module, an encoding module, a decoding module, and a feature output module;
[0055] The input module of the embodiment is used for input of two modalities of text and table picture, and both are in sequence form; and the input sequence only contains a [start] tag;
[0056] In the model training of the embodiment, the output sequence is in the form of a [start] tag as the beginning of the real output result; and in the prediction using the model, the output sequence only contains a [start] tag.
[0057] The data content embedding module of the embodiment is used for respectively embedding different modalities. The data content embedding module of the embodiment includes a text embedding module and a picture embedding module. The text embedding module of the embodiment is in the form of an Embedding embedding layer, and the size of a word table is 5000. When embedding a text key-value pair, the maximum length of a key is limited to 20 words, and the maximum length of a value is limited to 200 words. Texts that do not meet the word number limit are uniformly padded. The picture embedding module of the embodiment is a convolutional neural network layer. The first layer is a convolutional layer with a convolution kernel size of 3 and a step size of 1. The second layer is a convolutional layer with a convolution kernel size of 5 and a step size of 1. The third and fourth layers are convolutional layers with a convolution kernel size of 7 and a step size of 2. A residual block is added after each of the first and second convolutional layers. A normalization layer, an activation layer, and a residual block are sequentially added after each of the third and fourth convolutional layers.
[0058] The data position embedding module of the embodiment is used for representing the relative positions of a text sequence and a picture sequence. The text sequence adopts one-dimensional positions, and the two-dimensional coordinates of the picture sequence are arranged in one-dimensional positions in the order from left to right and from top to bottom. The specific calculation method is: p = sin(wx + t).
[0059] The encoding module of the embodiment includes a plurality of calculation units with the same structure. Each calculation unit includes, in sequence, a multi-head self-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer, and a normalization layer. Each addition operation layer and normalization layer is connected with a residual block.
[0060] The multi-head self-attention module of the embodiment is composed of a plurality of self-attention modules. Each self-attention module independently extracts information from different representation subspaces. The self-attention module of the embodiment is an attention module with the same input vector, which is used to calculate the weighting between different feature maps. The local similarity between different feature maps a and b is obtained by performing matrix multiplication and a Softmax operation on a and b. The result is multiplied by b to obtain a new feature map c with attention, which is the output of the attention module.
[0061] The decoding module of the embodiment includes, in sequence, a self-attention module, an addition operation layer, a normalization layer, a cross-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer, and a normalization layer. Each addition operation layer and normalization layer is connected with a residual block. The cross-attention module of the embodiment is an attention module with two different input vectors. One input is the output of the encoding module, and the other input is the output of the previous decoding module.
[0062] The feature output module of the embodiment is the output of the last decoding module. The result can be used as a representation feature of a sequence input during training.
[0063] The multimodal representation model in this embodiment consists of the corresponding key-value pair text and its screenshot in a table. Semantically, they are naturally 5-aligned, requiring no additional alignment operations. The feature outputs of the three are concatenated to form the feature vector of the key-value pair.
[0064] Step 4: Calculate the semantic distance between the vector representations from different data sources;
[0065] In this embodiment, cosine distance is used to calculate the distance between two vectors.
[0066] In this embodiment, a natural language processing model is used to obtain the representation vectors of data keys and values. Alternatively, existing models can be used.
[0067] Models, including but not limited to BERT, Roberta, GPT, ERNIE, etc., can be used to obtain the visual representation of zero data in tables using machine vision or document layout analysis models. Existing models, including but not limited to U-net, Faster R-CNN, Vit, Dit, LayoutLM, etc., can also be used.
[0068] Step 5: Evaluate the semantic distance between different data and determine the alignment result.
[0069] In this embodiment, the matching results are sorted according to the semantic distance calculated in step 4, and the n closest data pairs are selected; where n is the hyperparameter of the model.
[0070] The specific method for calculating the semantic distance between data pairs in this embodiment can also employ existing models, including but not limited to these methods.
[0071] The distance between features can be calculated using formulas such as Euclidean distance, Manhattan distance, Chebyshev distance, and Mahalanobis distance.
[0072] The data multimodal representation model used in this embodiment is a pre-trained data multimodal representation model; its training process includes the following steps:
[0073] (1) Input several sets of table data;
[0074] 0(2) Extract key-value pairs from the data and their positions in the table;
[0075] (3) Sample the key-value pairs and consider those that match as positive examples and those that do not match as negative examples.
[0076] (4) Using a multimodal data representation model, obtain text vector representations of keys and values, visually displaying keys and values within the table.
[0077] The expression;
[0078] 5(5) Calculate the semantic distance between different key-value pairs, and calculate the prediction result of matching according to the semantic distance;
[0079] (6) According to the predetermined loss function, the loss of the prediction result and the true positive and negative examples is calculated;
[0080] The loss function is:
[0081]
[0082] Where q and k + are a pair of positive samples, q and k i represent all sample sampling, and τ is a hyperparameter of the data multi-modal representation model;
[0083] In contrast learning, the input of the model is generally a pair of data, which can be a pair of data that can be matched (becoming "positive sample") or a pair of data that cannot be matched (becoming "negative sample"), and all positive samples and negative samples together become sample sampling of data. The model learns how to distinguish whether a pair of data can be matched through such training data.
[0084] (7) According to the loss function, the data multi-modal representation model is iteratively corrected until the model converges, and a trained data multi-modal representation model is obtained.
[0085] This embodiment takes a typical demand scenario in personnel information collection: taking the personnel information registration table in the MySQL database and the Word file of the university faculty as an example, using the data alignment method in the application, the key-value pairs describing the same content in the two are aligned. Specifically, the following steps are included:
[0086] Step 1: input several MySQL data tables including personnel information, and input several Word format personnel information registration tables. In order to simplify the description, this paper uses a MySQL table (Table 1 in the Figure 3 ) containing only three fields of "educational background", "academic achievement" and "research field", and a Word table (Table 2 in the Figure 3 ) containing only three fields of "main experience", "scientific research content" and "award experience" to illustrate the idea of the scheme.
[0087] Step 2: extract the key-value pairs and content positions in the MySQL table and the Word table respectively, such as Figure 4 .
[0088] Step 3: Take out a pair of key-value data, use the pre-trained Roberta model to represent the vectors of the text of the key and value, and use the Dit model to represent the vectors of the picture screenshot of the cell corresponding to the position of the key and value and the cells nearby. The vectors of the three are spliced together as the multi-modal data representation vector of the key-value pair; as Figure 5 .
[0089] Step 4: Set positive examples and negative examples, and in this example, the positive examples are "education background-main experience" and "academic achievement-research content", and the negative examples are the other two pairs of fields.
[0090] Step 5: Pair the vector representations of the key-value pairs from the two tables, calculate their semantic distance using the Cosine distance formula, and then use the cross-entropy with the Softmax function as the loss function.
[0091] Step 6: Update the model parameters in step 3 using the back propagation algorithm. Repeat the above process until the model converges to obtain the data alignment model in this scheme.
[0092] Step 7: When new data needs to be aligned and predicted, repeat steps 2 and 3 to calculate the semantic distance between the data, and determine whether the data matches according to the set threshold.
[0093] In this embodiment, natural language and machine vision models are used to represent the keys and values in the data from text and data in the table visually. Using the contrast learning paradigm, the positive and negative examples obtained from a certain sampling are used to iteratively train the representation of the aforementioned model. When training with massive table data, the training strategy in the present application only needs a small amount of manual training data screening to obtain multiple different modal data representation models.
[0094] The data to be matched in this embodiment obtains multi-modal high-dimensional vector representations of keys, values, and structures through the obtained model, calculates the similarity between multiple groups of data, and uses a fusion model to evaluate the results to determine the matching results of the data pairs from different data dimensions. This method uses multi-modal input to model content from different fields and different sources and extract abstract features, improving the generalization ability of the model and overcoming the shortcomings of existing matching methods.
[0095] The present application is applicable to the alignment between data that can be represented as key-value pairs, such as database-to-database data alignment, table-to-table data alignment, database-to-table data alignment, and table-to-database data alignment.
[0096] It should be understood that the above description is merely a detailed explanation of the preferred embodiments and is not intended to limit the patent protection scope of the present application. Any modification or alternation made by those skilled in the art without departing from the scope of the present application shall fall within the patent protection scope of the present application. The patent protection scope of the present application shall be subject to the appended claims.
Claims
1. A method for cross-domain cross-source data alignment, characterized in that, The method comprises the following steps: Step 1: inputting multiple sets of table data to be aligned; Step 2: extracting key-value pairs in the data and positions of the key-value pairs in the table; Step 3: generating vector representations of the key, value and visual position by using a data multi-modal representation model; The data multi-modal representation model is used for vector representation of the key and value in the data from text and data in the table visually; The multi-modal representation model comprises an input module, a data content embedding module, a data position embedding module, an encoding module, a decoding module and a feature output module; The input module is used for input of two modalities of text and table picture, and both are in sequence form; meanwhile, a sequence containing only a [start] tag is inputted; The data content embedding module comprises a text embedding module and a picture embedding module; the text embedding module is in the form of an Embedding embedding layer; the picture embedding module is a convolutional neural network layer, the first layer is a convolutional layer with a convolution kernel size of 3 and a step size of 1; the second layer is a convolutional layer with a convolution kernel size of 5 and a step size of 1; the third and fourth layers are convolutional layers with a convolution kernel size of 7 and a step size of 2; a residual block is added after the first and second convolutional layers, and a normalization layer, an activation layer and a residual block are sequentially added after the third and fourth convolutional layers; The data position embedding module is used for representing the relative positions of the text sequence and the picture sequence, wherein the text sequence adopts one-dimensional position, and the picture sequence arranges the original two-dimensional coordinates into one-dimensional position in the order from left to right and from top to bottom; The encoding module comprises a plurality of calculation units with the same structure, each calculation unit sequentially comprises a multi-head self-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer and a normalization layer; each addition operation layer and normalization layer is connected with a residual block; The multi-head self-attention module is composed of a plurality of self-attention modules, each of which independently extracts information from different representation subspaces; the self-attention module is an attention module with the same input vector, which is used for calculating the weighting between different feature maps; the local similarity between two different feature maps a and b is obtained by matrix multiplication and Softmax operation, and the result is multiplied by b to obtain a new feature map c with attention, which is used as the output of the attention module; The decoding module sequentially comprises a self-attention module, an addition operation layer, a normalization layer, a cross-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer and a normalization layer; each addition operation layer and normalization layer is connected with a residual block; the cross-attention module is an attention module with two different input vectors, one of which comes from the output of the encoding module, and the other of which comes from the output of the previous decoding module; The feature output module is the output of the last decoding module; Step 4: calculating the semantic distance of the vector representations from different data; Step 5: evaluating the semantic distance between different data to determine the alignment result.
2. The cross-domain cross-source data alignment method of claim 1, characterized in that: In step 1, the table data refers to data with "key-value pair" relationship, the value corresponding to the same key is unique or not unique, and the storage format of the table data is not limited, including Word, Excel, PDF, HTML.
3. The method of claim 1, wherein: In step 4, the semantic distance of the vector representation of data from different sources is calculated, and the cosine distance is used to calculate the distance between two vectors.
4. The method of claim 1, wherein: In step 5, the semantic distance between different data is evaluated according to the distance of the semantic distance calculated in step 4, and the matching result is sorted, and the nearest distance is taken as the data pair n ; wherein, n is a hyperparameter of the model.
5. The method of claim 1-4, wherein: The data multi-modal representation model is a trained data multi-modal representation model. The training process includes the following steps: (1) input several groups of table data; (2) extract the key-value pairs in the data and their positions in the table; (3) sample the key-value pairs, and consider the matching data pairs from different sources as positive examples, and the non-matching data pairs as negative examples; (4) use the data multi-modal representation model to obtain the text vector representation of the key and value, and the visual representation of the key and value in the table; (5) calculate the semantic distance between different key-value pairs, and calculate the prediction result of matching according to the semantic distance; (6) calculate the loss of the prediction result and the true positive and negative examples according to the predetermined loss function; The loss function is: wherein and is a pair of positive samples, and denotes all sample samples, is a hyperparameter of the data multi-modal representation model; (7) iteratively correct the data multi-modal representation model according to the loss function until the model converges, and obtain the trained data multi-modal representation model.
6. A cross-domain cross-source data alignment system, characterized in that, It includes the following modules: Module 1, for inputting multiple groups of table data to be aligned; Module 2, for extracting key-value pairs in the data and their positions in the table; Module 3, for using the data multi-modal representation model to generate vector representations of keys, values, and visual positions; The data multi-modal representation model is used to represent the key and value in the data from text and data in the table visually as vectors; The multi-modal representation model includes an input module, a data content embedding module, a data position embedding module, an encoding module, a decoding module, and a feature output module; The input module is used for input of two modalities of text and table picture, and both are in sequence form; meanwhile, a sequence containing only a [start] tag is inputted; The data content embedding module includes a text embedding module and a picture embedding module; the text embedding module is in the form of an Embedding embedding layer; the picture embedding module is a convolutional neural network layer, the first layer is a convolutional layer with a convolution kernel size of 3 and a step size of 1; the second layer is a convolutional layer with a convolution kernel size of 5 and a step size of 1; the third and fourth layers are convolutional layers with a convolution kernel size of 7 and a step size of 2; a residual block is added after the first and second convolutional layers, and a normalization layer, an activation layer, and a residual block are sequentially added after the third and fourth convolutional layers; The data position embedding module is used to represent the relative position of the text sequence and the picture sequence, wherein the text sequence adopts one-dimensional position, and the picture sequence arranges the original two-dimensional coordinates into one-dimensional position in the order from left to right and from top to bottom; The encoding module includes several calculation units with the same structure, each calculation unit includes a multi-head self-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer, and a normalization layer in sequence; each addition operation layer and normalization layer is connected with a residual block. The multi-head self-attention module is composed of several self-attention modules, each of which independently extracts information from different representation subspaces; the self-attention module is an attention module with the same vector as the input, which is used to calculate the weighting between different feature maps; taking different feature maps a and b, matrix multiplication and Softmax operation are performed on the two to obtain the local similarity between them, and the result is multiplied by b to obtain a new feature map c with attention as the output of the attention module; The decoding module sequentially includes a self-attention module, an addition operation layer, a normalization layer, a cross-attention module, an addition operation layer, a normalization layer, a feedforward network layer, an addition operation layer, and a normalization layer; each addition operation layer and normalization layer is connected with a residual block; the cross-attention module is an attention module with two different vectors as the input, one of which comes from the output of the encoding module, and the other comes from the output of the previous decoding module; The feature output module is the output of the last decoding module; Module 4 is used to calculate the semantic distance of the vector representation from different data; Module 5 is used to evaluate the semantic distance between different data to determine the alignment result.
7. An electronic device, comprising: Comprise: One or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the cross-domain cross-source data alignment method according to any one of claims 1 to 5.