Multi-modal data fusion method based on position sensitive optimization

Abstract

The invention relates to the field of computer data analysis and fusion, in particular to a position sensitive optimization-based multi-modal data fusion method, which comprises the following steps of: collecting data; preprocessing the data; labeling the processed data; the marked data information is extracted; calculating a specific vector value through the text information and the image information; substituting the specific vector value into a similarity formula, and calculating the similarity between the text information and the image information; fusing the information with high similarity between the text information and the image information; information promotion and matching of enterprises of the same type are realized; by optimizing the text Transform encoder and the image Transform encoder, key features and information in the image can be better captured, and a model is helped to effectively fuse the multi-modal data so as to obtain more comprehensive and accurate information.

Description

Technical Field

[0001] The present invention relates to the field of computer data analysis and fusion, and in particular to a method for multimodal data fusion based on position sensitive optimization. Background Art

[0002] In today's information age, data exists in various forms, including but not limited to text, images, videos, and audio. The same object or event can often be described in multiple modes, and each mode provides a unique perspective and information. In daily searches, people often need to enter key text or enter specific pictures to find target information that meets their requirements.

[0003] In the prior art, data search and fusion use a traditional CLIP model, which contains a text Transformer encoder and an image Transformer encoder. The position encoding in the text Transformer encoder is mainly used to help the model understand the relative or absolute position of each element in the text sequence, while the text Transformer encoder uses the original position encoding, which only has intra-segment encoding and cannot accurately capture the hierarchical structure and semantic information in the text, so that the model cannot understand the complex structure and meaning of the text. At the same time, the position encoding in the image Transformer encoder also uses a relatively primitive position encoding, and its original position encoding uses absolute position encoding or relative position encoding, which cannot well capture the spatial information and structural features in the image, so that the data obtained after analysis has certain differences, resulting in insufficient data fusion. Summary of the invention

[0004] The present invention provides a method for multimodal data fusion based on position-sensitive optimization. By optimizing the text Transformer encoder and the image Transformer encoder, the model can capture the hierarchical structure and semantic information in the text more finely, helping the model to understand the complex structure and meaning in the text. At the same time, the image Transformer encoder dynamically adjusts the representation of the position information according to the specific content and structure of the input image, so as to better capture the key features and information in the image, and help the model to effectively fuse these multimodal data to obtain more comprehensive and accurate information.

[0005] To achieve the above object, the present invention provides a method for multimodal data fusion based on position-sensitive optimization, comprising:

[0006] A method for multimodal data fusion based on position-sensitive optimization, characterized in that the method for multimodal data fusion based on position-sensitive optimization comprises the following steps:

[0007] S1. Data collection: The CLIP model obtains enterprise data information related to the enterprise from different channels. The enterprise data includes enterprise name, trademark, product pictures, and business scope. The CLIP model includes a text Transformer encoder and an image Transformer encoder.

[0008] S2. Preprocess the data: Use the data processing library in Python to preprocess the data;

[0009] S3. Annotate the processed data: Annotate the preprocessed data and upload it to a storage unit, wherein the data annotation method includes keyword highlighting extraction method recognition and image annotation software labelme;

[0010] S4. Extract the annotated data information: use the text Transformer encoder and the image Transformer encoder to analyze, extract and locate the annotated data in the storage unit, and obtain the embedding vector and projection matrix of the text and image, wherein the text Transformer encoder is provided with a double-layer position encoding, and the image Transformer encoder is provided with a dynamic position encoding, the double-layer position encoding obtains the position embedding vector in the text, and the dynamic position encoding obtains the vector sequence of the image in the image content;

[0011] S5. Calculate a specific vector value through text information and image information: Substitute the multiple feature vectors obtained above into the text Transformer encoder and the image Transformer encoder, calculate the mapped feature vectors in the text and image, and obtain the normalized feature vectors;

[0012] S6. Substitute the above-mentioned specific vector value into the similarity formula to calculate the similarity between the text information and the image information: use the text Transformer encoder and the image Transformer encoder to extract the text features and the image features respectively, combine N text features and N image features in pairs, and substitute them into the CLIP model to predict N2 possible text-image similarities, where the similarity is the cosine similarity of the text features and the image features, and the cosine similarity is obtained by matrix multiplication to obtain a similarity score, and the higher the similarity score, the higher the similarity between the text and the image;

[0013] S7. Fusion of highly similar text information and image information: Match highly similar text information and image information to accurately obtain various types of information about the enterprise and achieve accurate portrait;

[0014] S8. Realize information promotion and matching of similar enterprises: arrange examples based on text information and image information with high similarity.

[0015] Furthermore, the preprocessing model processes the enterprise data information, specifically including the following steps:

[0016] S21. For the enterprise data in the text part: use the text processing encoder to complete the enterprise data, modify the format and content, remove redundant data, and unify the terminology, abbreviations and data format of the text;

[0017] S22. For corporate data on images: by comparing the hash values ​​of the images, images with higher similarity are considered as duplicate images, and then the duplicate images are deleted. At the same time, a Gaussian filter or a median filter is used to remove random noise in the image. The clarity of the image edge is ensured while denoising. The SVM machine learning model is used to identify images that do not meet the standards, such as images with abnormal size and proportion. At the same time, the clarity, contrast, color saturation and other indicators of the image are detected, and low-quality images are deleted or marked.

[0018] Furthermore, labeling the pre-processed enterprise data specifically includes the following steps:

[0019] S31. For text-type enterprise data: using the keyword highlighting extraction method to identify, thereby highlighting key information in the text;

[0020] S32. For image-type enterprise data: using the image annotation software labelme, locate and annotate specific targets in the image, classify the image into predefined categories, and store them in the corresponding storage unit.

[0021] Furthermore, the text Transformer encoder and the image Transformer encoder analyze the enterprise data annotated in the storage unit and obtain different feature vectors, specifically including the following steps:

[0022] S41. Extracting textual enterprise data: The pre-entered natural language text will be segmented into a series of tokens (tags or words) that conform to the predefined vocabulary;

[0023] S42. Perform Token Embedding: Each of the tokens is converted into a high-dimensional vector representation by looking up the embedding matrix, wherein the embedding matrix is ​​a matrix of size (vocab_size, embedding_dim), where vocab_size is the size of the vocabulary, and embedding_dim is the dimension of the embedding vector, i.e., the dimension of the embedding vector is the text embedding vector Tf ;

[0024] S43. Double-layer position coding adds a position code to each of the tokens to indicate the position of the token in the sentence, thereby obtaining a text position embedding vector;

[0025] S44. Add the input token embedding vector and the position embedding vector, and the resulting value is sent to multiple self-attention heads in the text Transformer encoder. Each self-attention head maps the input vector to query, key, and value vectors through linear transformation, and then uses the scaled dot product attention mechanism to calculate the attention weight between each token and other tokens, and generates a weighted output vector, that is, the weighted output vector is the projection matrix W t ;

[0026] S45. Simultaneously concatenate the output vectors of multiple self-attention heads together, and obtain the final multi-head self-attention output through another set of linear transformations;

[0027] S46. The multi-head self-attention output is then passed through a feed-forward neural network to further extract and transform features;

[0028] S47. For image information analysis processing, each image block is flattened into a one-dimensional vector and mapped using a convolutional layer to convert it into a higher-dimensional vector space;

[0029] S48. In all vector sequences (image feature vector is I f ) to add a special classification token (Class Token) at the beginning of the image for subsequent image classification tasks;

[0030] S49. The image Transformer encoder adds a dynamic position code to each of the one-dimensional vector sequences;

[0031] S410. Input the vector sequence containing the class token and the dynamic position encoding into the image Transformer encoder, wherein the image Transformer encoder is composed of multiple self-attention layers and feedforward neural network layers, which are used to encode the input sequence. The self-attention layer can capture the mutual dependencies between the elements in the input sequence and perform weighted aggregation on the elements according to these dependencies, that is, the weighted aggregation is the projection matrix W t .

[0032] Furthermore, the calculation of the mapping feature vector generated by the text and the image specifically includes the following steps:

[0033] S51. Text feature vector: After being processed by the text Transformer encoder, each token corresponds to a high-dimensional feature vector, and in the CLIP model, the high-dimensional output vector corresponding to a specific token (such as the "[CLS]" tag) is used as the feature vector of the entire sentence. e Represents, or pooling the feature vectors of all tokens to get the overall feature vector T of the sentence e , where pooling is to generate the feature representation of the entire sentence by aggregating the feature information of all tokens in the sequence:

[0034] Assume that the text embedding vector is T f , the projection matrix is ​​W t , W is the weight matrix, which is the weight of each text corresponding position, then the mapped feature vector T e for:

[0035]

[0036] Get the normalized text feature vectors, which have unit length in the joint multimodal space, to facilitate the subsequent calculation of cosine similarity;

[0037] S52. Image feature vector: In the output sequence of the image Transformer encoder, take the vector corresponding to the first Class Token as the feature vector of the entire image. e :

[0038] Assume that the image feature vector is I f , the projection matrix is ​​W i , then the mapped feature vector I e :

[0039]

[0040] Finally, the normalized image feature vectors are obtained. These vectors have unit length in the joint multimodal space, which is convenient for the subsequent calculation of cosine similarity.

[0041] S53. The above mapped feature vector I e Input into a fully connected layer for classification prediction, and finally output the predicted enterprise type.

[0042] Furthermore, the calculating of the similarity between the text information and the image information specifically comprises the following steps:

[0043] S61. Using the text Transformer encoder and the image Transformer encoder, N text features and N image features can be extracted. By combining the N text features and the N image features in pairs, the CLIP model will predict N 2 The similarity of possible text-image pairs is the cosine similarity between text features and image features. The training goal of the CLIP model is to maximize the similarity of N positive samples while minimizing N 2 -Similarity of N negative samples;

[0044] The calculation of cosine similarity is achieved through matrix multiplication, and the result is a similarity score that represents the degree of match between the text and the image;

[0045] Assume that the feature vector after text mapping is T e , the feature vector after image mapping is I e , then the similarity score is:

[0046]

[0047] Among them, the higher the similarity score, the higher the similarity between the text and the image, that is, the more likely the text and the drawing are to match.

[0048] Furthermore, the highly similar text information and image information are matched and merged, specifically including the following steps:

[0049] S71. Calculating cosine similarity through matrix multiplication will generate N 2 samples, of which there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 -N text-image pairs are negative samples;

[0050] S72. The matching text information and image information are embedded into the same semantic space through contrastive learning, thereby realizing the fusion of cross-modal information.

[0051] Furthermore, the information promotion and matching of the same type of enterprises is realized, which specifically includes the following steps:

[0052] S81. The user inputs a text description related to the enterprise into the CLIP model, and the CLIP model can retrieve the image that best matches it (such as the enterprise logo, product image, etc.), thereby quickly acquiring enterprise information. Conversely, the user can also upload an image related to the enterprise (such as a product image), and the CLIP model can generate a text description related to it, thereby helping the user to understand the product information more intuitively.

[0053] S82. The remaining N when calculating cosine similarity 2 -N negative samples can be defined as companies of the same type that are similar to the company.

[0054] Compared with the prior art, the method for multimodal data fusion based on position-sensitive optimization according to an embodiment of the present invention sets the original position coding in the text Transformer encoder to double-layer position coding, and sets the original position coding of the image Transformer encoder to dynamic position coding, wherein the double-layer position coding in the text Transformer encoder can fuse intra-segment coding and inter-segment coding, so that the text data fits well with the intrinsic segmentation structure, and improves the understanding ability of the CLIP model, so that the CLIP model acquires text information more accurately, and the dynamic position coding in the image Transformer encoder can dynamically adjust the position coding according to the specific content of the input image, thereby improving the flexibility of the CLIP model and matching corporate information more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A basic flow chart of a method for multimodal data fusion based on position-sensitive optimization of the present invention;

[0056] Figure 2 A flowchart of a text Transformer encoder in a method for multimodal data fusion based on position-sensitive optimization of the present invention;

[0057] Figure 3 A flowchart of an image Transformer encoder in a method for multimodal data fusion based on position-sensitive optimization of the present invention; and

[0058] Figure 4 The present invention provides a flowchart for fusing text information and image information in a method for multimodal data fusion based on position-sensitive optimization. DETAILED DESCRIPTION

[0059] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0060] like Figure 1 As shown, a method for multimodal data fusion based on position-sensitive optimization provided in an embodiment of the present invention includes the following steps:

[0061] S1. Data collection: The CLIP model obtains enterprise data information related to the enterprise from different channels. The enterprise data information includes enterprise name, trademark, product pictures, enterprise introduction text and business scope. The CLIP model includes a text Transformer encoder and an image Transformer encoder.

[0062] S2. Preprocess the data: Use the data processing library in Python to preprocess the data;

[0063] S3. Annotate the processed data: Annotate the preprocessed data and upload it to a storage unit, wherein the data annotation method includes keyword highlighting extraction method recognition and image annotation software labelme;

[0064] S4. Extract the annotated data information: Use a text Transformer encoder and an image Transformer encoder to analyze, extract and locate the annotated data in the storage unit, and obtain the embedding vectors and projection matrices of the text and image, wherein the text Transformer encoder is provided with a double-layer position encoding, and the image Transformer encoder is provided with a dynamic position encoding, wherein the double-layer position encoding obtains a position embedding vector in the text, and the dynamic position encoding obtains a vector sequence of the image in the image content;

[0065] S5. Calculate a specific vector value through text information and image information: Substitute the multiple feature vectors obtained above into the data model in the text Transformer encoder and the image Transformer encoder, calculate the mapped feature vectors in the text and the image, and obtain the normalized feature vector;

[0066] S6. Substitute the above-mentioned specific vector value into the similarity formula to calculate the similarity between the text information and the image information: use the text Transformer encoder and the image Transformer encoder to extract the text features and the image features respectively, combine N text features and N image features in pairs, and substitute them into the CLIP model to predict N2 possible text-image similarities, where the similarity is the cosine similarity of the text features and the image features, and the cosine similarity is obtained by matrix multiplication to obtain a similarity score, and the higher the similarity score, the higher the similarity between the text and the image;

[0067] S7. Fusion of highly similar text information and image information: Match highly similar text information and image information to accurately obtain various types of information about the enterprise and achieve accurate portrait;

[0068] S8. Realize information promotion and matching of similar enterprises: arrange examples based on text information and image information with high similarity.

[0069] In an embodiment of the present invention, it is necessary to analyze the different collected data, such as finding similar products from a company introduction text, or finding products with similar functions from multiple products. At this time, it is necessary to process the text information and image information to facilitate subsequent annotation processing.

[0070] In one embodiment of the present invention, the preprocessing model processes the enterprise data information, specifically comprising the following steps:

[0071] S21. For the enterprise data in the text: use the data processing library in Python to complete the enterprise data, modify the format and content, remove redundant data, and unify the terminology, abbreviations and data format of the text;

[0072] S22. For corporate data on images: by comparing the hash values ​​of the images, images with higher similarity are considered as duplicate images, and then the duplicate images are deleted. At the same time, a Gaussian filter or a median filter is used to remove random noise in the image. The clarity of the image edge is ensured while denoising. The SVM machine learning model is used to identify images that do not meet the standards, such as images with abnormal size and proportion. At the same time, the clarity, contrast, color saturation and other indicators of the image are detected, and low-quality images are deleted or marked.

[0073] Among them, the data processing libraries in Python mostly use numpy and pandas to remove irrelevant characters in the text, such as HTML tags, special symbols, etc.; unify the terms, abbreviations, and number formats in the text for subsequent processing; and divide the text into words or phrases to facilitate the extraction of key information.

[0074] In one embodiment of the present invention, labeling the pre-processed enterprise data specifically includes the following steps:

[0075] S31. For text-type enterprise data: using the keyword highlighting extraction method to identify, thereby highlighting key information in the text;

[0076] S32. For image-type enterprise data: using the image annotation software labelme, locate and annotate specific targets in the image, classify the image into predefined categories, and store them in the corresponding storage unit.

[0077] Among them, text type information highlights the key information in the text, allowing users to understand and grasp the main content more quickly, while image type information locates and marks specific targets such as corporate logos, product details, corporate brochures, corporate honorary certificates, etc., and predefined categories include technology, finance, service, production, culture, etc.

[0078] In one embodiment of the present invention, Figure 2 and Figure 3 As shown, the text Transformer encoder and the image Transformer encoder analyze the enterprise data annotated in the storage unit and obtain different feature vectors, specifically including the following steps:

[0079] S41. Figure 2 As shown, extracting text enterprise data: the pre-entered natural language text will be segmented into a series of tokens (tags or words) that conform to the predefined vocabulary;

[0080] S42. Perform Token Embedding: Each of the tokens is converted into a high-dimensional vector representation by looking up the embedding matrix, wherein the embedding matrix is ​​a matrix of size (vocab_size, embedding_dim), where vocab_size is the size of the vocabulary, and embedding_dim is the dimension of the embedding vector, i.e., the dimension of the embedding vector is the text embedding vector T f ;

[0081] S43. Double-layer position coding adds a position code to each of the tokens to indicate the position of the token in the sentence, thereby obtaining a text position embedding vector;

[0082] S44. Add the input token embedding vector and position embedding vector, and send the obtained value to multiple self-attention heads in the text Transformer encoder. Each self-attention head maps the input vector to query, key, and value vectors through linear transformation, and then uses the scaled dot product attention mechanism to calculate the attention weight between each token and other tokens, and generates a weighted output vector, that is, the weighted output vector is the projection matrix W t ;

[0083] S45. Concatenate the output vectors of multiple self-attention heads together and obtain the final multi-head self-attention output through another set of linear transformations;

[0084] S46. The multi-head self-attention output is then passed through a feed-forward neural network to further extract and transform features;

[0085] S47. Figure 3 As shown, for image information analysis processing, each image block is flattened into a one-dimensional vector and mapped using a convolutional layer to convert it into a higher-dimensional vector space;

[0086] S48. In all vector sequences (image feature vector is I f ) to add a special classification token (Class Token) at the beginning of the image for subsequent image classification tasks;

[0087] S49. The image Transformer encoder adds a dynamic position code to each of the one-dimensional vector sequences;

[0088] S410. Input the vector sequence containing the class token and the dynamic position encoding into the image Transformer encoder. The image Transformer encoder consists of multiple self-attention layers and feedforward neural network layers to encode the input sequence. The self-attention layer can capture the mutual dependencies between the elements in the input sequence and perform weighted aggregation on the elements according to these dependencies, that is, the weighted aggregation is the projection matrix W t .

[0089] in, Figure 2 The "Residual Connection & Normalization" step in the above example is used to stabilize the data obtained in the previous step. The remaining data is calculated through the data model in the image Transformer encoder and the text Transformer encoder, the text embedding vector Tf of the text type information and the projection matrix W of the text. t , the image feature vector of image type information is I f and the image projection matrix W t .

[0090] In one embodiment of the present invention, calculating the mapping feature vector generated by the text and the image specifically includes the following steps:

[0091] S51. Text feature vector: After being processed by the text Transformer encoder, each token corresponds to a high-dimensional feature vector, and in the CLIP model, the high-dimensional output vector corresponding to a specific token (such as the "[CLS]" tag) is used as the feature vector of the entire sentence. e Represents, or pooling the feature vectors of all tokens to get the overall feature vector T of the sentence e , where pooling is to generate the feature representation of the entire sentence by aggregating the feature information of all tokens in the sequence:

[0092] Assume that the text embedding vector is T f , the projection matrix is ​​W t , W is the weight matrix, which is the weight of each text corresponding position, then the mapped feature vector T e for:

[0093]

[0094] Get the normalized text feature vectors, which have unit length in the joint multimodal space, to facilitate the subsequent calculation of cosine similarity;

[0095] S52. Image feature vector: In the output sequence of the image Transformer encoder, take the vector corresponding to the first Class Token as the feature vector of the entire image. e :

[0096] Assume that the image feature vector is I f , the projection matrix is ​​W i , then the mapped feature vector I e :

[0097]

[0098] Finally, the normalized image feature vectors are obtained. These vectors have unit length in the joint multimodal space, which is convenient for the subsequent calculation of cosine similarity.

[0099] S53. The above mapped feature vector I e Input into a fully connected layer for classification prediction, and finally output the predicted enterprise type.

[0100] In one embodiment of the present invention, the mapping feature vectors generated by the text and the image obtained by the above steps are used to calculate the similarity between the text and the image, which specifically includes the following steps:

[0101] S61. Using the text Transformer encoder and the image Transformer encoder, N text features and N image features can be extracted. By combining the N text features and the N image features in pairs, the CLIP model will predict N 2 The similarity of possible text-image pairs is the cosine similarity between text features and image features. The training goal of the CLIP model is to maximize the similarity of N positive samples while minimizing N 2 -Similarity of N negative samples;

[0102] The calculation of cosine similarity is achieved through matrix multiplication, and the result is a similarity score that represents the degree of match between the text and the image;

[0103] Assume that the feature vector after text mapping is T e , the feature vector after image mapping is I e , then the similarity score is:

[0104]

[0105] Among them, the higher the similarity score, the higher the similarity between the text and the image, that is, the more likely the text and the drawing are to match.

[0106] In one embodiment of the present invention, Figure 4 As shown, highly similar text information and image information are matched and fused, specifically including the following steps:

[0107] S71. Calculating cosine similarity through matrix multiplication will generate N 2 samples, of which there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 -N text-image pairs are negative samples;

[0108] S72. The matching text information and image information are embedded into the same semantic space through contrastive learning, thereby realizing the fusion of cross-modal information and forming a training data set.

[0109] In one embodiment of the present invention, information promotion and matching of enterprises of the same type are realized, specifically including the following steps:

[0110] S81. The user inputs a text description related to the enterprise into the CLIP model, and the CLIP model can retrieve the image that best matches it (such as the enterprise logo, product image, etc.), thereby quickly acquiring enterprise information. Conversely, the user can also upload an image related to the enterprise (such as a product image), and the CLIP model can generate a text description related to it, thereby helping the user to understand the product information more intuitively.

[0111] S82. The remaining N when calculating cosine similarity 2 -N negative samples can be defined as companies of the same type that are similar to the company;

[0112] In this way, when users input technology-based enterprises, they will not only be matched with the enterprises we hope to match in the training data set, but also with other technology-based enterprises.

[0113] In the description of the present invention, it is necessary to understand that the terms “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside” and “outside” orientations or positional relationships are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0114] The present invention has been described by the above-mentioned relevant embodiments, however, the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. A method for multimodal data fusion based on position-sensitive optimization, characterized in that: The method for multimodal data fusion based on position-sensitive optimization comprises the following steps: S1. Data collection: The CLIP model obtains enterprise data information related to the enterprise from different channels. The enterprise data includes enterprise name, trademark, product pictures, and business scope. The CLIP model includes a text Transformer encoder and an image Transformer encoder. S2. Preprocess the data: Use the data processing library in Python to preprocess the data; S3. Annotate the processed data: extract the preprocessed data using annotations and upload them to a storage unit, wherein the data annotation method includes keyword highlighting extraction method recognition and image annotation software labelme; S4. Extract the annotated data information: use the text Transformer encoder and the image Transformer encoder to analyze, extract and locate the annotated data in the storage unit, and obtain the embedding vector and projection matrix of the text and image, wherein the text Transformer encoder is provided with a double-layer position encoding, and the image Transformer encoder is provided with a dynamic position encoding, the double-layer position encoding obtains the position embedding vector in the text, and the dynamic position encoding obtains the vector sequence of the image in the image content; S5. Calculate a specific vector value through text information and image information: Substitute the multiple feature vectors obtained above into the data model in the text Transformer encoder and the image Transformer encoder, calculate the mapped feature vectors in the text and the image, and obtain the normalized feature vector; S6. Substitute the above-mentioned specific vector value into the similarity formula to calculate the similarity between the text information and the image information: use the text Transformer encoder and the image Transformer encoder to extract the text features and the image features respectively, combine N text features and N image features in pairs, and substitute them into the CLIP model to predict N2 possible text-image similarities, where the similarity is the cosine similarity of the text features and the image features, and the cosine similarity is obtained by matrix multiplication to obtain a similarity score, and the higher the similarity score, the higher the similarity between the text and the image; S7. Fusion of highly similar text information and image information: Match highly similar text information and image information to accurately obtain various types of enterprise information and achieve accurate portrait; S8. Realize information promotion and matching of similar enterprises: arrange examples based on text information and image information with high similarity.

2. The method for multimodal data fusion based on position-sensitive optimization according to claim 1, characterized in that: The preprocessing model processes enterprise data information, specifically including the following steps: S21. For the enterprise data in the text part: use the text processing encoder to complete the enterprise data, modify the format and content, remove redundant data, and unify the terminology, abbreviations and data format of the text; S22. For corporate data on images: by comparing the hash values ​​of the images, images with higher similarity are considered as duplicate images, and then the duplicate images are deleted. At the same time, a Gaussian filter or a median filter is used to remove random noise in the image. The clarity of the image edge is ensured while denoising. The SVM machine learning model is used to identify images that do not meet the standards, such as images with abnormal size and proportion. At the same time, the clarity, contrast, color saturation and other indicators of the image are detected, and low-quality images are deleted or marked.

3. The method for multimodal data fusion based on position-sensitive optimization according to claim 1, characterized in that: Labeling the pre-processed enterprise data specifically includes the following steps: S31. For text-type enterprise data: using the keyword highlighting extraction method to identify, thereby highlighting key information in the text; S32. For image-type enterprise data: using the image annotation software labelme, locate and annotate specific targets in the image, classify the image into predefined categories, and store them in the corresponding storage unit.

4. The method for multimodal data fusion based on position-sensitive optimization according to claim 1, characterized in that: The text Transformer encoder and the image Transformer encoder analyze the enterprise data annotated in the storage unit and obtain different feature vectors, specifically including the following steps: S41. Extracting textual enterprise data: The pre-entered natural language text will be segmented into a series of tokens (tags or words) that conform to the predefined vocabulary; S42. Perform Token Embedding: Each of the tokens is converted into a high-dimensional vector representation by looking up the embedding matrix, wherein the embedding matrix is ​​a matrix of size (vocab_size, embedding_dim), where vocab_size is the size of the vocabulary, and embedding_dim is the dimension of the embedding vector, i.e., the dimension of the embedding vector is the text embedding vector T f ; S43. Double-layer position coding adds a position code to each of the tokens to indicate the position of the token in the sentence, thereby obtaining a text position embedding vector; S44. Add the input token embedding vector and the position embedding vector, and the resulting value is sent to multiple self-attention heads in the text Transformer encoder. Each self-attention head maps the input vector to query, key, and value vectors through linear transformation, and then uses the scaled dot product attention mechanism to calculate the attention weight between each token and other tokens, and generates a weighted output vector, that is, the weighted output vector is the projection matrix W t ; S45. Simultaneously concatenate the output vectors of multiple self-attention heads together, and obtain the final multi-head self-attention output through another set of linear transformations; S46. The multi-head self-attention output is then passed through a feed-forward neural network to further extract and transform features; S47. For image information analysis processing, each image block is flattened into a one-dimensional vector and mapped using a convolutional layer to convert it into a higher-dimensional vector space; S48. In all vector sequences (image feature vector is I f ) to add a special classification token (ClassToken) at the beginning of the image for subsequent image classification tasks; S49. The image Transformer encoder adds a dynamic position code to each of the one-dimensional vector sequences; S410. Input the vector sequence containing the class token and the dynamic position encoding into the image Transformer encoder, wherein the image Transformer encoder is composed of multiple self-attention layers and feedforward neural network layers, which are used to encode the input sequence. The self-attention layer can capture the mutual dependencies between the elements in the input sequence and perform weighted aggregation on the elements according to these dependencies, that is, the weighted aggregation is the projection matrix W t .

5. The method for multimodal data fusion based on position-sensitive optimization according to claim 1, characterized in that: The calculation of the mapping feature vector generated by the text and the image specifically includes the following steps: S51. Text feature vector: After being processed by the text Transformer encoder, each token corresponds to a high-dimensional feature vector, and in the CLIP model, the high-dimensional output vector corresponding to a specific token (such as the "[CLS]" tag) is used as the feature vector of the entire sentence. e Represents, or pooling the feature vectors of all tokens to get the overall feature vector T of the sentence e , where pooling is to generate the feature representation of the entire sentence by aggregating the feature information of all tokens in the sequence: Assume that the text embedding vector is T f , the projection matrix is ​​W t , W is the weight matrix, which is the weight of each text corresponding position, then the mapped feature vector T e for: Get the normalized text feature vectors, which have unit length in the joint multimodal space, to facilitate the subsequent calculation of cosine similarity; S52. Image feature vector: In the output sequence of the image Transformer encoder, take the vector corresponding to the first Class Token as the feature vector of the entire image. e : Assume that the image feature vector is I f , the projection matrix is ​​W i , then the mapped feature vector I e : Finally, the normalized image feature vectors are obtained. These vectors have unit length in the joint multimodal space, which is convenient for the subsequent calculation of cosine similarity. S53. The above mapped feature vector I e Input into a fully connected layer for classification prediction, and finally output the predicted enterprise type.

6. The method for multimodal data fusion based on position-sensitive optimization according to claim 1, characterized in that: The calculating of the similarity between the text information and the image information specifically comprises the following steps: S61. Using the text Transformer encoder and the image Transformer encoder, N text features and N image features can be extracted. By combining the N text features and the N image features in pairs, the CLIP model will predict N 2 The similarity of possible text-image pairs is the cosine similarity between text features and image features. The training goal of the CLIP model is to maximize the similarity of N positive samples while minimizing N 2 -Similarity of N negative samples; The calculation of cosine similarity is achieved through matrix multiplication, and the result is a similarity score that represents the degree of match between the text and the image; Assume that the feature vector after text mapping is T e , the feature vector after image mapping is I e , then the similarity score is: Among them, the higher the similarity score, the higher the similarity between the text and the image, that is, the more likely the text and the drawing are to match.

7. The method for multimodal data fusion based on position-sensitive optimization according to claim 1, characterized in that: Matching and fusing highly similar text information and image information includes the following steps: S71. Calculating cosine similarity through matrix multiplication will generate N 2 samples, of which there are N positive samples, that is, text and images that truly belong to a pair (diagonal elements in the matrix), and the remaining N 2 -N text-image pairs are negative samples; S72. The matching text information and image information are embedded into the same semantic space through contrastive learning, thereby realizing the fusion of cross-modal information.

8. The method for multimodal data fusion based on position-sensitive optimization according to claim 7, characterized in that: To achieve information promotion and matching of enterprises of the same type, the following steps are specifically included: S81. The user inputs a text description related to the enterprise into the CLIP model, and the CLIP model can retrieve the image that best matches it (such as the enterprise logo, product image, etc.), thereby quickly acquiring enterprise information. Conversely, the user can also upload an image related to the enterprise (such as a product image), and the CLIP model can generate a text description related to it, thereby helping the user to understand the product information more intuitively. S82. The remaining N when calculating cosine similarity 2 -N negative samples can be defined as companies of the same type that are similar to the company.

Images