A method for multimodal data fusion based on location-sensitive optimization

Abstract

This invention relates to the field of computer data analysis and fusion, specifically to a method for multimodal data fusion based on location-sensitive optimization, comprising the following steps: data collection; data preprocessing; data annotation; extraction of information from the annotated data; calculation of specific vector values ​​using text and image information; substituting the specific vector values ​​into a similarity formula to calculate the similarity between text and image information; fusion of text and image information with high similarity; promotion and matching of information from similar enterprises; and optimization of the text Transformer encoder and image Transformer encoder to better capture key features and information in the image, helping the model effectively fuse these multimodal data to obtain more comprehensive and accurate information.

Description

Technical Field

[0001] This invention relates to the field of computer data analysis and fusion, and in particular to a method for multimodal data fusion based on location-sensitive optimization. Background Technology

[0002] In today's information age, data exists in many forms, including but not limited to text, images, videos, and audio. The same object or event can often be described in multiple modalities, and each modal provides a unique perspective and information. In daily searches, people often need to enter key text or specific images to find target information that meets their requirements.

[0003] In existing technologies, data search and fusion employ the traditional CLIP model. This model contains a text Transformer encoder and an image Transformer encoder. The positional encoding in the text Transformer encoder primarily helps the model understand the relative or absolute positions of elements within a text sequence. However, the text Transformer encoder uses primitive positional encoding, which only provides intra-segment encoding and cannot precisely capture the hierarchical structure and semantic information within the text. Consequently, the model cannot understand the complex structures and meanings within the text. Similarly, the image Transformer encoder also uses relatively primitive positional encoding, employing absolute or relative positional encoding methods. This also fails to effectively capture the spatial information and structural features of the image, resulting in discrepancies in the analyzed data and leading to insufficient data fusion. Summary of the Invention

[0004] This invention provides a method for multimodal data fusion based on position-sensitive optimization. By optimizing the text Transformer encoder and the image Transformer encoder, the model can capture the hierarchical structure and semantic information in the text more precisely, helping the model understand the complex structure and meaning in the text. At the same time, the image Transformer encoder dynamically adjusts the representation of positional information according to the specific content and structure of the input image, thereby better capturing the key features and information in the image. This helps the model effectively fuse these multimodal data to obtain more comprehensive and accurate information.

[0005] To achieve the above objectives, the present invention provides a method for multimodal data fusion based on location-sensitive optimization, comprising:

[0006] A method for multimodal data fusion based on location-sensitive optimization, characterized in that the method includes the following steps:

[0007] S1. Data Collection: The CLIP model obtains enterprise data information related to the enterprise from different channels. The enterprise data includes the enterprise name, trademarks, product images, and business scope. The CLIP model includes a text Transformer encoder and an image Transformer encoder.

[0008] S2. Data preprocessing: Use data processing libraries in Python to preprocess the data;

[0009] S3. Label the processed data: Label the preprocessed data and upload it to the storage unit. The labeling methods include keyword highlighting and extraction methods and image labeling software labelme.

[0010] S4. Extracting labeled data information: Using the text Transformer encoder and the image Transformer encoder, the labeled data in the storage unit is analyzed, extracted, and located to obtain the embedding vectors and projection matrices of the text and image. The text Transformer encoder is equipped with double-layer position encoding, and the image Transformer encoder is equipped with dynamic position encoding. The double-layer position encoding yields the position embedding vector in the text, and the dynamic position encoding yields the vector sequence of the image in the image content.

[0011] S5. Calculate specific vector values ​​using text and image information: Substitute the multiple feature vectors obtained above into the text Transformer encoder and the image Transformer encoder to calculate the mapped feature vectors in the text and image, and obtain the normalized feature vectors.

[0012] S6. Substitute the normalized feature vectors into the similarity formula to calculate the similarity between text and image information: Use a text Transformer encoder and an image Transformer encoder to extract text and image features respectively, combine N text features and N image features in pairs, and substitute them into the CLIP model to predict N. 2 The similarity of a possible text-image is calculated as the cosine similarity between the text features and the image features. The cosine similarity score is obtained by matrix multiplication. The higher the similarity score, the higher the similarity between the text and the image.

[0013] S7. Integrate highly similar text and image information: Match highly similar text and image information to accurately obtain various types of information about the enterprise and achieve accurate profiling;

[0014] S8. Achieve information promotion and matching for similar enterprises: Arrange examples based on text and image information with high similarity.

[0015] Furthermore, S2 specifically includes the following steps:

[0016] S21. For enterprise data in the text section: Use a text processing encoder to complete, modify the format and content of the enterprise data, remove redundant data, and standardize the terminology, abbreviations and data formats of the text;

[0017] S22. For enterprise data in the image section: By comparing the hash values ​​of the images, images with high similarity are considered as duplicate images and then deleted. At the same time, Gaussian filters or median filters are used to remove random noise in the images, ensuring the clarity of image edges while denoising. An SVM machine learning model is used to identify images that do not meet the standards, and the image's clarity, contrast, and color saturation indicators are detected. Low-quality images are deleted or marked.

[0018] Furthermore, S3 specifically includes the following steps:

[0019] S31. For enterprise data of text type: the keyword highlighting extraction method is used to identify and highlight key information in the text;

[0020] S32. For enterprise data of image type: Use the image annotation software labelme to locate and annotate specific targets in the image, classify the image into a predefined category, and store it in the corresponding storage unit.

[0021] Furthermore, S4 specifically includes the following steps:

[0022] S41. Extract enterprise data from text: The pre-input natural language text will be segmented into a series of tokens that conform to a predefined vocabulary;

[0023] S42. Perform embedding tagging: Each token is converted into a high-dimensional vector representation by looking up an embedding matrix, wherein the embedding matrix is ​​a matrix of size vocab_size × embedding_dim, where vocab_size is the size of the vocabulary and embedding_dim is the dimension of the embedding vector, i.e., the dimension of the embedding vector is the text embedding vector T. f ;

[0024] S43. Two-layer positional encoding adds a positional code to each token to represent the position of the token in the sentence, thereby deriving a text positional embedding vector;

[0025] S44. The input token embedding vector and position embedding vector are added together, and the resulting value is fed into multiple self-attention heads within the text Transformer encoder. Each self-attention head maps the input vector to a query, key, and value vector through a linear transformation. Then, a scaled dot product attention mechanism is used to calculate the attention weights between each token and other tokens, generating a weighted output vector, i.e., the weighted output vector is the projection matrix. ;

[0026] S45. Simultaneously concatenate the output vectors of multiple self-attention heads together, and obtain the final multi-head self-attention output through another set of linear transformations;

[0027] S46. The multi-head self-attention output is then further processed and transformed through a feedforward neural network;

[0028] S47. For image information analysis and processing, each image patch is flattened into a one-dimensional vector and mapped using a convolutional layer to transform it into a higher-dimensional vector space;

[0029] S48. Add a special token, called the Class Token, to the beginning of all vector sequences. The Class Token is used for subsequent image classification tasks.

[0030] S49. The image Transformer encoder adds a dynamic positional code to each of the higher-dimensional vector spaces;

[0031] S410. Input the vector sequence containing the Class Token and dynamic position encoding into the image Transformer encoder, which consists of multiple self-attention layers and feedforward neural network layers to encode the input sequence. The self-attention layers can capture the dependencies between elements in the input sequence and perform weighted aggregation of the elements according to these dependencies, i.e., weighted aggregation into a projection matrix. .

[0032] Furthermore, S5 specifically includes the following steps:

[0033] S51. Text Feature Vector: After processing by the text Transformer encoder, each token corresponds to a high-dimensional feature vector. In the CLIP model, the high-dimensional output vector corresponding to a specific token is used as the feature vector for the entire sentence. This means that, or pooling the feature vectors of all tokens, we can obtain the overall feature vector of the sentence. Pooling, in particular, generates the feature representation of the entire sentence by aggregating the feature information of all tokens in the sequence.

[0034] Assume the text embedding vector is The projection matrix is , Let the weight matrix be the weight of each position in the text, then the mapped feature vector is... for:

[0035]

[0036] We obtain normalized text feature vectors, which have unit length in the joint multimodal space, making it easier to calculate cosine similarity later.

[0037] S52. Image Feature Vector: In the output sequence of the image Transformer encoder, extract the vector corresponding to the Class Token at the first position as the feature vector of the entire image. :

[0038] Assume the image feature vector is The projection matrix is Then the mapped feature vector :

[0039]

[0040] Finally, we obtain the normalized image feature vectors, which have unit length in the joint multimodal space, making it convenient to calculate the cosine similarity later.

[0041] S53. The mapped feature vectors described above The input is fed into a fully connected layer for classification prediction, and the final output is the predicted enterprise type.

[0042] Furthermore, the calculation of the similarity between text information and image information specifically includes the following steps:

[0043] S61. Using a text Transformer encoder and an image Transformer encoder, N text features and N image features can be extracted. By combining the N text features and N image features in pairs, the CLIP model will predict N. 2 The CLIP model calculates the similarity of N possible text-image pairs, where the similarity is the cosine similarity between the text features and the image features. The training objective of the CLIP model is to maximize the similarity of N positive samples while minimizing N... 2 - The similarity of N negative samples;

[0044] Cosine similarity is calculated using matrix multiplication, and the result is a similarity score, which represents the degree of matching between text and image.

[0045] Assume the feature vector after text mapping is The feature vector after image mapping is The similarity score is:

[0046] Cosine_simlarity

[0047] The higher the similarity score, the higher the similarity between the text and the image, meaning the more likely the text and the drawing are to match.

[0048] Furthermore, S7 specifically includes the following steps:

[0049] S71. Calculating cosine similarity using matrix multiplication will generate N. 2 There are N samples, of which N are positive samples, i.e., the text and image that truly belong to a pair, and the remaining N are negative samples. 2 - N text-image pairs are negative samples;

[0050] S72. By embedding the matched text and image information into the same semantic space through contrastive learning, cross-modal information fusion is achieved.

[0051] Furthermore, to achieve information promotion and matching for similar enterprises, the specific steps include:

[0052] S81. The user inputs a text description related to the enterprise into the CLIP model, and the CLIP model can retrieve the most matching image to quickly obtain enterprise information. Conversely, the user can also upload an image related to the enterprise, and the CLIP model can generate a text description related to it to help the user understand product information more intuitively.

[0053] S82. The remaining N when calculating cosine similarity 2 -N negative samples can be defined as similar enterprises of the same type as the enterprise.

[0054] Compared with existing technologies, the multimodal data fusion method based on position-sensitive optimization according to embodiments of the present invention sets the original position encoding in the text Transformer encoder to a dual-layer position encoding, while setting the original position encoding in the image Transformer encoder to a dynamic position encoding. The dual-layer position encoding in the text Transformer encoder can fuse intra-segment encoding and inter-segment encoding, making the text data fit the inherent segmentation structure well and improving the understanding ability of the CLIP model, enabling the CLIP model to obtain text information more accurately. Meanwhile, the dynamic position encoding in the image Transformer encoder can dynamically adjust the position encoding according to the specific content of the input image, improving the flexibility of the CLIP model and thus more accurately matching enterprise information. Attached Figure Description

[0055] Figure 1 This is a basic flowchart of a multimodal data fusion method based on location-sensitive optimization according to the present invention;

[0056] Figure 2 This is a flowchart of the text Transformer encoder in a location-sensitive optimization-based multimodal data fusion method of the present invention;

[0057] Figure 3 This is a flowchart of an image Transformer encoder in a position-sensitive optimization-based multimodal data fusion method of the present invention; and

[0058] Figure 4 This is a flowchart illustrating the fusion of text and image information in a location-sensitive optimization-based multimodal data fusion method according to the present invention. Detailed Implementation

[0059] To provide a further understanding of the purpose, structure, features, and functions of the present invention, detailed descriptions are provided below with reference to specific embodiments.

[0060] like Figure 1 As shown, a method for multimodal data fusion based on location-sensitive optimization according to an embodiment of the present invention includes the following steps:

[0061] S1. Data Collection: The CLIP model acquires enterprise-related data information from different channels, including enterprise name, trademark, product images, enterprise introduction text, and business scope. The CLIP model includes a text Transformer encoder and an image Transformer encoder.

[0062] S2. Data preprocessing: Use data processing libraries in Python to preprocess the data;

[0063] S3. Label the processed data: Label the preprocessed data and upload it to the storage unit. The labeling methods include keyword highlighting and extraction methods and image labeling software labelme.

[0064] S4. Extracting labeled data information: Using a text Transformer encoder and an image Transformer encoder, the labeled data in the storage unit is analyzed, extracted, and located to obtain the embedding vectors and projection matrices of the text and image. The text Transformer encoder is equipped with a two-layer position encoding, and the image Transformer encoder is equipped with a dynamic position encoding. The two-layer position encoding yields a position embedding vector in the text, and the dynamic position encoding yields a vector sequence of the image in the image content.

[0065] S5. Calculate specific vector values ​​using text and image information: Substitute the multiple feature vectors obtained above into the data models of the text Transformer encoder and the image Transformer encoder to calculate the mapped feature vectors in the text and image, and obtain the normalized feature vectors.

[0066] S6. Substitute the specific vector values ​​mentioned above into the similarity formula to calculate the similarity between text information and image information: Use a text Transformer encoder and an image Transformer encoder to extract text features and image features respectively, combine N text features and N image features in pairs, and substitute them into the CLIP model to predict N. 2 The similarity of a possible text-image is calculated as the cosine similarity between the text features and the image features. The cosine similarity score is obtained by matrix multiplication. The higher the similarity score, the higher the similarity between the text and the image.

[0067] S7. Integrate highly similar text and image information: Match highly similar text and image information to accurately obtain various types of information about the enterprise and achieve accurate profiling;

[0068] S8. Achieve information promotion and matching for similar enterprises: Arrange examples based on text and image information with high similarity.

[0069] In this embodiment of the invention, it is necessary to analyze the different data collected, such as finding similar products from the company introduction text, or finding products with similar functions from multiple products. At this time, it is necessary to process the text information and image information to facilitate subsequent annotation processing.

[0070] In one embodiment of the present invention, step S2 specifically includes the following steps:

[0071] S21. For the enterprise data in the text section: Use data processing libraries in Python to perform operations such as completing the enterprise data, modifying the format and content, and removing redundant data, and standardize the terminology, abbreviations and data formats of the text;

[0072] S22. For enterprise data in the image section: By comparing the hash values ​​of the images, images with high similarity are considered as duplicate images and then deleted. At the same time, Gaussian filters or median filters are used to remove random noise in the images, ensuring the clarity of image edges while denoising. An SVM machine learning model is used to identify images that do not meet the standards, such as images with abnormal size or proportions. Simultaneously, the image's clarity, contrast, color saturation, and other indicators are detected, and low-quality images are deleted or marked.

[0073] In Python, data processing libraries often use NumPy and Pandas to remove irrelevant characters from text, such as HTML tags and special symbols; standardize terminology, abbreviations, and number formats in text for easier subsequent processing; and segment text into words or phrases to facilitate the extraction of key information.

[0074] In one embodiment of the present invention, S3 specifically includes the following steps:

[0075] S31. For enterprise data of text type: the keyword highlighting extraction method is used to identify and highlight key information in the text;

[0076] S32. For enterprise data of image type: Use the image annotation software labelme to locate and annotate specific targets in the image, classify the image into a predefined category, and store it in the corresponding storage unit.

[0077] Among them, text-type information highlights key information in the text, enabling users to understand and grasp the main content more quickly, while image-type information locates and labels specific targets such as company logos, product details, company brochures, and company honor certificates, etc., and the predefined categories include technology, finance, service, production, and culture.

[0078] In one embodiment of the present invention, such as Figure 2 and Figure 3 As shown, the text Transformer encoder and the image Transformer encoder analyze the labeled enterprise data in the storage unit and derive different feature vectors, specifically including the following steps:

[0079] S41. For example Figure 2 As shown, extracting enterprise text data involves: pre-inputting natural language text being segmented into a series of tokens (markers or lexical units) that conform to a predefined vocabulary;

[0080] S42. Token Embedding: Each token is transformed into a high-dimensional vector representation by looking up an embedding matrix, where the embedding matrix is ​​a matrix of size (vocab_size, embedding_dim), where vocab_size is the size of the vocabulary and embedding_dim is the dimension of the embedding vector, i.e., the dimension of the embedding vector is the text embedding vector T. f ;

[0081] S43. Two-layer positional encoding adds a positional code to each token to represent the position of the token in the sentence, thereby deriving a text positional embedding vector;

[0082] S44. The input token embedding vector and position embedding vector are added together, and the resulting value is fed into multiple self-attention heads within the text Transformer encoder. Each self-attention head maps the input vector to query, key, and value vectors through a linear transformation. Then, a scaled dot product attention mechanism is used to calculate the attention weights between each token and other tokens, generating a weighted output vector, which is the projection matrix. ;

[0083] S45. Concatenate the output vectors of multiple self-attention heads together and obtain the final multi-head self-attention output through another set of linear transformations;

[0084] S46. The multi-head self-attention output is then further processed and transformed through a feedforward neural network;

[0085] S47. For example Figure 3 As shown, for image information analysis and processing, each image patch is flattened into a one-dimensional vector and mapped using a convolutional layer to transform it into a higher-dimensional vector space.

[0086] S48. In all vector sequences (image feature vectors are...) Add a special token at the beginning of the image classification task. The token is called Class Token and is used for subsequent image classification tasks.

[0087] S49. The image Transformer encoder adds a dynamic positional code to each of the higher-dimensional vector spaces;

[0088] S410. Input the vector sequence containing the Class Token and dynamic position encoding into the image Transformer encoder. The image Transformer encoder consists of multiple self-attention layers and feedforward neural network layers to encode the input sequence. The self-attention layers can capture the interdependencies between elements in the input sequence and perform weighted aggregation of the elements based on these dependencies, i.e., weighted aggregation into a projection matrix. .

[0089] in, Figure 2 The "residual connection & normalization" step is used to stabilize the data obtained in the previous step. The remaining data is calculated using the data models within the image Transformer encoder and the text Transformer encoder, including the text embedding vector Tf for text type information and the text projection matrix. The image feature vector of image type information is and the projection matrix of the image .

[0090] In one embodiment of the present invention, calculating the mapping feature vector generated from text and images specifically includes the following steps:

[0091] S51. Text Feature Vector: After processing by the text Transformer encoder, each token corresponds to a high-dimensional feature vector. In the CLIP model, the high-dimensional output vector corresponding to a specific token (such as the "[CLS]" marker) is used as the feature vector for the entire sentence. This means that, or pooling the feature vectors of all tokens, we can obtain the overall feature vector of the sentence. Pooling, in particular, generates the feature representation of the entire sentence by aggregating the feature information of all tokens in the sequence.

[0092] Assume the text embedding vector is The projection matrix is , Let the weight matrix be the weight of each position in the text, then the mapped feature vector is... for:

[0093]

[0094] We obtain normalized text feature vectors, which have unit length in the joint multimodal space, making it easier to calculate cosine similarity later.

[0095] S52. Image Feature Vector: In the output sequence of the image Transformer encoder, extract the vector corresponding to the Class Token at the first position as the feature vector of the entire image. :

[0096] Assume the image feature vector is The projection matrix is Then the mapped feature vector :

[0097]

[0098] Finally, we obtain the normalized image feature vectors, which have unit length in the joint multimodal space, making it convenient to calculate the cosine similarity later.

[0099] S53. The mapped feature vectors described above The input is fed into a fully connected layer for classification prediction, and the final output is the predicted enterprise type.

[0100] In one embodiment of the present invention, the similarity between text and image is calculated using the mapping feature vectors generated from the text and image obtained through the above steps, specifically including the following steps:

[0101] S61. Using a text Transformer encoder and an image Transformer encoder, N text features and N image features can be extracted. By combining the N text features and N image features in pairs, the CLIP model will predict N. 2 The CLIP model calculates the similarity of N possible text-image pairs, where the similarity is the cosine similarity between the text features and the image features. The training objective of the CLIP model is to maximize the similarity of N positive samples while minimizing N... 2 - The similarity of N negative samples;

[0102] Cosine similarity is calculated using matrix multiplication, and the result is a similarity score, which represents the degree of matching between text and image.

[0103] Assume the feature vector after text mapping is The feature vector after image mapping is The similarity score is:

[0104] Cosine_simlarity

[0105] The higher the similarity score, the higher the similarity between the text and the image, meaning the more likely the text and the drawing are to match.

[0106] In one embodiment of the present invention, such as Figure 4 As shown, matching and fusing highly similar text and image information specifically includes the following steps:

[0107] S71. Calculating cosine similarity using matrix multiplication will generate N. 2 There are N samples, of which N are positive samples, i.e., the text and image that truly belong to a pair (the diagonal elements of the matrix), and the remaining N are... 2 - N text-image pairs are negative samples;

[0108] S72. By embedding the matched text and image information into the same semantic space through contrastive learning, cross-modal information fusion is achieved, thereby forming a training dataset.

[0109] In one embodiment of the present invention, the information promotion and matching of similar enterprises is achieved by the following steps:

[0110] S81. The user inputs a text description related to the company into the CLIP model, and the CLIP model can retrieve the most matching image (such as company logo, product image, etc.) to quickly obtain company information. Conversely, the user can also upload images related to the company (such as product images), and the CLIP model can generate a text description related to it to help the user understand product information more intuitively.

[0111] S82. The remaining N when calculating cosine similarity 2 -N negative samples can be defined as similar enterprises of the same type as this enterprise;

[0112] This way, when users input "technology companies," they will not only be matched with the companies they want to match from the training dataset, but also with other technology companies.

[0113] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" and their orientation or positional relationships are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0114] The present invention has been described in the above-described embodiments; however, these embodiments are merely examples for implementing the present invention. It must be noted that the disclosed embodiments do not limit the scope of the present invention. Conversely, any modifications and refinements made without departing from the spirit and scope of the present invention are within the scope of patent protection of the present invention.

Claims

1. A method for multimodal data fusion based on location-sensitive optimization, characterized in that, The location-sensitive optimization-based multimodal data fusion method includes the following steps: S1. Data Collection: The CLIP model obtains enterprise data information related to the enterprise from different channels. The enterprise data information includes the enterprise name, trademark, product images, and business scope. The CLIP model includes a text Transformer encoder and an image Transformer encoder. S2. Data preprocessing: Use data processing libraries in Python to preprocess the data; S3. Label the processed data: Use the labeling and extraction of the preprocessed data and upload it to the storage unit. The data labeling methods include keyword highlighting and extraction methods and image labeling software labelme. S4. Extracting labeled data information: Using the text Transformer encoder and the image Transformer encoder, the labeled data in the storage unit is analyzed, extracted, and located to obtain the embedding vectors and projection matrices of the text and image. The text Transformer encoder is equipped with double-layer position encoding, and the image Transformer encoder is equipped with dynamic position encoding. The double-layer position encoding yields the position embedding vector in the text, and the dynamic position encoding yields the vector sequence of the image in the image content. S5. Calculate specific vector values ​​using text and image information: Substitute the multiple feature vectors obtained above into the data models within the text Transformer encoder and the image Transformer encoder to calculate the mapped feature vectors in the text and image, and obtain the normalized feature vectors. S6. Substitute the normalized feature vectors into the similarity formula to calculate the similarity between text and image information: Use a text Transformer encoder and an image Transformer encoder to extract text and image features respectively, combine N text features and N image features in pairs, and substitute them into the CLIP model to predict N. 2 The similarity of a possible text-image is calculated as the cosine similarity between the text features and the image features. The cosine similarity score is obtained by matrix multiplication. The higher the similarity score, the higher the similarity between the text and the image. S7. Integrate highly similar text and image information: Match highly similar text and image information to accurately obtain various types of information about the enterprise and achieve accurate profiling; S8. Achieve information promotion and matching for similar enterprises: Arrange examples based on text and image information with high similarity.

2. The method for multimodal data fusion based on location-sensitive optimization according to claim 1, characterized in that, S2 specifically includes the following steps: S21. For enterprise data in the text section: Use a text processing encoder to complete, modify the format and content of the enterprise data, remove redundant data, and standardize the terminology, abbreviations and data formats of the text; S22. For enterprise data in the image section: By comparing the hash values ​​of the images, images with high similarity are considered as duplicate images and then deleted. At the same time, Gaussian filters or median filters are used to remove random noise in the images, ensuring the clarity of image edges while denoising. An SVM machine learning model is used to identify images that do not meet the standards, and the image's clarity, contrast, and color saturation indicators are detected. Low-quality images are deleted or marked.

3. The method for multimodal data fusion based on location-sensitive optimization according to claim 1, characterized in that, S3 specifically includes the following steps: S31. For enterprise data of text type: the keyword highlighting extraction method is used to identify and highlight key information in the text; S32. For enterprise data of image type: Use the image annotation software labelme to locate and annotate specific targets in the image, classify the image into a predefined category, and store it in the corresponding storage unit.

4. The method for multimodal data fusion based on location-sensitive optimization according to claim 1, characterized in that, S4 specifically includes the following steps: S41. Extract enterprise data from text: The pre-input natural language text will be segmented into a series of tokens that conform to a predefined vocabulary; S42. Perform embedding tagging: Each token is converted into a high-dimensional vector representation by looking up an embedding matrix, wherein the embedding matrix is ​​a matrix of size vocab_size × embedding_dim, where vocab_size is the size of the vocabulary and embedding_dim is the dimension of the embedding vector, i.e., the dimension of the embedding vector is the text embedding vector T. f ; S43. Two-layer positional encoding adds a positional code to each token to represent the position of the token in the sentence, thereby deriving a text positional embedding vector; S44. The input token embedding vector and position embedding vector are added together, and the resulting value is fed into multiple self-attention heads within the text Transformer encoder. Each self-attention head maps the input vector to a query, key, and value vector through a linear transformation. Then, a scaled dot product attention mechanism is used to calculate the attention weights between each token and other tokens, generating a weighted output vector, i.e., the weighted output vector is the projection matrix. ; S45. Simultaneously concatenate the output vectors of multiple self-attention heads together, and obtain the final multi-head self-attention output through another set of linear transformations; S46. The multi-head self-attention output is then further processed and transformed through a feedforward neural network; S47. For image information analysis and processing, each image patch is flattened into a one-dimensional vector and mapped using a convolutional layer to transform it into a higher-dimensional vector space; S48. Add a special token, called the Class Token, to the beginning of all vector sequences. The Class Token is used for subsequent image classification tasks. S49. The image Transformer encoder adds a dynamic positional code to each of the higher-dimensional vector spaces; S410. Input the vector sequence containing the Class Token and dynamic position encoding into the image Transformer encoder, which consists of multiple self-attention layers and feedforward neural network layers to encode the input sequence. The self-attention layers can capture the dependencies between elements in the input sequence and perform weighted aggregation of the elements according to these dependencies, i.e., weighted aggregation into a projection matrix. .

5. The method for multimodal data fusion based on location-sensitive optimization according to claim 4, characterized in that, S5 specifically includes the following steps: S51. Text Feature Vector: After processing by the text Transformer encoder, each token corresponds to a high-dimensional feature vector. In the CLIP model, the high-dimensional output vector corresponding to a specific token is used as the feature vector for the entire sentence. This means that, or pooling the feature vectors of all tokens, we can obtain the overall feature vector of the sentence. Pooling, in particular, generates the feature representation of the entire sentence by aggregating the feature information of all tokens in the sequence. Assume the text embedding vector is The projection matrix is , Let be the weight matrix, which represents the weight of each position in the text. Then the mapped feature vector... for: ； We obtain normalized text feature vectors, which have unit length in the joint multimodal space, making it easier to calculate cosine similarity later. S52. Image Feature Vector: In the output sequence of the image Transformer encoder, extract the vector corresponding to the Class Token at the first position as the feature vector of the entire image. : Assume the image feature vector is The projection matrix is Then the mapped feature vector : ； Finally, we obtain the normalized image feature vectors, which have unit length in the joint multimodal space, making it convenient to calculate the cosine similarity later. S53. The mapped feature vectors described above The input is fed into a fully connected layer for classification prediction, and the final output is the predicted enterprise type.

6. The method for multimodal data fusion based on location-sensitive optimization according to claim 1, characterized in that, The calculation of the similarity between text information and image information specifically includes the following steps: S61. Using a text Transformer encoder and an image Transformer encoder, extract N text features and N image features. Combine the N text features and N image features in pairs. The CLIP model will predict N. 2 The CLIP model calculates the similarity of N possible text-image pairs, where the similarity is the cosine similarity between the text features and the image features. The training objective of the CLIP model is to maximize the similarity of N positive samples while minimizing N... 2 - The similarity of N negative samples; Cosine similarity is calculated using matrix multiplication, and the result is a similarity score, which represents the degree of matching between text and image. Assume the feature vector after text mapping is The feature vector after image mapping is The similarity score is: Cosine_simlarity ； The higher the similarity score, the higher the similarity between the text and the image, meaning the more likely the text and the drawing are to match.

7. The method for multimodal data fusion based on location-sensitive optimization according to claim 1, characterized in that, S7 specifically includes the following steps: S71. Calculating cosine similarity using matrix multiplication will generate N. 2 There are N samples, of which N are positive samples, i.e., the text and image that truly belong to a pair, and the remaining N are negative samples. 2 - N text-image pairs are negative samples; S72. By embedding the matched text and image information into the same semantic space through contrastive learning, cross-modal information fusion is achieved.

8. The method for multimodal data fusion based on location-sensitive optimization according to claim 7, characterized in that, To achieve information promotion and matching with similar businesses, the following steps are included: S81. The user inputs a text description related to the enterprise into the CLIP model, and the CLIP model can retrieve the most matching image to quickly obtain enterprise information. Conversely, the user can also upload an image related to the enterprise, and the CLIP model can generate a text description related to it to help the user understand product information more intuitively. S82. The remaining N when calculating cosine similarity 2 -N negative samples are defined as similar enterprises of the same type as the enterprise.

Images