Large model multi-modal data semantic representation alignment method

Abstract

The invention discloses a large-model multi-modal data semantic representation alignment method, and the method comprises the steps: carrying out the fine-grained local feature extraction of first modal data and second modal data through employing a data processing technology based on deep learning, the method comprises the following steps: acquiring first modal data and second modal data, calculating similarity between fine-grained local features of the first modal data and the second modal data as an initial alignment loss function value, and then further combining global context semantic information to carry out semantic enhancement coding on the fine-grained local features of the first modal data and the second modal data to obtain an initial alignment loss function value; according to the method, a fine alignment loss function value is obtained through context-level semantic alignment measurement, and then model training is guided based on the fine alignment loss function value and an initial alignment loss function value, so that semantic representation alignment of multi-modal data is realized. In this way, the alignment precision of the multi-modal data on the semantic level can be remarkably improved, and a more accurate feature basis is provided for subsequent data fusion and application.

Description

Technical Field

[0001] The present application relates to the field of large models, and more specifically, to a method for aligning semantic representations of multimodal data of large models. Background Art

[0002] With the rapid development of information technology, multimedia data is becoming more and more widely disseminated on the Internet. Multimedia data usually contains information in multiple modes, such as text, images, audio, and video. There are complex correlations between these different modal data, and how to effectively understand and utilize this cross-modal information has become a hot research issue.

[0003] In traditional multimodal data analysis, different modal data are often processed independently, lacking an effective cross-modal information interaction mechanism. This method not only fails to capture the deep correlation between modalities, but also fails to fully tap the potential value of each modal data. In recent years, the development of deep learning technology has provided a new approach to solving this problem. By building a deep neural network model, joint representation learning of multimodal data can be achieved, thereby better understanding the relationship between modalities and improving the application value of data.

[0004] Traditional multimodal data semantic alignment methods are mainly based on simple feature extraction and linear mapping. However, in practical applications, since data of different modalities have different characteristics and expressions, the heterogeneity between modalities leads to differences in data distribution. It becomes difficult to directly map multimodal data into the same semantic space for comparison and fusion, and it is easy to cause information loss and distortion.

[0005] Therefore, an optimized semantic representation alignment method for large-model multimodal data is desired. Summary of the invention

[0006] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a large-model multimodal data semantic representation alignment method, which extracts fine-grained local features of the first modal data and the second modal data respectively by adopting a data processing technology based on deep learning, and calculates the similarity between the fine-grained local features of the first modal data and the second modal data as the initial alignment loss function value, and then further combines the global contextual semantic information to perform semantic enhancement encoding on the fine-grained local features of the first modal data and the second modal data, and obtains the fine alignment loss function value through the context-level semantic alignment measurement, and then, guides the model training based on the fine alignment loss function value and the initial alignment loss function value to achieve semantic representation alignment of multimodal data. In this way, the alignment accuracy of multimodal data at the semantic level can be significantly improved, providing a more accurate feature basis for subsequent data fusion and application.

[0007] According to one aspect of the present application, a method for aligning semantic representations of multimodal data of a large model is provided, which includes: Acquire first modal data and second modal data, wherein the first modal data is a picture and the second modal data is a text description of the image content; Performing fine-grained local feature extraction on the first modality data and the second modality data respectively to obtain a set of image block embedding coding vectors and a set of image content text description word embedding coding vectors; Calculating an initial alignment loss function value based on the semantic similarity between the set of image content text description word embedding coding vectors and the set of image block embedding coding vectors; Performing context semantic association coding on the set of image block embedded coding vectors to obtain a set of context image block semantic coding vectors; Performing field-driven contextual semantic association enhancement on the set of image content text description word embedding coding vectors to obtain a set of contextual image content text description word semantic enhancement coding vectors; Calculating a precise alignment loss function value based on the semantic similarity between the set of semantic encoding vectors of the context image blocks and the set of semantic enhancement encoding vectors of the context image content text description words; A weighted sum of the fine alignment loss function value and the initial alignment loss function value is calculated as the loss function value to train the model.

[0008] Compared with the prior art, the present application provides a large-model multimodal data semantic representation alignment method, which uses deep learning-based data processing technology to extract fine-grained local features of the first modal data and the second modal data, respectively, and calculates the similarity between the fine-grained local features of the first modal data and the second modal data as the initial alignment loss function value, and then further combines the global contextual semantic information to perform semantic enhancement encoding on the fine-grained local features of the first modal data and the second modal data, and obtains the fine alignment loss function value through the context-level semantic alignment measurement, and then, based on the fine alignment loss function value and the initial alignment loss function value, guides model training to achieve semantic representation alignment of multimodal data. In this way, the alignment accuracy of multimodal data at the semantic level can be significantly improved, providing a more accurate feature basis for subsequent data fusion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 It is a flowchart of a method for aligning semantic representations of multimodal data of a large model according to an embodiment of the present application; Figure 2 A data flow diagram of a large model multimodal data semantic representation alignment method according to an embodiment of the present application; Figure 3 This is a flowchart of sub-step S5 of the large model multimodal data semantic representation alignment method according to an embodiment of the present application. DETAILED DESCRIPTION

[0011] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0012] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0013] Although the present application makes various references to certain modules in the system according to the embodiments of the present application, any number of different modules can be used and run on the user terminal and / or server. The modules are only illustrative, and different aspects of the system and method can use different modules.

[0014] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously as required. Meanwhile, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0015] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0016] In traditional multimodal data analysis, different modal data are often processed independently, lacking an effective cross-modal information interaction mechanism. This method not only fails to capture the deep correlation between modalities, but also fails to fully tap the potential value of each modal data. In recent years, the development of deep learning technology has provided a new idea for solving this problem. By constructing a deep neural network model, joint representation learning of multimodal data can be achieved, so as to better understand the relationship between modalities and improve the application value of data. Traditional multimodal data semantic alignment methods are mainly based on simple feature extraction and linear mapping. However, in practical applications, since data of different modalities have different characteristics and expressions, the heterogeneity between modalities leads to differences in data distribution. It becomes difficult to directly map multimodal data into the same semantic space for comparison and fusion, and it is easy to cause information loss and distortion. Therefore, an optimized semantic representation alignment method for large-model multimodal data is expected.

[0017] Based on this, in the technical solution of the present application, a method for aligning semantic representations of large-model multimodal data is proposed. Figure 1 The present invention is a flowchart of a method for aligning semantic representations of multimodal data of a large model according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a method for aligning semantic representations of large-scale multimodal data according to an embodiment of the present application. Figure 1 and Figure 2As shown, according to the embodiment of the present application, the large-model multimodal data semantic representation alignment method includes the following steps: S1, obtaining first modal data and second modal data, wherein the first modal data is a picture and the second modal data is a text description of the image content; S2, respectively performing fine-grained local feature extraction on the first modal data and the second modal data to obtain a set of image block embedding coding vectors and a set of image content text description word embedding coding vectors; S3, based on the semantic similarity between the set of image content text description word embedding coding vectors and the set of image block embedding coding vectors, calculating the initial alignment loss function value; S4, performing local feature extraction on the image The set of block embedding coding vectors is subjected to contextual semantic association coding to obtain a set of context image block semantic coding vectors; S5, the set of image content text description word embedding coding vectors is subjected to field effect-driven contextual semantic association enhancement to obtain a set of context image content text description word semantic enhancement coding vectors; S6, based on the semantic similarity between the set of context image block semantic coding vectors and the set of context image content text description word semantic enhancement coding vectors, a fine alignment loss function value is calculated; S7, a weighted sum of the fine alignment loss function value and the initial alignment loss function value is calculated as the loss function value to train the model.

[0018] In particular, S1 obtains first modal data and second modal data, wherein the first modal data is a picture, and the second modal data is a text description of the image content. It should be understood that in order to achieve semantic representation alignment of multimodal data, a data set containing multiple types of data is first required. Pictures and corresponding text descriptions of image content constitute the most basic multimodal data pair. Pictures provide visual information, while text descriptions of image content provide text descriptions of the image content. Although the two modal data have different forms of expression, they are related at the semantic level. Utilizing this correlation can help the model learn how to map data of different modalities into the same semantic space, thereby achieving semantic alignment.

[0019] In particular, the S2 performs fine-grained local feature extraction on the first modal data and the second modal data respectively to obtain a set of image block embedding coding vectors and a set of image content text description word embedding coding vectors. Specifically, first, the first modal data is segmented into an image grid to obtain a set of image blocks; and the second modal data is segmented to obtain a set of image content text description words. Considering that directly performing feature extraction and alignment processing on the first modal data and the second modal data may result in the loss of local detail information, especially when the data contains complex scenes or multiple objects. Therefore, in order to capture the local detail features in the image and the text description of the image content to achieve fine-grained semantic alignment between the image data and the text data, the present application further performs image grid segmentation on the first modal data to refine the granularity of data processing and form a set of image blocks. At the same time, by performing word segmentation processing on the second modal data, the long text description is decomposed into a series of word units to form a set of image content text description words. In this way, it helps to capture the subtle connection between the image and the text, thereby establishing a more accurate alignment relationship. Next, an image block embedding encoder is used to embed the image blocks in the set of image blocks to obtain a set of image block embedding encoding vectors; in the technical solution of the present application, in order to extract the semantic feature representations of image data and text data respectively, thereby providing a basis for cross-modal alignment between the two, the present application further embeds the image block set and the image content text description word set to achieve data format unification and semantic information extraction. Specifically, for image data, the present application uses an image block embedding encoder to embed the image blocks in the set of image blocks. In an embodiment of the present application, the image block embedding encoder is constructed based on a spatial convolutional neural network, which can effectively capture the key information of each image block through a convolution operation, and convert each image block from a high-dimensional pixel space to a low-dimensional semantic embedding space, thereby obtaining the semantic feature representation of each image block, and obtaining a set of image block embedding encoding vectors. At the same time, for text data, a word embedding encoder based on a Bert model is used to semantically embed the image content text description words in the set of image content text description words to obtain a set of image content text description word embedding encoding vectors. Those skilled in the art should know that the Bert model is a pre-trained language model that is pre-trained with large-scale text data and has good performance in word embedding coding. By processing each image content text description word through the Bert model, it can ensure that the semantic information of the text data is fully expressed and encoded, and a vector representation containing rich semantic information is generated for each image content text description word, thereby obtaining a set of image content text description word embedding coding vectors.

[0020] In particular, the S3 calculates the initial alignment loss function value based on the semantic similarity between the set of the image content text description word embedding coding vectors and the set of the image block embedding coding vectors. In order to reveal the semantic representation alignment between the image data and the text data, so as to further optimize the alignment performance of the model, in the technical solution of the present application, by mapping the set of the image block embedding coding vectors and the set of the image content text description word embedding coding vectors to a low-dimensional hash space, the hash distance between each group of corresponding image content text description word embedding coding vectors and image block embedding coding vectors is calculated to measure the semantic alignment quality between the two. Specifically, the calculation of the hash distance is not limited by the dimension of the input data. By converting the input data into a binary hash code of uniform length and calculating the Hamming distance between the hash codes, a fast semantic similarity comparison between the image block and the text description word is achieved. It usually has higher computational efficiency and lower storage requirements than other similarity metrics (such as cosine similarity, Euclidean distance, etc.), and is suitable for processing large-scale data. Then, the sum of the generated initial alignment hash distance set is used as the initial alignment loss function value to guide the training optimization of the model.

[0021] In particular, the S4 performs context semantic association encoding on the set of image block embedding coding vectors to obtain a set of context image block semantic coding vectors. In the technical solution of the present application, the set of image block embedding coding vectors is input into an image context semantic encoder based on a transformer structure to obtain a set of context image block semantic coding vectors. In particular, in view of the complex semantic associations and context information in image data and text data, relying solely on the semantic alignment metric of local features may result in poor alignment due to the deviation of the overall semantic understanding. Therefore, the present application further introduces global context information, performs semantic enhancement encoding on the fine-grained local features of the first modal data and the second modal data, and considers the global context information to more accurately understand the semantic alignment relationship between the image and the text. Specifically, for image data, the present application uses an image context semantic encoder based on a transformer structure to encode the set of image block embedding coding vectors, so as to capture the long-distance dependencies between image blocks by using the self-attention mechanism of the Transformer structure, and encode the global context information of the image into each image block embedding feature to enhance its semantic expression ability, thereby obtaining a set of context image block semantic coding vectors containing global context information.

[0022] In particular, the S5 performs field-effect-driven contextual semantic association enhancement on the set of image content text description word embedding coding vectors to obtain a set of contextual image content text description word semantic enhancement coding vectors. Considering that in the word segmentation process of text data, due to the different lengths of terms, each image content text description word usually has a different information density. Therefore, in order to more effectively encode the context information of text data, the present application proposes a field-effect-driven contextual semantic association enhancement module to process the set of image content text description word embedding coding vectors, and learn the semantic association effect between each image content text description word by analyzing the semantic field effect distribution of each image content text description word, and use this contextual association effect to perform feature update and semantic enhancement processing on the set of image content text description word embedding coding vectors, thereby improving the semantic feature quality of each text description word and the overall semantic alignment effect. In a specific example of the present application, such as Figure 3 As shown, the S5 includes: S51, based on the semantic field effect distribution of the set of image content text description word embedding coding vectors, updating the set of image content text description word embedding coding vectors to obtain a set of updated image content text description word embedding coding vectors; S52, performing bidirectional contextual semantic association enhancement on the set of updated image content text description word embedding coding vectors to obtain a set of contextual image content text description word semantic enhancement coding vectors.

[0023] Specifically, the S51, based on the semantic field effect distribution of the set of image content text description word embedding coding vectors, updates the set of image content text description word embedding coding vectors to obtain a set of updated image content text description word embedding coding vectors. In an embodiment of the present application, first, a semantic reconstructor is used to perform feature reconstruction on each image content text description word embedding coding vector in the set of image content text description word embedding coding vectors to obtain a set of reconstructed image content text description word embedding coding vectors; that is, by using a semantic reconstructor to perform domain transformation on each image content text description word embedding coding vector in the set of image content text description word embedding coding vectors, feature dimension unification is achieved to generate a set of reconstructed image content text description word embedding coding vectors. Next, the semantic depth implicit representation value of each reconstructed image content text description word embedding coding vector in the set of the reconstructed image content text description word embedding coding vector is calculated to obtain a set of semantic depth implicit representation values ​​of the image content text description word; in the technical solution of the present application, by calculating the semantic depth implicit representation value of each reconstructed image content text description word embedding coding vector, the information density or complexity carried by each image content text description word in the overall information of the image content text description is measured, and the importance and influence of each image content text description word in the overall information of the image content text description is quantified. Furthermore, based on the sorting of the set of semantic depth implicit representation values ​​of the image content text description word, the set of image content text description word embedding coding vectors is re-sorted to obtain the set of updated image content text description word embedding coding vectors. That is, according to the semantic depth implicit representation value of each image content text description word, the feature order in the set is further adjusted, and the information structure of the image content text description is reorganized to ensure that key information is given priority in the subsequent context semantic association enhancement coding process.

[0024] Among them, the specific steps of using a semantic reconstructor to reconstruct the features of each image content text description word embedding coding vector in the set of image content text description word embedding coding vectors include: using a first weight parameter matrix and a second weight parameter matrix to perform continuous linear transformation on the image content text description word embedding coding vector to obtain the reconstructed image content text description word embedding coding vector.

[0025] More specifically, the specific steps of calculating the semantic depth implicit representation value of each reconstructed image content text description word embedding coding vector in the set of reconstructed image content text description word embedding coding vectors include: calculating the square of the Euclidean norm of the reconstructed image content text description word embedding coding vector to obtain the initial semantic depth implicit representation value of the image content text description word; calculating the ratio of the sum of the initial semantic depth implicit representation value of the image content text description word and the difference between the sum of the initial semantic depth implicit representation value of the image content text description word and the initial semantic depth implicit representation value of the image content text description word to obtain the semantic depth representation modulation coefficient; calculating the logarithmic function value of the semantic depth representation modulation coefficient with base 2 to obtain the semantic depth implicit representation value of the image content text description word.

[0026] Specifically, the S52 performs bidirectional contextual semantic association enhancement on the set of the updated image content text description word embedding coding vectors to obtain the set of context image content text description word semantic enhancement coding vectors. In an embodiment of the present application, first, the set of the updated image content text description word embedding coding vectors is input into a first semantic encoder based on a forward LSTM model to obtain a set of forward image content text description word context semantic coding feature vectors; and the set of the updated image content text description word embedding coding vectors is input into a second semantic encoder based on a backward LSTM model to obtain a set of backward image content text description word context semantic coding feature vectors; that is, a bidirectional long short-term memory (LSTM) model is used to encode the updated image content text description word embedding coding vector set from the forward and backward directions respectively. Among them, the forward LSTM captures the information flow pattern from front to back, while the backward LSTM operates in reverse to identify the dependency from back to front. This dual-perspective encoding mechanism can capture the global dependency of image content text description information at different levels, and thereby enhance the semantic feature expression of each image content text description word. Finally, the set of the forward image content text description word context semantic coding feature vectors and the set of the backward image content text description word context semantic coding feature vectors are fused to obtain the set of context image content text description word semantic enhancement coding vectors. Here, the bidirectional context semantic information of each image content text description word is integrated by fusing the coding results of the bidirectional LSTM model to form a richer and more comprehensive set of context image content text description word semantic enhancement coding vectors.

[0027] Among them, the specific steps of fusing the set of context semantic coding feature vectors of the forward image content text description words and the set of context semantic coding feature vectors of the backward image content text description words include: using the first weighting parameter and the second weighting parameter to weighted modulate the set of context semantic coding feature vectors of the forward image content text description words and the set of context semantic coding feature vectors of the backward image content text description words respectively, and calculating the corresponding points between the set of weighted modulated forward image content text description words context semantic coding feature vectors, the set of weighted modulated backward image content text description words context semantic coding feature vectors and the bias item to obtain the set of context image content text description words semantic enhancement coding vectors.

[0028] In summary, in the above embodiment, the set of the image content text description word embedding coding vectors is enhanced based on the field effect driven contextual semantic association to obtain a set of contextual image content text description word semantic enhancement coding vectors, including: processing the set of the image content text description word embedding coding vectors with the following feature enhancement formula to obtain a set of contextual image content text description word semantic enhancement coding vectors, wherein the feature enhancement formula is:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] in, represents the set of embedding encoding vectors of the text description words of the image content, , , and Respectively represent the first, second, and third words in the set of embedding encoding vectors of the image content text description words and The image content text description word embedding encoding vector, is the number of embedding encoding vectors of the text description words of the image content, and denote the first weight parameter matrix and the second weight parameter matrix respectively, For the reconstructed image content text description word embedding encoding vector, represents the calculation of the Euclidean norm of a vector, represents the logarithmic function with base 2, Indicates The semantic depth implicit representation value of the image content text description word, represents the set of embedding encoding vectors for updating the text description words of the image content, , , and Respectively represent the first, second, and third in the set of embedding encoding vectors of the updated image content text description word and Update the image content text description word embedding encoding vector, represents the forward LSTM model, represents the backward LSTM model, Represents the set of contextual semantic encoding feature vectors of the forward image content text description words, Represents the set of contextual semantic encoding feature vectors of the backward image content text description words, and denote the first weighting parameter and the second weighting parameter respectively, is the bias term, A set of semantically enhanced encoding vectors of text description words of the context image content.

[0036] In particular, the S6 calculates the value of the precise alignment loss function based on the semantic similarity between the set of the context image block semantic coding vectors and the set of the context image content text description word semantic enhancement coding vectors. In a specific example of the present application, the hash distance between each group of corresponding context image block semantic coding vectors and context image content text description word semantic enhancement coding vectors in the set of the context image block semantic coding vectors and the set of the context image content text description word semantic enhancement coding vectors is calculated to obtain a set of precise alignment hash distances, and the sum of the set of precise alignment hash distances is calculated as the value of the precise alignment loss function. That is, by calculating the hash distance between each group of corresponding context image block semantic coding vectors and context image content text description word semantic enhancement coding vectors in the set of the context image block semantic coding vectors and the set of the context image content text description word semantic enhancement coding vectors, the quality of the semantic alignment between the two is measured, and the model's understanding of the complex semantic relationship between image content and text description is further improved. Finally, the sum of the obtained precise alignment hash distances is used as the precise alignment loss function value to guide the model to perform iterative optimization, so as to achieve the purpose of improving the semantic alignment accuracy of image data and text data.

[0037] Preferably, calculating the hash similarity between each corresponding set of context image block semantic coding vectors and context image content text description word semantic enhancement coding vectors in the set of context image block semantic coding vectors and the set of context image content text description word semantic enhancement coding vectors comprises: Cascading the context image block semantic coding vector and the context image content text description word semantic enhancement coding vector into a context image block semantics-context image content text description word semantics joint coding feature vector; The absolute value sum and the square root of the sum of squares of all eigenvalues ​​of the context image block semantics-context image content text description word semantics joint encoding feature vector are calculated to obtain the first image block semantics-image content text description word semantics joint encoding space structure value and the second image block semantics-image content text description word semantics joint encoding space structure value, that is:

[0038]

[0039] in, The first character represents the semantics of the context image block and the semantics of the context image content text description word joint encoding feature vector eigenvalues, Represents the spatial structure value of the semantics of the first image block and the semantics of the image content text description word joint encoding, Represents the second image block semantics-image content text description word semantics joint encoding space structure value; Determine the number of eigenvalues ​​of all eigenvalues ​​of the context image block semantics-context image content text description word semantics joint encoding feature vector , i.e., the length of the context image block semantics-context image content text description word semantics joint encoding feature vector;

[0040] For each eigenvalue of the context image block semantics-context image content text description word semantics joint encoding feature vector, calculate the first image block semantics-image content text description word semantics joint encoding space structure value minus the product of the eigenvalue and the number of eigenvalues ​​to obtain the first image block semantics-image content text description word semantics joint encoding long-range dependency value ,in, Represents the spatial structure value of the semantics of the first image block and the semantics of the image content text description word joint encoding, The first character represents the semantics of the context image block and the semantics of the context image content text description word joint encoding feature vector eigenvalues, The number of eigenvalues ​​representing all eigenvalues ​​of the context image block semantics-context image content text description word semantics joint encoding feature vector, Indicates the long-range dependency value of the first image block semantics-image content text description word semantics joint encoding; Calculate the product of the square root of the number of eigenvalues ​​multiplied by the eigenvalue minus the second image block semantics-image content text description word semantics joint coding spatial structure value to obtain the second image block semantics-image content text description word semantics joint coding long-range dependency value ,in, Represents the spatial structure value of the semantics of the second image block and the semantics of the image content text description word joint encoding, The first character represents the semantics of the context image block and the semantics of the context image content text description word joint encoding feature vector eigenvalues, The number of eigenvalues ​​representing all eigenvalues ​​of the context image block semantics-context image content text description word semantics joint encoding feature vector, Represents the long-range dependency value of the second image block semantics-image content text description word semantics joint encoding; The exponential value calculated by taking the long-range dependency value of the first image block semantics-image content text description word semantics joint coding as the exponent of the natural constant and the inverse of the long-range dependency value of the second image block semantics-image content text description word semantics joint coding are weighted summed to obtain the optimized eigenvalue corresponding to each eigenvalue ,in, Indicates the long-range dependency value of the first image block semantics-image content text description word semantics joint encoding, Indicates the long-range dependency value of the second image block semantics-image content text description word semantics joint encoding, represents a natural constant, and represents the weighted hyperparameter, Indicates the optimized eigenvalue corresponding to each eigenvalue; Combining the optimized feature values ​​into an optimized context image block semantics-context image content text description word semantics joint encoding feature vector; Splitting the optimized context image block semantics-context image content text description word semantics joint coding feature vector into an optimized context image block semantic coding vector and an optimized context image content text description word semantics reinforcement coding vector; Calculate the hash similarity between the optimized context image block semantic encoding vector and the optimized context image content text description word semantic enhancement encoding vector.

[0041] Here, the set of context image block semantic coding vectors and the set of context image content text description word semantic enhancement coding vectors represent the set of image block context semantic coding features and the set of image content word granularity context semantic coding features, respectively. Considering that when the set of image content text description word embedding coding vectors is input into the semantic feature enhancement module based on sequence field effect distribution analysis, this will strengthen the local neighborhood semantic associations between the image content text description word embedding coding vectors in the set of image content text description word embedding coding vectors, but it will also make each group of corresponding context image block semantic coding vectors and context image content text description word semantic enhancement coding vectors in the set of context image block semantic coding vectors and the set of context image content text description word semantic enhancement coding vectors have spatial structural differences due to the feature enhancement direction offset, thereby deviating from the semantic matching space domain mapping comprehension represented by the source coding semantic feature distribution, affecting the calculation accuracy of the hash similarity.

[0042] Based on this, in view of the possible spatial structure loss in the high-dimensional space of the feature set of the context image block semantics-context image content text description word semantics jointly encoded feature vector, which causes the decoder weight to implicitly infer the spatial structure information based on the features, resulting in inconsistent convergence. A long-distance feature dependency relationship is established based on the spatial structure representation of the context image block semantics-context image content text description word semantics jointly encoded feature vector based on the overall feature scale of the context image block semantics-context image content text description word semantics jointly encoded feature vector, so as to establish the feature local connectivity of the context image block semantics-context image content text description word semantics jointly encoded feature vector, and capture the spatial ambiguous information of the object feature value through the unstructured feature value point prediction of the context image block semantics-context image content text description word semantics jointly encoded feature vector, thereby improving the spatial inductive bias perception ability of the feature set of the context image block semantics-context image content text description word semantics jointly encoded feature vector, so as to improve the understanding consistency between the hash similarity metric mapping regression and the feature extraction process, and improve the calculation accuracy of the hash similarity.

[0043] In particular, the S7 calculates the weighted sum between the fine alignment loss function value and the initial alignment loss function value as the loss function value, and trains the model. That is, the weighted sum between the fine alignment loss function value and the initial alignment loss function value is calculated as the loss function value, and the image block embedding encoder, the word embedding encoder based on the Bert model, the image context semantic encoder based on the converter structure, and the context semantic association reinforcement module based on the field effect drive are trained, and the model parameters are continuously adjusted by minimizing the loss function value, and the alignment relationship between the image block and the text word is gradually optimized, so as to achieve fine adjustment of the semantic alignment between cross-modal data.

[0044] In summary, according to the large model multimodal data semantic representation alignment method of the embodiment of the present application is explained, which uses deep learning-based data processing technology to extract fine-grained local features of the first modal data and the second modal data respectively, and calculates the similarity between the fine-grained local features of the first modal data and the second modal data as the initial alignment loss function value, and then further combines the global contextual semantic information to perform semantic enhancement encoding on the fine-grained local features of the first modal data and the second modal data, and obtains the fine alignment loss function value through the context-level semantic alignment measurement, and then, based on the fine alignment loss function value and the initial alignment loss function value, guides model training to achieve semantic representation alignment of multimodal data. In this way, the alignment accuracy of multimodal data at the semantic level can be significantly improved, providing a more accurate feature basis for subsequent data fusion and application.

[0045] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A semantic representation alignment method for large-scale multimodal data, characterized in that: include: Acquire first modal data and second modal data, wherein the first modal data is a picture and the second modal data is a text description of the image content; Performing fine-grained local feature extraction on the first modality data and the second modality data respectively to obtain a set of image block embedding coding vectors and a set of image content text description word embedding coding vectors; Calculating an initial alignment loss function value based on the semantic similarity between the set of image content text description word embedding coding vectors and the set of image block embedding coding vectors; Performing context semantic association coding on the set of image block embedded coding vectors to obtain a set of context image block semantic coding vectors; Performing field-driven contextual semantic association enhancement on the set of image content text description word embedding coding vectors to obtain a set of contextual image content text description word semantic enhancement coding vectors; Calculating a precise alignment loss function value based on the semantic similarity between the set of semantic encoding vectors of the context image blocks and the set of semantic enhancement encoding vectors of the context image content text description words; A weighted sum of the fine alignment loss function value and the initial alignment loss function value is calculated as the loss function value to train the model.

2. The method for aligning semantic representations of large-scale multimodal data according to claim 1 is characterized in that: Fine-grained local feature extraction is performed on the first modality data and the second modality data respectively to obtain a set of image block embedding coding vectors and a set of image content text description word embedding coding vectors, including: Performing image grid segmentation on the first modality data to obtain a set of image blocks; Performing word segmentation processing on the second modality data to obtain a set of image content text description words; Using an image block embedding encoder to embed the respective image blocks in the set of image blocks to obtain a set of image block embedding coding vectors; A word embedding encoder based on the Bert model is used to perform semantic embedding encoding on each image content text description word in the set of image content text description words to obtain a set of image content text description word embedding encoding vectors.

3. The method for aligning semantic representations of large-scale multimodal data according to claim 2, characterized in that: Calculating the initial alignment loss function value based on the semantic similarity between the set of image content text description word embedding encoding vectors and the set of image block embedding encoding vectors, including: Calculate the hash distance between each corresponding group of image content text description word embedding coding vectors and image block embedding coding vectors in the set of image content text description word embedding coding vectors and the set of image block embedding coding vectors to obtain a set of initial alignment hash distances, and calculate the sum of the set of initial alignment hash distances as the initial alignment loss function value.

4. The method for aligning semantic representations of large-model multimodal data according to claim 3 is characterized in that: Performing context semantic association coding on the set of image block embedded coding vectors to obtain a set of context image block semantic coding vectors, comprising: The set of image block embedding coding vectors is input into an image context semantic encoder based on a transformer structure to obtain the set of context image block semantic coding vectors.

5. The method for aligning semantic representations of large-model multimodal data according to claim 4 is characterized in that: The set of the image content text description word embedding coding vectors is subjected to field effect driven context semantic association enhancement to obtain a set of context image content text description word semantic enhancement coding vectors, including: Based on the semantic field effect distribution of the set of image content text description word embedding coding vectors, updating the set of image content text description word embedding coding vectors to obtain an updated set of image content text description word embedding coding vectors; The set of updated image content text description word embedding coding vectors is subjected to bidirectional context semantic association enhancement to obtain the set of context image content text description word semantic enhancement coding vectors.

6. The method for aligning semantic representations of large-model multimodal data according to claim 5, characterized in that: Based on the semantic field effect distribution of the set of image content text description word embedding coding vectors, the set of image content text description word embedding coding vectors is updated to obtain an updated set of image content text description word embedding coding vectors, including: Using a semantic reconstructor, feature-reconstructing each image content text description word embedding coding vector in the set of image content text description word embedding coding vectors to obtain a set of reconstructed image content text description word embedding coding vectors; Calculating the semantic depth implicit representation value of each reconstructed image content text description word embedding coding vector in the set of reconstructed image content text description word embedding coding vectors to obtain a set of semantic depth implicit representation values ​​of image content text description words; Based on the sorting of the set of semantic depth implicit representation values ​​of the image content text description words, the set of image content text description word embedding coding vectors is re-sorted to obtain the set of updated image content text description word embedding coding vectors.

7. The method for aligning semantic representations of large-model multimodal data according to claim 6, characterized in that: Using a semantic reconstructor to perform feature reconstruction on each image content text description word embedding coding vector in the set of image content text description word embedding coding vectors to obtain a set of reconstructed image content text description word embedding coding vectors, including: The image content text description word embedding coding vector is continuously linearly transformed using the first weight parameter matrix and the second weight parameter matrix to obtain the reconstructed image content text description word embedding coding vector.

8. The method for aligning semantic representations of large-model multimodal data according to claim 7, characterized in that: Performing bidirectional contextual semantic association enhancement on the set of updated image content text description word embedding coding vectors to obtain the set of context image content text description word semantic enhancement coding vectors, including: Inputting the set of updated image content text description word embedding encoding vectors into a first semantic encoder based on a forward LSTM model to obtain a set of forward image content text description word context semantic encoding feature vectors; Inputting the set of updated image content text description word embedding encoding vectors into a second semantic encoder based on a backward LSTM model to obtain a set of backward image content text description word context semantic encoding feature vectors; The set of the forward image content text description word context semantic coding feature vectors and the set of the backward image content text description word context semantic coding feature vectors are fused to obtain the set of context image content text description word semantic reinforcement coding vectors.

9. The method for aligning semantic representations of large-model multimodal data according to claim 8, characterized in that: Calculating a precise alignment loss function value based on the semantic similarity between the set of context image block semantic encoding vectors and the set of context image content text description word semantic enhancement encoding vectors includes: Calculate the hash distance between each corresponding group of context image block semantic coding vectors and context image content text description word semantic enhancement coding vectors in the set of context image block semantic coding vectors and the set of context image content text description word semantic enhancement coding vectors to obtain a set of precise alignment hash distances, and calculate the sum of the set of precise alignment hash distances as the precise alignment loss function value.

Images