Semantic extraction model training method and application method

Abstract

The invention discloses a training method and an application method of a semantic extraction model, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a multi-modal data sample; performing feature extraction on the multi-modal data sample through a preset encoder, and mapping the extracted features to a semantic space of a target dimension to obtain a multi-modal feature vector corresponding to the multi-modal data sample; on the basis of a temperature sparse attention mechanism, attention weights among the multi-modal feature vectors are calculated; when it is detected that a certain modal feature is missing, complementing the missing modal feature through a hypergraph structure constructed by a multi-modal feature vector to obtain a complemented multi-modal feature vector; and for the complemented multi-modal feature vector, mixed loss formed by uniformity loss and comparison loss is calculated, and parameters in the temperature sparse attention mechanism are optimized according to the mixed loss. The invention provides a training method capable of solving the problem that an existing semantic extraction model is poor in precision.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a training method and application method for a semantic extraction model. Background Technology

[0002] Existing semantic understanding solutions can already extract semantic vectors from multimodal data. Related technologies typically employ a series of fixed steps to achieve joint embedding and representation learning of multimodal data for semantic extraction from cross-modal data. For example, first, a pre-trained model encodes multimodal data such as images, text, and audio to obtain multimodal features. Then, these multimodal features are mapped to a unified low-dimensional semantic space, and cosine similarity is calculated as attention weights to obtain the corresponding vector representations.

[0003] This approach presents at least the following technical problems: 1. The semantic space dimension of joint embedding is low, making it difficult to effectively carry the fine-grained semantic information of multimodal data, resulting in poor recognition accuracy (such as retrieval accuracy) in the application process; 2. In the attention weight calculation, if the text modality weight is too high, other modality information will be overwhelmed. When data for a certain modality is missing, the recognition performance will drop significantly. 3. The single loss function leads to uneven distribution of embedding vectors in high-dimensional space, resulting in poor clustering performance and affecting the accuracy of semantic extraction in the application process.

[0004] Therefore, how to propose a method that can solve the problem of poor semantic extraction accuracy in the currently applied semantic extraction models has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, this application provides a training method and application method for a semantic extraction model, the main purpose of which is to improve at least one of the above-mentioned technical problems in the currently applied semantic extraction models, which leads to poor semantic extraction accuracy.

[0006] Firstly, this application provides a method for training a semantic extraction model, including: Acquire multimodal data samples; the multimodal data samples include at least two of the following: image samples, text samples, and audio samples; The multimodal data samples are subjected to feature extraction by a preset encoder, and the extracted features are mapped to the semantic space of the target dimension to obtain the multimodal feature vector corresponding to the multimodal data samples. Based on the temperature sparse attention mechanism, the attention weights between the multimodal feature vectors are calculated; When a certain modal feature is detected to be missing, the missing modal feature is filled in by a hypergraph structure constructed from the multimodal feature vectors to obtain the filled multimodal feature vectors. For the completed multimodal feature vector, a hybrid loss consisting of uniformity loss and contrast loss is calculated, and the parameters in the temperature sparse attention mechanism are optimized based on the hybrid loss.

[0007] Optionally, the multimodal data samples include image samples, text samples, and audio samples; Feature extraction is performed on the multimodal data samples using a preset encoder, and the extracted features are mapped to the semantic space of the target dimension to obtain the multimodal feature vector corresponding to the multimodal data samples. This includes: encoding the image samples using an EfficientNetV2-XL model as an image encoder to obtain initial image features; encoding the text samples using a DeBERTaV3-large model as a text encoder to obtain initial text features; encoding the audio samples using an Audio Spectrogram Transformer model as an audio encoder to obtain initial audio features; and mapping the initial image features, the initial text features, and the initial audio features to the semantic space of the target dimension through a linear transformation layer to obtain the multimodal feature vector.

[0008] Optionally, after extracting features from the multimodal data samples using a preset encoder and mapping the extracted features to the semantic space of the target dimension to obtain the multimodal feature vectors corresponding to the multimodal data samples, the method further includes: compressing the multimodal feature vectors to obtain the compressed representations corresponding to the multimodal feature vectors; the compression of the multimodal feature vectors to obtain the compressed representations corresponding to the multimodal feature vectors includes: dividing the multimodal feature vectors into a preset number of equal parts to obtain a preset number of feature sub-vectors; calculating the quantization value and residual value of each feature sub-vector; and constructing a compressed representation of each feature sub-vector based on the quantization value and the residual value.

[0009] Optionally, the multimodal feature vector includes image feature vector, text feature vector, and audio feature vector; The temperature-based sparse attention mechanism for calculating attention weights between multimodal feature vectors includes: calculating global statistics of the image feature vectors; using the global statistics as input to a multilayer perceptron to generate temperature parameters; the temperature parameters are used to determine the similarity between any two modalities; calculating a similarity matrix between the multimodal feature vectors using the temperature parameters; calling pre-set sparse parameters and calculating a sparse mask based on the sparse parameters, the temperature parameters, and the similarity matrix; the sparse mask is used to ignore modal information with correlation below a preset threshold; and calculating attention weights between each modality based on the sparse mask and the similarity matrix.

[0010] Optionally, the multimodal feature vector includes image feature vector, text feature vector, and audio feature vector; When a certain modality feature is detected to be missing, the missing modality feature is filled in using a hypergraph structure constructed from the multimodal feature vectors to obtain a filled multimodal feature vector. This includes: constructing the hypergraph structure using the non-missing modality feature vectors from the image feature vector, the text feature vector, and the audio feature vector as nodes; generating virtual nodes for the missing modality feature vectors and adding the virtual nodes as hyperedges to the hypergraph structure; and filling in the missing modality feature using the neural network corresponding to the hypergraph structure.

[0011] Optionally, the missing modal features can be completed using the neural network corresponding to the hypergraph structure, processed according to the following formula:

[0012] In the formula, This represents the completed multimodal feature vector. It is the multimodal feature vector before completion; It is the sigmoid activation function, used to ensure that the attention weights of the output are in the range [0, 1], D v D is the degree matrix of the nodes. e It is the degree matrix of the hyperedge. Learnable diagonal weight matrix.

[0013] Optionally, after calculating a hybrid loss consisting of uniformity loss and contrast loss on the completed multimodal feature vector, and optimizing the parameters in the temperature sparse attention mechanism based on the hybrid loss, the method further includes: optimizing the multimodal feature vector through a Stiefel manifold; The multimodal feature vectors are optimized using the Stiefel manifold, processed according to the following formula:

[0014] In the formula, L represents the gradient computed on the Stiefel manifold for optimizing the multimodal feature vector X, where L is the hybrid loss computed based on the uniformity loss and the contrast loss, and X represents the multimodal feature vector.

[0015] Secondly, this application provides an application method for a semantic extraction model, including: Obtain search information input by the user; the search information includes at least one of image information, text information, or audio information; By using a trained semantic extraction model, the retrieved information is retrieved to obtain retrieval results; The completed semantic extraction model is obtained according to the training method of the semantic extraction model as described in any one of the first aspects above.

[0016] Thirdly, this application provides a training device for a semantic extraction model, comprising: The acquisition unit is configured to acquire multimodal data samples; the multimodal data samples include at least two of image samples, text samples, and audio samples; The extraction unit is configured to extract features from the multimodal data sample using a preset encoder, and map the extracted features to the semantic space of the target dimension to obtain the multimodal feature vector corresponding to the multimodal data sample. The computing unit is configured to calculate the attention weights between the multimodal feature vectors based on a temperature sparse attention mechanism; The completion unit is configured to, upon detecting a missing modal feature, complete the missing modal feature using a hypergraph structure constructed from the multimodal feature vectors to obtain a completed multimodal feature vector; wherein the hypergraph structure is constructed from the multimodal feature vectors. The optimization unit is configured to calculate a hybrid loss consisting of uniformity loss and contrast loss for the completed multimodal feature vector, and optimize the parameters in the temperature sparse attention mechanism based on the hybrid loss.

[0017] Fourthly, this application provides an application apparatus for a semantic extraction model, comprising: The acquisition unit is configured to acquire search information input by the user; the search information includes at least one of image information, text information, or audio information. The retrieval unit is configured to retrieve the retrieval information using a trained semantic extraction model to obtain retrieval results; The completed semantic extraction model is obtained according to the training method of the semantic extraction model as described in any one of the first aspects above.

[0018] By employing the above technical solution, this application provides a training method and application method for a semantic extraction model. First, multimodal data samples are acquired; these samples include at least two of image, text, and audio samples. Features are extracted from the multimodal data samples using a pre-defined encoder, and the extracted features are mapped to a semantic space of the target dimension, resulting in multimodal feature vectors corresponding to the multimodal data samples. Based on a temperature-sparse attention mechanism, attention weights between the multimodal feature vectors are calculated. When a missing modality feature is detected, the missing modality feature is completed using a hypergraph structure constructed from the multimodal feature vectors, resulting in a completed multimodal feature vector. For the completed multimodal feature vector, a hybrid loss consisting of uniformity loss and contrast loss is calculated, and the parameters in the temperature-sparse attention mechanism are optimized based on the hybrid loss. Compared with related technologies, firstly extracting features from the multimodal data samples using a pre-defined encoder and then mapping them to a semantic space of the target dimension effectively unifies the dimensions of data from different modalities and increases the dimension of the embedded semantic space, thereby improving the fine-grainedness of semantic information. Furthermore, when a missing modality feature is detected, the missing modality feature is completed using the topological relationships of the hypergraph structure to improve recognition performance. Finally, considering the distribution characteristics of the embedding vectors in high-dimensional space, the parameters in the model are optimized using a hybrid loss consisting of uniformity loss and contrastive loss to improve the overall accuracy of semantic extraction.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a training method for a semantic extraction model provided in an embodiment of this application is shown. Figure 2 A schematic flowchart of a residual quantization compression method provided in an embodiment of this application is shown; Figure 3 A flowchart illustrating a method for calculating attention weights based on a temperature sparse attention mechanism provided in an embodiment of this application is shown. Figure 4 This illustration shows a flowchart of a method for completing missing modal information using a hypergraph structure, as provided in an embodiment of this application. Figure 5 A flowchart illustrating a hybrid loss calculation step provided in an embodiment of this application is shown; Figure 6 This illustration shows a flowchart of a refinement process using a Stiefel manifold, as provided in an embodiment of this application. Figure 7 This illustration shows a schematic diagram of the overall process of training a semantic extraction model according to an embodiment of this application; Figure 8 A flowchart illustrating an application method for a semantic extraction model provided in an embodiment of this application is shown. Figure 9 A schematic diagram of the structure of a training device for a semantic extraction model provided in an embodiment of this application is shown; Figure 10 A schematic diagram of the structure of an application device for a semantic extraction model provided in an embodiment of this application is shown. Detailed Implementation

[0023] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0024] As mentioned earlier, semantic understanding methods have wide applications, such as in current cross-modal data retrieval systems and in fields involving semantic data entry and classification. Currently, existing semantic understanding methods typically employ a series of fixed steps to achieve joint embedding and representation learning of multimodal data. Specifically, the training process involves, for example, encoding images using a ResNet-152 model to generate 2048-D (Dimension) image feature vectors; encoding text using a BERT-base model to generate 768-D text feature vectors; and encoding audio using a VGGish model to generate 128-D feature vectors. These feature vectors from different modalities are then mapped to a unified 512-D semantic space through a multi-layer fully connected network and normalized. Finally, the system calculates the cosine similarity between the feature vectors of each modality and uses it as an attention weight to perform a weighted summation of the feature vectors from each modality, thus obtaining the final 512-D representation vector.

[0025] However, in the training process of existing semantic understanding models, the joint features obtained are initially limited by the dimensionality of each modality's features, resulting in a final joint feature set of 512 dimensions. This low spatial dimensionality makes it difficult to effectively carry the fine-grained semantic information of multimodal data, leading to excessively small Euclidean distances between similar samples. In some application scenarios, such as cross-modal retrieval, this manifests as low retrieval recall and poor recognition accuracy. Secondly, the current calculation of attention weights relies on cosine similarity, which is highly dependent on the feature vectors of the text modality, causing other information to be easily overlooked. Furthermore, if data from a particular modality is missing, the system performance will significantly degrade.

[0026] To at least improve one of the aforementioned technical problems, this embodiment proposes a training method for a semantic extraction model. This method can be applied to a training device or electronic device for a semantic extraction model, which can be installed or integrated into some training systems and, during runtime, can execute any of the semantic extraction model training methods mentioned below. Figure 1 As shown, the method includes: S101, acquire multimodal data samples.

[0027] The initial semantic extraction model acquires multimodal data samples. These multimodal data samples include image, text, and audio modalities, and must include at least two of these modalities. This means that a specific modality of data can be artificially introduced to fill in missing features, creating a specific application scenario. This allows the model to be trained to complete the missing modal features.

[0028] S102, the multimodal data samples are feature extracted by a preset encoder, and the extracted features are mapped to the semantic space of the target dimension to obtain the multimodal feature vector corresponding to the multimodal data samples.

[0029] The preset encoder includes encoders that extract features from data samples of each modality, mapping the data samples of each modality to higher-dimensional feature vectors, thereby adding more semantic information. In S102, unlike the encoders used in related technologies, the preset encoder used in this embodiment extracts features and maps the extracted features to a semantic space of the target dimension, enabling multimodal data to be mapped to a higher-dimensional space, thereby expressing semantic information more precisely.

[0030] S103 calculates the attention weights between multimodal feature vectors based on the temperature sparse attention mechanism.

[0031] Specifically, the temperature-sparse attention mechanism refers to setting a temperature parameter to determine the similarity between any two modalities, and a coefficient parameter to ignore a modality with poor correlation. The attention weights used in related techniques rely excessively on the text modality, resulting in a low proportion of information from other modalities. By introducing the temperature-sparse attention mechanism, it is more advantageous to adjust the attention weights between multimodal feature vectors, thus reducing the strong dependence on text modality data. Calculating the attention weights between multimodal feature vectors can, to some extent, be understood as the fusion of multimodal feature vectors, resulting in a higher-dimensional joint embedding vector that better represents semantic information.

[0032] S104. When a certain modality feature is detected to be missing, the missing modality feature is filled in by a hypergraph structure constructed from the multimodal feature vectors to obtain the filled multimodal feature vector.

[0033] It should be noted that a sparse parameter is introduced in S103, which can ignore modalities with poor correlation. That is, if a modal feature is detected as missing in S104, it may be because the multimodal data input in S101 itself lacks the corresponding modality, or it may be because the temperature-based sparse attention mechanism introduced in S103 causes the data for that modality to be ignored. To address this, S104 uses a hypergraph structure constructed from the multimodal feature vectors to complete the missing modal features, thereby improving the robustness of the application system using this model in the case of missing modalities.

[0034] S105: For the completed multimodal feature vector, calculate the mixed loss consisting of uniformity loss and contrast loss, and optimize the parameters in the temperature sparse attention mechanism based on the mixed loss.

[0035] Finally, to avoid using a single training objective, a hybrid loss consisting of uniformity loss and contrast loss is used to optimize the parameters in the temperature sparse attention mechanism.

[0036] In this embodiment, multimodal data samples are first acquired. These samples include at least two of image, text, and audio samples. Features are extracted from the multimodal data samples using a pre-defined encoder, and the extracted features are mapped to a semantic space of the target dimension to obtain multimodal feature vectors corresponding to the multimodal data samples. Attention weights between the multimodal feature vectors are calculated based on a temperature-sparse attention mechanism. If a missing modality feature is detected, the missing modality feature is completed using a hypergraph structure constructed from the multimodal feature vectors to obtain a completed multimodal feature vector. A hybrid loss consisting of uniformity loss and contrast loss is calculated for the completed multimodal feature vector, and the parameters in the temperature-sparse attention mechanism are optimized based on this hybrid loss. Compared with related technologies, first extracting features from the multimodal data samples using a pre-defined encoder and then mapping them to a semantic space of the target dimension effectively unifies the dimensions of data from different modalities and increases the dimension of the embedded semantic space, thereby improving the fine-grainedness of semantic information. Furthermore, when a missing modality feature is detected, the missing modality feature is completed using the topological relationships of the hypergraph structure to improve recognition performance. Finally, considering the distribution characteristics of the embedding vectors in high-dimensional space, the parameters in the model are optimized using a hybrid loss consisting of uniformity loss and contrastive loss to improve the overall accuracy of semantic extraction.

[0037] Optionally, features are extracted from multimodal data samples using a pre-defined encoder, and the extracted features are mapped to the semantic space of the target dimension to obtain multimodal feature vectors corresponding to the multimodal data samples. This includes: encoding image samples using the EfficientNetV2-XL model as an image encoder to obtain initial image features; encoding text samples using the DeBERTaV3-large model as a text encoder to obtain initial text features; encoding audio samples using the Audio Spectrogram Transformer model as an audio encoder to obtain initial audio features; and mapping the initial image features, initial text features, and initial audio features to the semantic space of the target dimension through a linear transformation layer to obtain multimodal feature vectors.

[0038] This embodiment explains how to obtain multimodal feature vectors corresponding to ultra-high-dimensional multimodal data samples through a preset encoder. First, the image samples, i.e., the input images, are adjusted to a preset size to facilitate model processing. Then, the EfficientNetV2-XL model extracts ultra-high-dimensional global feature vectors of 5376-D. These are then mapped to the target-dimensional semantic space through a linear transformation layer to obtain ultra-high-dimensional image feature vectors. In this embodiment, the target dimension is 2048-D. Before inputting the image samples into the EfficientNetV2-XL model, data augmentation processing can be performed to improve the model's generalizability.

[0039] For text samples, the text samples are input into the DeBERTaV3-large model to extract 1024-D feature vectors. Then, a linear transformation layer maps the 1024-D feature vectors to a target-dimensional 2048-D semantic space and performs L2 normalization to obtain the target-dimensional text feature vectors. In this embodiment, word segmentation and text completion can also be performed before inputting the text samples into the DeBERTaV3-large model.

[0040] For audio text, the audio text is processed using the Audio Spectrogram Transformer model to extract 768-D audio features. These 768-D features are then mapped to a target-dimensional 2048-D semantic space using a linear transformation layer and subjected to L2 normalization to obtain the target-dimensional audio feature vector. In this embodiment, data augmentation processing can also be performed before the audio text is input into the Audio Spectrogram Transformer model.

[0041] In one feasible embodiment, image samples are resized to a preset size, and the RandAugment(2,15) data augmentation method is used to randomly augment the images to improve the model's generalization ability. The augmented images are then fed into the EfficientNetV2-XL model to extract a 5376-D global feature vector. This is achieved through a linear transformation layer (weight matrix...). (Initialized as XavierUniform with a learning rate of 1e-3), the feature vectors are mapped to a 2048-D space and L2 normalized to obtain the 2048-D feature vectors of the image. .

[0042] For text samples, SentencePiece was used for word segmentation, with a maximum length of 128, and 15% of the tokens were randomly masked to simulate partial information loss in real-world scenarios. The processed text was then input into the DeBERTaV3-large model to extract the 1024-D feature vector at the [CLS] position. Then, it was passed through a linear transformation layer (weight matrix...). The feature vectors are mapped to a 2048-D space and then L2 normalized to obtain the 2048-D feature vectors of the text. .

[0043] For audio samples, the samples are transformed into 128×1024 Log-Mel spectrograms, and data augmentation is performed on the spectrograms using SpecAugment (freqmask=48, timemask=192). The augmented spectrograms are then fed into the Audio Spectrogram Transformer model to extract 768-D feature vectors. Next, a linear transformation layer (weight matrix...) is applied. The feature vectors are mapped to a 2048-D space and then L2 normalized to obtain the 2048-D feature vectors of the audio. By following the steps above, 2048-D feature vectors for images, text, and audio can be obtained.

[0044] Optionally, after extracting features from the multimodal data samples using a preset encoder and mapping the extracted features to the semantic space of the target dimension to obtain the multimodal feature vectors corresponding to the multimodal data samples, the method further includes: compressing the multimodal feature vectors to obtain the compressed representations corresponding to the multimodal feature vectors.

[0045] Specifically, the compression processing of multimodal feature vectors to obtain the compressed representation of the multimodal feature vectors includes: dividing the multimodal feature vectors into equal parts according to a preset number to obtain a preset number of feature sub-vectors; calculating the quantization value and residual value of each feature sub-vector; and constructing the compressed representation of each feature sub-vector based on the quantization value and residual value.

[0046] Can be combined Figure 2 As shown in this embodiment, a residual quantization compression method is proposed to reduce storage space and improve computational efficiency. It should be noted that this step only requires converting the multimodal data samples into multimodal feature vectors before using the residual quantization compression method to compress the multimodal feature vectors, thus using the compressed representation for training throughout the entire model training process. This effectively compresses high-dimensional feature vectors, significantly reducing storage space while maintaining high precision, providing an efficient data format for subsequent processing.

[0047] The specific steps are as follows: Divide the 2048-D feature vector into equal parts according to a preset number of segments, for example, into 8 segments. Each segment is 256-D. For each segment... Calculate the quantized value using Formula 1 : (Formula 1) in To quantize the step size, this embodiment uses 0.02. For random offsets, ) represents a random offset It is A random variable uniformly distributed between , where sign() is the sign function. " indicates rounding down. Then the residuals are calculated." and truncate the residual to Within the specified range, it is stored in float16 format. Each segment occupies 8 bytes of integer + 8 bytes of residual, totaling 64 bytes. Decompression is performed when a specific feature vector is actually used. Recovery of feature vectors, measured recovery error The impact on subsequent calculations is negligible.

[0048] Optionally, the multimodal feature vector includes image feature vectors, text feature vectors, and audio feature vectors; Based on a temperature-based sparse attention mechanism, the attention weights between multimodal feature vectors are calculated, including: calculating global statistics of image feature vectors; using the global statistics as input to a multilayer perceptron to generate temperature parameters; using the temperature parameters to determine the similarity between any two modalities; using the temperature parameters to calculate a similarity matrix between multimodal feature vectors; calling pre-set sparse parameters and calculating a sparse mask based on the sparse parameters, temperature parameters, and similarity matrix; using the sparse mask to ignore modal information with correlation below a preset threshold; and calculating the attention weights between each modality based on the sparse mask and the similarity matrix.

[0049] After obtaining the multimodal feature vectors of 2048-D, these feature vectors are fused using a temperature-sparse attention mechanism to optimize the fusion effect of the multimodal data. This can be combined with... Figure 3 As shown in this embodiment, the process of calculating the attention weights between multimodal feature vectors using a temperature sparse attention mechanism is explained. Firstly, using image feature vectors... Calculate the global statistic. Then, process the global statistic using a multilayer perceptron to obtain the temperature parameter. Specifically, the global statistic is calculated using Formula 2: (Formula 2) In Formula 2, These are global statistics, designed to stably represent the overall distribution of the image feature vector. `mean()` represents the mean of all sampled points in the image feature vector, describing the overall average level of the feature vector; `std()` represents the variance of all sampled points in the image feature vector, describing the overall fluctuation of the feature vector. Furthermore, the global statistics are processed using two layers of MLP (256 hidden layers, GELU activation function). Temperature parameters are obtained through processing. ,in: (Formula 3) In Formula 3, softplus() is a smooth activation function used to maintain gradient stability during training. It is a learnable weight matrix. The learnable range is 0.01 to 10. Next, the original similarity matrix is ​​calculated using Formula 4: (Formula 4) In Formula 4, It is the similarity matrix between the image feature vector and the feature vector of another modality, where That is, text feature vectors or audio feature vectors. This represents the vector transpose. Therefore, the temperature parameter... To determine the similarity between any two modalities, this embodiment employs an attention weight calculation method that differs from related technologies that calculate cosine similarity based on text feature vectors, thus eliminating the strong dependence on text feature vectors. Furthermore, to further optimize the attention weights, sparse parameters are introduced. and The sparse mask is calculated using Formula 5, where the sparse mask is used to ignore modal information with correlation below a preset threshold.

[0050] (Formula 5) In Formula 5, It is a sparse mask, essentially an element in the topological relationship matrix between modes, representing the strength of the topological relationship between the i-th mode and the j-th mode. ReLU() is the activation function. ( ) is the hyperbolic tangent function. If the sparse mask is less than the preset threshold, it proves that the information of this modality may be directly sparsified to 0, that is, ignored. Finally, the attention weights are calculated using Formula 6: (Formula 6) In Formula Six, represents the attention weight between the i-th modal feature vector and the j-th modal feature vector; exp() is an exponential function used to calculate the exponential weighted similarity measure and modal relationship strength. The strength of the topological relationship between the i-th and k-th modes. This embodiment uses a temperature-sparse dual-parameter attention mechanism to dynamically adjust the weights of each mode, rather than relying too heavily on text modality information by calculating cosine similarity based on text feature vectors, thus avoiding the over-subjection of information from any particular mode. Specifically, for example, sparse parameters... The initial value is set to 1.2. The initial value is set to 0.45. For a sample of "white Samoyed running in the snow", the attention weight from image modality features to text modality features is... Attention weights from image modal features to audio modal features If the audio modality is missing, Automatically adjusted to 0.34, the audio branch was sparsified to 0, and the weight was... By increasing the resolution to 0.81, the system automatically focuses on the text modality, avoiding the introduction of noise. This mechanism significantly improves the flexibility and robustness of multimodal data fusion.

[0051] Optionally, when a certain modality feature is detected to be missing, the missing modality feature is filled in using a hypergraph structure constructed from multimodal feature vectors to obtain a filled multimodal feature vector. This includes: constructing a hypergraph structure using feature vectors from image feature vectors, text feature vectors, and audio feature vectors that are not missing modalities as nodes; generating virtual nodes for the missing modality feature vectors and adding the virtual nodes as hyperedges to the hypergraph structure; and filling in the missing modality feature using the neural network corresponding to the hypergraph structure.

[0052] In this embodiment, as Figure 4 As shown, to avoid the impact of missing modality information on the semantic extraction performance of the system, the missing modality features are filled in using the topological relationship of the neural network corresponding to the hypergraph structure. Specifically, the three-modality feature vector of a sample is treated as nodes of a uniform hyperedge to construct the hypergraph structure. It should be noted that here, the three modality feature vectors, including the input multimodal data samples, do not have missing modalities, and no modality is ignored through sparsity during the attention weight calculation. Alternatively, if a modality is missing, its features can be filled in before constructing the hypergraph structure. For example, to detect missing modalities, a two-layer MLP (128 hidden layers, ReLU activation function) is used, with the input being... , representing the average of the image feature vector and the text feature vector, outputs the completion probability p, and sets the threshold to . =0.5. If p> This indicates that the audio feature vector of the sample is missing. Therefore, in this case, a virtual node for the audio feature vector is generated and a hyperedge is added.

[0053] After adding virtual nodes with missing modalities to the hypergraph structure, the missing modal features are completed through the neural network corresponding to the hypergraph structure, which is also known as message passing, as shown in Equation 7: (Formula 7) In Formula 7, This represents the feature matrix obtained after processing by the Hypergraph Neural Network (HGNN), which is also the completed multimodal feature vector. This represents the feature matrix of the current layer, which is also the multimodal feature vector before completion. It is the sigmoid activation function, used to ensure that the attention weights of the output are in the range [0, 1], D v D is the degree matrix of the nodes. e It is the degree matrix of the hyperedge. A learnable diagonal weight matrix. This process effectively completes missing modal information, significantly improving the system's robustness in the case of missing modalities.

[0054] Finally, after completing the topology, the distribution of embedding points in the high-dimensional space needs to be further optimized using a hybrid loss function. Specifically, for the completed multimodal feature vectors, a hybrid loss consisting of uniformity loss and contrast loss is calculated, and the parameters in the temperature-sparse attention mechanism are optimized based on this hybrid loss.

[0055] Specifically, such as Figure 5 As shown, the completed feature vector is subjected to spherical normalization to obtain the joint embedding vector. Then, the uniformity loss is calculated using Formula 8 for the joint embedding vectors.

[0056] (Formula 8) In Formula 8, This is a uniform loss function used to optimize the distribution of the joint embedding vector in a high-dimensional space; It is a joint embedding vector The covariance matrix; is a constant term to avoid the singular matrix problem in numerical calculations; d is the vector dimension. ( ) is the natural logarithm of the logarithmic determinant of a logarithm. In this embodiment... d=2048. The purpose of the uniformity loss function is to improve the generalization ability and clustering effect of the model by maximizing the logarithm of the determinant of the covariance matrix of the embedding vectors, so that the embedding vectors are uniformly distributed in the high-dimensional space.

[0057] Next, the comparative loss is calculated using Formula Nine. In this embodiment, temperature parameters are taken. InfoNCE loss function: (Formula 9) In Formula Nine, To compare the losses, N is the batch size, which is 8192 in this embodiment; and These are the joint embedding vectors for modes i and z, respectively.

[0058] Total loss function ,in =0.2 is the weight coefficient for the uniformity loss. Further, the LAMB optimizer is used for training, with a learning rate set to 2e-3, a weight decay coefficient of 0.01, and a training duration of 100 epochs. In the batch containing the above samples, , The total loss is 7.01. Through gradient backpropagation, the temperature parameter... and sparse parameter α, The process involves updating the attention weights, which are then significantly redistributed in the next round, further optimizing the embedding performance. This process significantly improves the uniformity and clustering effect of the embedding points in the high-dimensional space, providing higher-quality feature representations for downstream tasks.

[0059] Optionally, after calculating a mixed loss consisting of uniformity loss and contrast loss on the completed multimodal feature vector, and optimizing the parameters in the temperature sparse attention mechanism based on the mixed loss, the method further includes optimizing the multimodal feature vector through the Stiefel manifold.

[0060] In this embodiment, as Figure 6 As shown, to further improve the optimization embedding effect, the joint embedding vector is transferred to the Stiefel manifold for L-BFGS refinement at regular training epochs. The Grassmann gradient is calculated. Specifically, the multimodal feature vectors are optimized using the Stiefel manifold, processed according to Equation 10: (Formula 10) In Formula 10, L represents the gradient computed on the Stiefel manifold, used to optimize the multimodal feature vector X. L is the hybrid loss computed based on uniformity loss and contrastive loss, and X is the multimodal feature vector. Here, the multimodal feature vector referred to as X is a joint embedding vector based on the fusion of multimodal feature vectors.

[0061] Figure 7 This document illustrates the overall flowchart of the training method for a semantic extraction model proposed in this embodiment. Through ultra-high-dimensional joint embedding technology, multiple modalities such as images, text, and audio are mapped to a unified high-dimensional semantic space, greatly enriching the semantic expressive power of the data and fully showcasing fine-grained semantic information. This effectively improves the recall rate of cross-modal retrieval, providing users with more accurate search results. Secondly, the introduced temperature-sparse attention mechanism dynamically adjusts the importance weights of each modality based on the semantic relevance of the data, avoiding excessive dominance of any one modality and ensuring the balance and flexibility of multimodal data fusion. This significantly enhances the system's robustness when facing modality missing or incomplete data. Furthermore, the application of a topology completion hypergraph neural network further explores the topological relationships between modalities, achieving effective completion of missing modality information and further improving the system's robustness. Even with missing data in some modalities, the system can still maintain high performance. Furthermore, the design of the hybrid loss function balances the uniform distribution of embedding points and semantic clustering effects, optimizing the geometric structure of the embedding space. This results in a more uniform distribution of embedded data in the high-dimensional space, leading to better clustering and providing higher-quality feature representations for downstream tasks, further enhancing the overall system performance. Finally, the application of residual quantization compression effectively reduces the storage space of the embedding vectors, lowers computational costs, and maintains high data accuracy. This makes the system more efficient in large-scale data processing, enabling it to quickly respond to user query requests and meet the needs of real-time retrieval and other application scenarios. In summary, the technical solution proposed in this embodiment improves cross-modal data processing performance while also considering system robustness, storage efficiency, and computational efficiency, demonstrating significant practical value and broad application prospects.

[0062] like Figure 8 As shown, this embodiment also provides an application method for a semantic extraction model, which can be applied to some retrieval systems or semantic classification systems to perform retrieval based on the semantics of the input content. The method includes: S201, Obtain the search information input by the user; In S201, the retrieved information includes at least one of image information, text information, or audio information. In practical applications, users only need to input one type of retrieved information, as the model, after the above training, already possesses the ability to complete multimodal information. For example, when only an image is input, the system will extract image features and generate virtual nodes for text and audio to complete the missing modalities, thus completing the retrieval even if other modal data are missing.

[0063] The actual application of the model is the downstream task after model training. After model training, each sample information is finally stored in the database in a compressed representation after residual quantization compression. For example, each sample is ultimately stored in the form of 64 bytes of quantization code + 64 bytes of residual. During retrieval, the stored quantization code and residual are first decompressed to restore the 2048-D feature vector, and then the retrieval is performed. The latency of a single retrieval is less than 5 milliseconds.

[0064] S202 uses a trained semantic extraction model to retrieve information and obtain search results.

[0065] The completed semantic extraction model is trained based on the training method of the semantic extraction model mentioned in any of the above embodiments.

[0066] Furthermore, as Figures 1 to 7 The specific implementation of the method shown in this embodiment provides a training device for a semantic extraction model, such as... Figure 9 As shown, the device includes: an acquisition unit 901, an extraction unit 902, a calculation unit 903, a completion unit 904, and an optimization unit 905.

[0067] The acquisition unit 901 is configured to acquire multimodal data samples; the multimodal data samples include at least two of image samples, text samples, and audio samples; The extraction unit 902 is configured to extract features from the multimodal data sample using a preset encoder, and map the extracted features to the semantic space of the target dimension to obtain the multimodal feature vector corresponding to the multimodal data sample. The computing unit 903 is configured to calculate the attention weights between the multimodal feature vectors based on a temperature sparse attention mechanism. The completion unit 904 is configured to, when a certain modality feature is detected to be missing, complete the missing modality feature through a hypergraph structure constructed from the multimodal feature vectors to obtain a completed multimodal feature vector; wherein, the hypergraph structure is constructed from the multimodal feature vectors; The optimization unit 905 is configured to calculate a hybrid loss consisting of uniformity loss and contrast loss for the completed multimodal feature vector, and optimize the parameters in the temperature sparse attention mechanism based on the hybrid loss.

[0068] In a specific application scenario, the extraction unit 902 is specifically configured to extract features from the multimodal data samples using a preset encoder, and map the extracted features to the semantic space of the target dimension to obtain the multimodal feature vector corresponding to the multimodal data samples. This includes: encoding the image samples using an EfficientNetV2-XL model as an image encoder to obtain initial image features; encoding the text samples using a DeBERTaV3-large model as a text encoder to obtain initial text features; encoding the audio samples using an Audio Spectrogram Transformer model as an audio encoder to obtain initial audio features; and mapping the initial image features, the initial text features, and the initial audio features to the semantic space of the target dimension through a linear transformation layer to obtain the multimodal feature vector.

[0069] In specific application scenarios, the extraction unit 902 is further configured to compress the multimodal feature vector to obtain the compressed representation corresponding to the multimodal feature vector; The step of compressing the multimodal feature vector to obtain the compressed representation of the multimodal feature vector includes: dividing the multimodal feature vector into a preset number of segments to obtain a preset number of feature sub-vectors; calculating the quantization value and residual value of each feature sub-vector; and constructing a compressed representation of each feature sub-vector based on the quantization value and residual value.

[0070] In a specific application scenario, the computing unit 903 is further configured to: calculate the global statistics of the image feature vector; use the global statistics as input to the multilayer perceptron to generate temperature parameters; use the temperature parameters to determine the similarity between any two modalities; use the temperature parameters to calculate the similarity matrix between the multimodal feature vectors; call a pre-set sparse parameter, and calculate a sparse mask based on the sparse parameter, the temperature parameter, and the similarity matrix; use the sparse mask to ignore modal information with correlation below a preset threshold; and calculate the attention weights between each modality according to the sparse mask and the similarity matrix.

[0071] In specific application scenarios, the completion unit 904 is further configured to construct the hypergraph structure using the feature vectors of the image feature vector, the text feature vector, and the audio feature vector that do not have missing modal features as nodes; generate virtual nodes for the missing modal feature vectors and add the virtual nodes as hyperedges to the hypergraph structure; and complete the missing modal features through the neural network corresponding to the hypergraph structure.

[0072] In specific application scenarios, the optimization unit 905 is further configured to optimize multimodal feature vectors through the Stiefel manifold.

[0073] It should be noted that other corresponding descriptions of the functional units involved in the semantic extraction model training device provided in this embodiment can be found in [reference needed]. Figures 1 to 7 The corresponding description in [the document] will not be repeated here.

[0074] Furthermore, as Figure 8 The specific implementation of the method shown in this embodiment provides an application device for a semantic extraction model, such as... Figure 10 As shown, the device includes an acquisition unit 1001 and a retrieval unit 1002.

[0075] The acquisition unit 1001 is configured to acquire search information input by the user; the search information includes at least one of image information, text information, or audio information. The retrieval unit 1002 is configured to retrieve the retrieval information using a trained semantic extraction model to obtain retrieval results; The completed semantic extraction model is obtained according to the training method of the semantic extraction model as described in any of the above embodiments.

[0076] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented in hardware. By applying the scheme of this embodiment, compared with related technologies, firstly, after extracting features from multimodal data samples using a preset encoder, and mapping them to the semantic space of the target dimension, it is possible to effectively unify the dimensions of data from different modalities and increase the dimension of the embedded semantic space, thereby improving the fine-grainedness of semantic information. Furthermore, when a missing modal feature is detected, the missing modal feature is completed through the topological relationship of the hypergraph structure to improve recognition performance. Finally, considering the distribution characteristics of the embedded vectors in high-dimensional space, the parameters in the model are optimized through a hybrid loss consisting of uniformity loss and contrast loss to improve the overall accuracy of semantic extraction.

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for training a semantic extraction model, the method comprising: The method comprises: acquiring a multi-modal data sample; the multi-modal data sample comprises at least two of an image sample, a text sample and an audio sample; extracting features of the multi-modal data sample by a preset encoder, and mapping the extracted features to a semantic space of a target dimension to obtain a multi-modal feature vector corresponding to the multi-modal data sample; based on a temperature sparse attention mechanism, calculating attention weights between the multi-modal feature vectors; in the case of detecting the absence of a certain modal feature, completing the missing modal feature through a hypergraph structure constructed by the multi-modal feature vectors to obtain a completed multi-modal feature vector; calculating a mixed loss composed of uniformity loss and contrast loss on the completed multi-modal feature vector, and optimizing parameters in the temperature sparse attention mechanism according to the mixed loss.

2. The method of claim 1, wherein, The multi-modal data sample comprises an image sample, a text sample and an audio sample; extracting features of the multi-modal data sample by a preset encoder, and mapping the extracted features to a semantic space of a target dimension to obtain a multi-modal feature vector corresponding to the multi-modal data sample, comprising: encoding the image sample by an EfficientNetV2-XL model as an image encoder to obtain initial image features; encoding the text sample by a DeBERTaV3-large model as a text encoder to obtain initial text features; encoding the audio sample by an Audio Spectrogram Transformer model as an audio encoder to obtain initial audio features; mapping the initial image features, the initial text features and the initial audio features to a semantic space of a target dimension through a linear transformation layer to obtain the multi-modal feature vector.

3. The method of claim 2, wherein, After extracting features of the multi-modal data sample by a preset encoder and mapping the extracted features to a semantic space of a target dimension to obtain a multi-modal feature vector corresponding to the multi-modal data sample, the method further comprises: compressing the multi-modal feature vector to obtain a compressed representation corresponding to the multi-modal feature vector; the compression of the multi-modal feature vector to obtain a compressed representation corresponding to the multi-modal feature vector comprises: equally dividing the multi-modal feature vector by a preset number to obtain a feature sub-vector of a preset number of segments; calculating a quantization value and a residual value of each segment of the feature sub-vector; based on the quantization value and the residual value, constructing a compressed representation of each segment of the feature sub-vector.

4. The method of claim 1, wherein, The multi-modal feature vector comprises an image feature vector, a text feature vector and an audio feature vector; the calculation of the attention weights between the multi-modal feature vectors based on the temperature sparse attention mechanism comprises: calculating a global statistic of the image feature vector; taking the global statistic as an input of a multi-layer perception machine to generate a temperature parameter; the temperature parameter is used to determine the similarity between any two modalities; calculating a similarity matrix between the multi-modal feature vectors by using the temperature parameter; Call the preset sparse parameter, and calculate a sparse mask based on the sparse parameter, the temperature parameter and the similarity matrix; the sparse mask is used to ignore modal information with a correlation lower than a preset threshold; According to the sparse mask and the similarity matrix, the attention weight between each modality is calculated.

5. The method of claim 1, wherein, The multi-modal feature vector includes an image feature vector, a text feature vector and an audio feature vector; In the case where it is detected that a certain modal feature is missing, the missing modal feature is completed through a hypergraph structure constructed by the multi-modal feature vector to obtain a completed multi-modal feature vector, including: The hypergraph structure is constructed by taking the modal feature vectors that are not missing in the image feature vector, the text feature vector and the audio feature vector as nodes; A virtual node is generated for the missing modal feature vector, and the virtual node is added to the hypergraph structure as a hyperedge; The missing modal feature is completed through the neural network corresponding to the hypergraph structure.

6. The method of claim 5, wherein, The missing modal feature is completed through the neural network corresponding to the hypergraph structure, and is processed according to the following formula: wherein, denotes the completed multi-modal feature vector, is the multi-modal feature vector before completion; is a Sigmoid activation function to ensure the output attention weight is in the range of [0, 1], D v is the degree matrix of nodes, D e is the degree matrix of hyper-edges, is a learnable diagonal weight matrix.

7. The method of claim 1, wherein, After calculating the mixed loss composed of uniformity loss and contrast loss on the completed multi-modal feature vector, and optimizing the parameters in the temperature sparse attention mechanism according to the mixed loss, the method further includes: Optimizing the multi-modal feature vector through Stiefel manifold; Optimizing the multi-modal feature vector through Stiefel manifold is processed according to the following formula: wherein represents the gradient computed on the Stiefel manifold for optimizing the multi-modal feature vector X, L is a hybrid loss computed from the uniformity loss and the contrast loss, and X represents a multi-modal feature vector.

8. An application method of a semantic extraction model, characterized by, It includes: Obtain the retrieval information input by the user; The retrieval information includes at least one of image information, text information or audio information; Retrieving the retrieval information through the trained semantic extraction model to obtain a retrieval result; The trained semantic extraction model is obtained according to the training method of the semantic extraction model in any one of claims 1-7.

9. A training device for a semantic extraction model, characterized in that, It includes: The acquisition unit is configured to acquire multi-modal data samples; the multi-modal data samples include at least two of image samples, text samples and audio samples; The extraction unit is configured to extract features from the multi-modal data samples through a preset encoder, and map the extracted features to a semantic space of a target dimension to obtain a multi-modal feature vector corresponding to the multi-modal data samples; The calculation unit is configured to calculate the attention weight between the multi-modal feature vectors based on a temperature sparse attention mechanism; The completion unit is configured to, in the case where it is detected that a certain modal feature is missing, complete the missing modal feature through a hypergraph structure constructed by the multi-modal feature vector to obtain a completed multi-modal feature vector; the hypergraph structure is constructed by the multi-modal feature vector; The optimization unit is configured to calculate a mixed loss composed of uniformity loss and contrast loss on the completed multi-modal feature vector, and optimize the parameters in the temperature sparse attention mechanism according to the mixed loss.

10. An application device for a semantic extraction model, characterized in that, It includes: The acquisition unit is configured to acquire retrieval information input by the user; The search information includes at least one of image information, text information, or audio information; The search unit is configured to search the search information by using a trained semantic extraction model to obtain a search result. The trained semantic extraction model is obtained according to the training method of the semantic extraction model in any one of claims 1-7.

Images