Learning video recommendation methods, media, and devices based on modal semantic space alignment

Abstract

The present invention proposes a learning video recommendation method, medium, and device based on modal semantic space alignment, which relates to the field of multimodal semantic space alignment technology. The method includes: extracting user behavior vectors and vectors of different modalities of learning videos, including text, image, audio, and structure vectors; using a multi-layer perceptron to project the extracted vectors of different modalities into a common semantic space, and within the common semantic space, vector splicing the structure vector with the text vector, image vector, and audio vector respectively; performing modal comparison and modal matching on the spliced ​​vectors, and obtaining a multimodal feature vector of the learning video by optimizing the semantic alignment between different modalities; fusing the multimodal feature vectors, calculating the similarity between the user behavior vector and the fused multimodal feature vector using cosine similarity, and ranking and recommending all videos. The present invention can fully understand the modal semantic structure of learning videos, achieving accurate matching and recommendation.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal semantic space alignment, and in particular to a learning video recommendation method, medium, and device based on modal semantic space alignment. Background Art

[0002] With the rapid development of online education and the increasing abundance of learning resources, learning videos, as a crucial teaching resource in the field, often exhibit multimodal content. For example, a video's subtitles, title, description, cover image, keyframes, and even the voice information within the video audio all contain rich instructional semantic information. Combining this modal information, recommendation systems can recommend personalized learning videos based on users' learning interests and needs, thereby improving learning efficiency. For example, a user's learning behavior data (such as viewing history, click history, likes, and comments) can reflect their learning goals, while the multimodal features of learning videos can comprehensively showcase their content themes and knowledge points. By integrating user behavior with multimodal video data, the system can provide precise learning resource recommendations, optimize learning paths, and enhance user experience and learning outcomes.

[0003] Multimodal recommendation systems demonstrate significant advantages in recommendation tasks by integrating data from multiple modalities (such as text, images, audio, video, and user behavior). Compared to traditional single-modal recommendation systems, multimodal recommendation systems can more comprehensively understand user interests and recommended content, and provide more precise and personalized services.

[0004] Traditional multimodal information fusion methods typically simply concatenate or combine features from different modalities, failing to effectively capture the semantic relationships between modalities. This can lead to loss or redundancy of modal information in the learning video. These deficiencies make it difficult for existing recommendation systems to accurately model user preferences and fail to fully understand the modal semantic structure of the learning video, thus affecting the accuracy and performance of recommendation results. Summary of the Invention

[0005] The purpose of the present invention is to solve the problem that existing multimodal information fusion methods cannot effectively capture the semantic relationship between modalities. A learning video recommendation method based on modal semantic space alignment is proposed, which includes the following steps:

[0006] S1. Obtain user behavior information and learn the text, image, and audio information of the video;

[0007] S2. Extract user behavior vectors, extract text vectors, image vectors, and audio vectors from the learning videos; use a multi-layer graph neural network to extract features from the structural information between the learning videos, generate structural vectors, and obtain vectors of different modalities of the learning videos;

[0008] S3. Project the extracted text vector, image vector, audio vector, and structure vector into a common semantic space using a multi-layer perceptron. In the common semantic space, concatenate the structure vector with the text vector, image vector, and audio vector respectively.

[0009] S4. Perform modal comparison and modal matching on the concatenated text vector, image vector, and audio vector, and obtain a multimodal feature vector of the learning video by optimizing the semantic alignment between different modalities.

[0010] S5. Fuse the multimodal feature vectors of the learning video, calculate the similarity between the user behavior vector and the fused multimodal feature vector of the learning video through cosine similarity, and rank and recommend all videos.

[0011] Furthermore, the RNN model is used to extract user behavior vectors , using the BERT model to extract text vectors from learning videos , use ResNet model to extract image vector , using MFCC (Mel-frequency cepstralcoefficients) feature extraction method to extract audio vector .

[0012] Furthermore, the structure vector is based on the graph Construction,node V represents learning videos, edge E represents the relationship between videos, and edge types include: video knowledge point overlap, user behavior association of watching videos, and video content similarity;

[0013] Define edge weights based on different types of edges:

[0014] ,

[0015] in, express and The degree of overlap of knowledge points, and Represent the knowledge point sets of the i-th video and the j-th video respectively, express and The number of knowledge points in the intersection, express and The number of knowledge points in the collection;

[0016] represents the association weight of user behaviors between watching the i-th video and the j-th video, and measures the similarity of user behaviors between the i-th video and the j-th video;

[0017] ,

[0018] in, represents the content similarity weight between the i-th video and the j-th video, and are the text feature vectors of the i-th video and the j-th video respectively;

[0019] Construct the adjacency matrix:

[0020] ,

[0021] in, represents the edge weight between the i-th video and the j-th video, is the weight parameter of the relationship type.

[0022] Furthermore, a multi-layer graph neural network is used to extract features from the structural information between learning videos and generate a structural vector, which is expressed as:

[0023] ,

[0024] ,

[0025] in, represents the structure vector, represents the feature representation of the i-th node in the L-th layer, where L represents the number of graph convolution layers. represents the feature representation of the i-th node in the l+1 layer, r represents the relationship, represents the neighbor nodes of the i-th node, represents the sigmoid activation function, Representing relationships The normalized attention weights of the next i-th node and the j-th node, Representing relationships The adjacency matrix of the i-th node and the j-th node, represents the feature representation of the jth neighbor node of the i-th node in the l-th layer, Representing relationships The weight matrix at layer l;

[0026] ,

[0027] in, Representing relationships The attention of the next i-th node and j-th node, Representing relationships The attention of the i-th node and the k-th node, the k-th node is the neighbor node of the i-th node, Represents the neighbor node of the i-th node;

[0028] ,

[0029] Among them, MLP represents multi-layer perceptron, represents the feature representation of the i-th node in the l-th layer, Indicates the type of relationship between the i-th node and the j-th node.

[0030] Furthermore, the extracted text vector, image vector, audio vector and structure vector are projected into a common semantic space using a multi-layer perceptron, which is expressed as:

[0031] ,

[0032] in, 、 、 、 Represent the text vector, image vector, audio vector, and structure vector projected into the common semantic space, respectively. 、 、 、 Represent the text vector, image vector, audio vector, and structure vector before being projected into the common semantic space, respectively. and They represent the weight matrix, represents the sigmoid activation function, and represent the bias terms respectively.

[0033] Furthermore, the concatenated text vector, image vector, and audio vector are subjected to modal comparison and modal matching. The loss functions of modal comparison and modal matching are:

[0034] ,

[0035] in, represents the loss of modal comparison and modal matching, represents the modality contrast loss, represents the modality matching loss;

[0036] ,

[0037] in, Expressing hope, They represent the hyperparameters for adjusting the loss weight, represents the cross entropy loss, Indicates the actual matching labels of text modality and image modality, Represents the similarity from text modality to image modality, Indicates the actual matching labels of text modality and audio modality, Represents the similarity from text modality to audio modality, represents the true matching labels of image modality and text modality, represents the similarity from image modality to text modality, represents the true matching labels of the image modality and audio modality, represents the similarity from the image modality to the audio modality, Indicates the actual matching labels of audio modality and text modality, Indicates the similarity from audio modality to text modality, represents the true matching labels of the audio modality and image modality, Represents the similarity from audio modality to image modality;

[0038] ,

[0039] in, is the true label, y=1 indicates a match, y=0 indicates a mismatch; is the matching probability predicted by the model.

[0040] The present invention also proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0041] The present invention also proposes an electronic device, comprising a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0042] The beneficial effects brought about by the technical solution provided by the present invention are:

[0043] This paper proposes a learning video recommendation method based on modal semantic space alignment. By mapping the multimodal features of a learning video, such as text, image, audio, and structure, into a unified semantic space and combining them with the user's learning behavior characteristics, it constructs high-quality user interest vectors and video feature vectors. This method fully understands the modal semantic structure of the learning video and enables precise matching and recommendation. This method solves the semantic alignment problem between multimodal features through modal contrast learning and modal matching optimization. It also avoids the loss of modal features found in traditional methods through dynamic weighted fusion technology, significantly improving the performance of the recommendation system and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flowchart of a method for learning video recommendation based on modal semantic space alignment according to an embodiment of the present invention;

[0045] Figure 2 It is a block diagram of an electronic device in an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0047] The flowchart of the learning video recommendation method based on modal semantic space alignment according to the embodiment of the present invention is as follows: Figure 1 , specifically including the following steps:

[0048] S1. Obtain user behavior information, including text, image, and audio information from the learning video. User behavior information, such as viewing history and clickthrough data, can reflect their interest in the learning video. Learning videos contain multimodal information, and each source or form of information can be referred to as a modality. In this invention, the multimodality of learning videos includes text, images, and audio.

[0049] S2. Extract user behavior vectors, extract text vectors, image vectors, and audio vectors from learning videos; use a multi-layer graph neural network to extract features from the structural information between learning videos, generate structural vectors, and obtain vectors of different modalities of learning videos.

[0050] Specifically, the RNN model is used to extract user behavior vectors , which can be expressed as:

[0051] ,

[0052] ,

[0053] ,

[0054] in, represents the hidden state at time step t, represents the weight matrix input to the hidden layer, represents the input at time step t, Represents the weight matrix from the previous hidden state to the current hidden state, represents the hidden state at time step t-1, is the bias term, is the activation function, represents the hidden state at the last time step, represents the input of the last time step, represents the hidden state at the T-1 time step, Represents the user behavior vector, where the input is the user's behavior information.

[0055] Use the BERT model to extract text vectors from learning videos , text information ,in Represents the length of the text after preprocessing. First, a shallow convolutional network is used to perform window operations. The process is as follows:

[0056] ,

[0057] in, Indicates the text information after the window operation. represents a nonlinear function, represents a full connection operation, represents the feature transformation function, Represents a splicing operation, represents the convolution kernel, W represents the mapping matrix, Represents a convolution operation.

[0058] Will get Feed it into the BERT model and generate text feature vectors through a multi-layer Transformer encoder .

[0059] Use ResNet model to extract image vectors , the specific formula is as follows:

[0060] ,

[0061] in, represents the input of layer l+1, represents the activation function, represents the convolution kernel of layer l, represents the input of layer l, represents the bias of layer l, Represents residual connection, which is used to capture multi-level features, and then global average pooling (GAP) is used to obtain the image feature vector :

[0062] ,

[0063] in, represents the vector extracted from the i-th image information, and N represents the number of image information.

[0064] Use MFCC (Mel-frequency cepstral coefficients) feature extraction method to extract audio vectors , first pre-emphasize the audio signal to enhance the high-frequency part, then perform Fourier transform on each frame of audio signal to obtain the spectrum:

[0065] ,

[0066] in, is the frame length, is the original audio signal, It is the audio signal after Fourier transform, that is, the spectrum of the audio signal.

[0067] The spectrum of the audio signal passes through the Mel filter bank to obtain the Mel spectrum: ,in is the frequency; then perform discrete cosine transform (DCT) on the Mel spectrum to obtain the MFCC coefficient: , represents the energy output by the kth Mel filter, K represents the total number of Mel filter banks, n represents the frequency index, and the MFCC coefficients are mean-pooled to obtain the audio feature vector: .

[0068] The structural vector is extracted through graph neural network. First, a graph is constructed ,Node V represents a learning video, edge E represents the relationship between videos, and the edge weight reflects the strength of the relationship.,The edge types include: video knowledge point overlap, user behavior association of watching videos, and video content similarity.

[0069] Based on different types of edges, edge weights are defined. For overlapping knowledge points, the edge weights are calculated as follows:

[0070] ,

[0071] in, express and The degree of overlap of knowledge points, and Represent the knowledge point sets of the i-th video and the j-th video respectively, express and The number of knowledge points in the intersection, express and The number of knowledge points in the collection.

[0072] For user behavior association, use represents the association weight of user behavior between watching the i-th video and the j-th video, and measures the similarity in user behavior between the i-th video and the j-th video.

[0073] For content similarity relations, edge weights are calculated as follows:

[0074] ,

[0075] in, represents the content similarity weight between the i-th video and the j-th video, and are the text feature vectors of the i-th video and the j-th video respectively; by calculating the cosine similarity of these two vectors, the similarity of the video content can be evaluated.

[0076] Based on the above relationships, the adjacency matrix is ​​constructed:

[0077] ,

[0078] in, represents the edge weight between the i-th video and the j-th video, is the weight parameter of the relationship type.

[0079] In learning video relationship graphs, different types of relationships (knowledge points, user behaviors, content similarity) may contribute differently to node features. A relationship attention mechanism is introduced to dynamically assign weights to each relationship to calculate the importance of the relationship.

[0080] First, an initial feature vector is assigned to each video node ,This vector can be the original features of the node, such as the metadata of the video (length, resolution, etc.), or it can be obtained through some ,feature extraction method.

[0081] Next, a multi-layer perceptron (MLP) is used to calculate the relationship attention between nodes. For each pair of nodes, the relationship attention Calculated as follows: , where MLP represents a multi-layer perceptron, represents the feature representation of the i-th node in the l-th layer, represents the feature representation of the jth neighbor node of the i-th node in the l-th layer, Indicates the type of relationship between the i-th node and the j-th node, and the semicolon indicates a vector join operation.

[0082] In order to convert the attention scores into weights and ensure that these weights can reasonably represent the relative importance between nodes, normalization is required, which is achieved through the softmax function: ,in, Representing relationships The attention of the next i-th node and j-th node, Representing relationships The attention of the i-th node and the k-th node, the k-th node is the neighbor node of the i-th node, Represents the set of neighbor nodes of the i-th node. Normalization ensures that the sum of the attention weights of all neighbor nodes is 1, allowing it to pay more attention to important neighbor nodes.

[0083] Using the calculated attention weights, the features of neighboring nodes are weighted summed to aggregate information: ,in, represents the feature representation of the i-th node in the l+1 layer, r represents the relationship, represents the neighbor nodes of the i-th node, represents the sigmoid activation function, Representing relationships The normalized attention weights of the next i-th node and the j-th node, Representing relationships The adjacency matrix of the i-th node and the j-th node, represents the feature representation of the jth neighbor node of the i-th node in the l-th layer, Representing relationships The weight matrix at layer l.

[0084] pass After layer graph convolution and relational attention mechanism, the final structure embedding vector of each video node is generated: ,in, represents the structure vector, represents the feature representation of the i-th node in the L-th layer, where L represents the number of graph convolution layers. It is a comprehensive representation of the local features, neighborhood information, and global structural relationships of the learning video. It includes the original features of the video and also incorporates its complex relationships in the graph structure, enabling a deeper understanding of the video content.

[0085] The loss function in the graph convolutional network training process uses MI maximization to fuse local representation into global representation, which can be obtained from local and global The angle captures more common local features.

[0086] The concatenated vectors of two different modalities of the same video are combined into a positive sample pair, and the concatenated vectors of two different modalities of different videos are combined into a negative sample pair. For positive samples, by maximizing , so that the discriminator can correctly identify positive samples: , where N represents the number of positive samples, Indicates Under the conditions, we seek expectations. Represents the positive sample input data and attachment information, It is The local and global representations of positive samples, are shared semantic features, such as the global context of the video, It is a discriminator that is used to determine whether the local representation and the global representation match. For negative samples, by maximizing , so that the discriminator can correctly identify negative samples: , where M represents the number of positive samples, Indicates Under the conditions, we seek expectations. Represents negative sample input data and attachment information, It is The local and global representations of the negative samples are obtained by weighting the losses of the positive and negative samples using a normalization factor to ensure that the scale of the loss function is consistent.

[0087] The final loss function during graph convolutional network training:

[0088] ,

[0089] in, Represents the final loss function during graph convolutional network training, which is optimized , the model is able to learn the correlation between local and global representations, thereby maximizing their mutual information.

[0090] S3. Use a multi-layer perceptron to project the extracted text vector, image vector, audio vector, and structure vector into a common semantic space. In the common semantic space, concatenate the structure vector with the text vector, image vector, and audio vector respectively.

[0091] The text vectors, image vectors, audio vectors, and structure vectors of learning videos exist in spaces of different dimensions. Directly mixing them will encounter compatibility issues. The feature vectors of different modalities are projected into a common feature space so that these feature vectors can be processed and understood in a unified manner.

[0092] Each modal vector of the learning video is spatially mapped using a hidden layer MLP to the same latent space:

[0093] ,

[0094] in, 、 、 、 Represent the text vector, image vector, audio vector, and structure vector projected into the common semantic space, respectively. 、 、 、 Represent the text vector, image vector, audio vector, and structure vector before being projected into the common semantic space, respectively. and They represent the weight matrix, represents the sigmoid activation function, and represent the bias terms respectively.

[0095] Perform vector concatenation operations on the structure vector with the text vector, image vector, and audio vector of the learning video to obtain:

[0096] ,

[0097] in 、 、 Represent the concatenated text vector, image vector, and audio vector respectively.

[0098] S4. Perform modality comparison and modality matching on the concatenated text vector, image vector, and audio vector. By optimizing the semantic alignment between the different modalities, a multimodal feature vector of the learning video is obtained. Here, the text modality corresponds to the concatenated text vector, the image modality corresponds to the concatenated image vector, and the audio modality corresponds to the concatenated audio vector.

[0099] The total loss function of modality comparison and modality matching is:

[0100] ,

[0101] in, represents the total loss of modal contrast and modal matching, represents the modality contrast loss, represents the modality matching loss.

[0102] ,

[0103] in, Expressing hope, They represent the hyperparameters for adjusting the loss weight, represents the cross entropy loss, Indicates the actual matching labels of text modality and image modality, Represents the similarity from text modality to image modality, Indicates the actual matching labels of text modality and audio modality, Represents the similarity from text modality to audio modality, represents the true matching labels of image modality and text modality, represents the similarity from image modality to text modality, represents the true matching labels of the image modality and audio modality, represents the similarity from the image modality to the audio modality, Indicates the actual matching labels of audio modality and text modality, Indicates the similarity from audio modality to text modality, represents the true matching labels of the audio modality and image modality, Represents the similarity from the audio modality to the image modality.

[0104] The similarity between two modalities is calculated using the following formula. Here, the similarity between the image modality and the text modality is taken as an example. Other similarities are also calculated using this formula:

[0105] ,

[0106] in, express and The similarity calculation function of represents a learnable hyperparameter, Indicates text mode, Indicates image modality.

[0107] The similarity calculation function is as follows:

[0108] ,

[0109] in, Represents the similarity between u and v, where u and v are any two modalities among text modality, image modality, and audio modality.

[0110] ,

[0111] in, is the true label, y=1 indicates a match, y=0 indicates a mismatch; is the matching probability predicted by the model.

[0112] S5. Fuse the multimodal feature vectors of the learning video, calculate the similarity between the user behavior vector and the fused multimodal feature vector of the learning video through cosine similarity, and rank and recommend all videos.

[0113] Concatenate the feature vectors of different modalities of the learning video into a high-dimensional vector: , user feature vector and video feature vector Matching is usually done using cosine similarity. All videos are sorted based on the calculated cosine similarity. The higher the similarity, the higher the ranking. The top N videos with the highest similarity are selected and recommended to the user.

[0114] In an exemplary embodiment, a computer-readable storage medium is included, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned learning video recommendation method based on modality semantic space alignment is implemented.

[0115] See also Figure 2 In an exemplary embodiment, an electronic device is also included, including at least one processor, at least one memory, and at least one communication bus.

[0116] A computer program is stored in the memory, and the computer program includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through a communication bus to execute the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0117] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video recommendation method based on learning from modal semantic space alignment, characterized in that: The following steps are involved: S1. Obtain user behavior information and learn the text, image, and audio information of the video; S2. Extract user behavior vectors, extract text vectors, image vectors, and audio vectors from the learning videos; use a multi-layer graph neural network to extract features from the structural information between the learning videos, generate structural vectors, and obtain vectors of different modalities of the learning videos; S3. Project the extracted text vector, image vector, audio vector, and structure vector into a common semantic space using a multi-layer perceptron. In the common semantic space, concatenate the structure vector with the text vector, image vector, and audio vector respectively. S4. Perform modal comparison and modal matching on the concatenated text vector, image vector, and audio vector, and obtain a multimodal feature vector of the learning video by optimizing the semantic alignment between different modalities. S5. Fuse the multimodal feature vectors of the learning video, calculate the similarity between the user behavior vector and the fused multimodal feature vector of the learning video through cosine similarity, and rank and recommend all videos.

2. The method for learning video recommendation based on modal semantic space alignment according to claim 1, characterized in that: Use RNN model to extract user behavior vector , using the BERT model to extract text vectors from learning videos , use ResNet model to extract image vector , using MFCC feature extraction method to extract audio vector .

3. The method for learning video recommendation based on modal semantic space alignment according to claim 1, characterized in that: Structural vectors based on graphs Construction,node V represents learning videos, edge E represents the relationship between videos, and edge types include: video knowledge point overlap, user behavior association of watching videos, and video content similarity; Define edge weights based on different types of edges: ， in, express and The degree of overlap of knowledge points, and Represent the knowledge point sets of the i-th video and the j-th video respectively, express and The number of knowledge points in the intersection, express and The number of knowledge points in the set; represents the association weight of user behaviors between watching the i-th video and the j-th video, and measures the similarity of user behaviors between the i-th video and the j-th video; ， in, represents the content similarity weight between the i-th video and the j-th video, and are the text feature vectors of the i-th video and the j-th video respectively; Construct the adjacency matrix: ， in, represents the edge weight between the i-th video and the j-th video, is the weight parameter of the relationship type.

4. The method for learning video recommendation based on modal semantic space alignment according to claim 3, characterized in that: A multi-layer graph neural network is used to extract features from the structural information between learning videos and generate a structure vector, which is expressed as: ， ， in, represents the structure vector, represents the feature representation of the i-th node in the L-th layer, where L represents the number of graph convolution layers. represents the feature representation of the i-th node in the l+1 layer, r represents the relationship, represents the neighbor nodes of the i-th node, represents the sigmoid activation function, Representing relationships The normalized attention weights of the next i-th node and the j-th node, Representing relationships The adjacency matrix of the i-th node and the j-th node, represents the feature representation of the jth neighbor node of the i-th node in the l-th layer, Representing relationships The weight matrix at layer l; ， in, Representing relationships The attention of the next i-th node and j-th node, Representing relationships The attention of the i-th node and the k-th node, the k-th node is the neighbor node of the i-th node, Represents the neighbor node of the i-th node; ， Among them, MLP represents multi-layer perceptron, represents the feature representation of the i-th node in the l-th layer, Indicates the type of relationship between the i-th node and the j-th node.

5. The method for learning video recommendation based on modal semantic space alignment according to claim 1, characterized in that: The extracted text vector, image vector, audio vector and structure vector are projected into a common semantic space using a multi-layer perceptron, which is expressed as: ， in, 、 、 、 Represent the text vector, image vector, audio vector, and structure vector projected into the common semantic space, respectively. 、 、 、 Represent the text vector, image vector, audio vector, and structure vector before being projected into the common semantic space, respectively. and They represent the weight matrix, represents the sigmoid activation function, and represent the bias terms respectively.

6. The method for learning video recommendation based on modal semantic space alignment according to claim 1, characterized in that: The concatenated text vector, image vector, and audio vector are subjected to modal comparison and modal matching. The loss functions of modal comparison and modal matching are: ， in, represents the loss of modal contrast and modal matching, represents the modality contrast loss, represents the modality matching loss; ， in, Expressing hope, They represent the hyperparameters for adjusting the loss weight, represents the cross entropy loss, Indicates the actual matching labels of text modality and image modality, Represents the similarity from text modality to image modality, Indicates the actual matching labels of text modality and audio modality, Represents the similarity from text modality to audio modality, represents the true matching labels of image modality and text modality, represents the similarity from image modality to text modality, represents the true matching labels of the image modality and audio modality, represents the similarity from the image modality to the audio modality, Indicates the actual matching labels of audio modality and text modality, Indicates the similarity from audio modality to text modality, represents the true matching labels of the audio modality and image modality, Represents the similarity from audio modality to image modality; ， in, is the true label, y=1 indicates a match, y=0 indicates a mismatch; is the matching probability predicted by the model.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the processor and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the method according to any one of claims 1 to 6.

Images