Learning video recommendation method based on modal semantic space alignment, medium and equipment

Abstract

The invention provides a learning video recommendation method based on modal semantic space alignment, a medium and equipment, and relates to the technical field of multi-modal semantic space alignment, and the method comprises the steps: extracting a user behavior vector and vectors of different modes of a learning video, including text, image, audio and structure vectors; projecting the extracted vectors of different modes to a public semantic space by using a multi-layer perceptron, and in the public semantic space, performing vector splicing on the structure vector with a text vector, an image vector and an audio vector; performing modal comparison and modal matching on the spliced vectors, and obtaining a multi-modal feature vector of the learning video by optimizing semantic alignment between different modals; and fusing the multi-modal feature vectors, calculating the similarity between the user behavior vector and the fused multi-modal feature vector through cosine similarity, and carrying out sorting recommendation on all videos. According to the method, the modal semantic structure of the learning video can be fully understood, and accurate matching and recommendation are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-modal semantic space alignment, and particularly to a learning video recommendation method, medium, and device based on modal semantic space alignment. Background Art

[0002] With the rapid development of online education and the increasing richness of learning resources, in the field of online education, learning videos, as an important teaching resource, usually have multi-modal characteristics. For example, the subtitles, titles, descriptions, cover images, key frames of the videos, as well as the speech information in the video audio, all contain rich teaching semantic information. Combining these modal information, the recommendation system can recommend personalized learning videos according to the learning interests and needs of users, thereby improving learning efficiency. For example, the learning behavior data of users (such as viewing history, click records, likes and comments) can reflect their learning goals, while the multi-modal features of learning videos can comprehensively present their content themes and knowledge points. By integrating user behavior and video multi-modal data, the system can provide accurate learning resource recommendations, optimize the learning path, and improve the user experience and learning effect.

[0003] Multi-modal recommendation systems have shown significant superiority in recommendation tasks by integrating data of multiple modalities (such as text, images, audio, video, user behavior, etc.). Compared with traditional single-modal recommendation systems, multi-modal recommendation can understand user interests and recommended content more comprehensively and provide more accurate and personalized services.

[0004] Traditional multi-modal information fusion methods usually simply splice or combine features of different modalities, failing to effectively capture the semantic relationships between modalities, which may lead to the loss or redundancy of modal information of learning videos. The above deficiencies make it difficult for existing recommendation systems to accurately model user preferences and fully understand the modal semantic structure of learning videos, thus affecting the accuracy and performance of recommendation results. Summary of the Invention

[0005] The purpose of the present invention is to: in order to solve the problem that existing multi-modal information fusion methods cannot effectively capture the semantic relationships between modalities, a learning video recommendation method based on modal semantic space alignment is proposed, including the following steps: S1. Obtain the behavior information of the user, the text information, image information, and audio information of the learning video; S2. Extract the user behavior vector, extract the text vector, image vector, and audio vector of the learning video; use a multi-layer graph neural network to extract features from the structural information between learning videos to generate a structural vector, and obtain vectors of different modalities of the learning video; S3. Project the extracted text vectors, image vectors, audio vectors, and structure vectors into a common semantic space using a multi-layer perceptron. In the common semantic space, concatenate the structure vector with the text vector, image vector, and audio vector respectively; S4. Perform modality contrast and modality matching on the concatenated text vectors, image vectors, and audio vectors, and obtain the multi-modal feature vectors of the learning video by optimizing the semantic alignment between different modalities; S5. Fuse the multi-modal feature vectors of the learning video, calculate the similarity between the user behavior vector and the fused multi-modal feature vectors of the learning video through cosine similarity, and sort and recommend all videos.

[0006] Further, an RNN model is used to extract the user behavior vector , a BERT model is used to extract the text vector of the learning video , a ResNet model is used to extract the image vector , and an MFCC (Mel-frequency cepstral coefficients) feature extraction method is used to extract the audio vector .

[0007] Further, the structure vector is constructed based on a graph . The node V represents the learning video, and the edge E represents the relationship between videos. The types of edges include: video knowledge point overlap, user behavior association of watching videos, and video content similarity; Based on different types of edges, define the edge weights: , where represents the degree of knowledge point overlap between and , and represent the knowledge point sets of the i-th video and the j-th video respectively, represents the number of knowledge points in the intersection of and , represents the number of knowledge points in the union of and ; represents the user behavior association weight of watching the i-th video and the j-th video, measuring the similarity of the i-th video and the j-th video in user behavior; , where represents the content similarity weight of the i-th video and the j-th video, and They are the text feature vectors of the i-th video and the j-th video respectively; Construct an adjacency matrix: , where, represents the edge weight between the i-th video and the j-th video, is the weight parameter of the relationship type.

[0008] Furthermore, use a multi-layer graph neural network to extract features from the learned structural information between videos and generate a structure vector, denoted as: , , where, represents the structure vector, represents the feature representation of the i-th node at the L-th layer, L represents the number of layers of graph convolution, represents the feature representation of the i-th node at the l+1-th layer, r represents the relationship, represents the neighbor nodes of the i-th node, represents the sigmoid activation function, represents the relationship the normalized attention weight between the i-th node and the j-th node under the relationship, represents the relationship the adjacency matrix between the i-th node and the j-th node under the relationship, represents the feature representation of the j-th neighbor node of the i-th node at the l-th layer, represents the relationship the weight matrix at the l-th layer under the relationship; , where, represents the attention between the i-th node and the j-th node under the relationship, represents the attention between the i-th node and the k-th node under the relationship, the k-th node is the neighbor node of the i-th node, represents the neighbor nodes of the i-th node; , where MLP represents a multi-layer perceptron, represents the feature representation of the i-th node at the l-th layer,

[0009] represents the relationship type between the i-th node and the j-th node.

[0009] Furthermore, use a multi-layer perceptron to project the extracted text vector, image vector, audio vector and structure vector into a common semantic space, denoted as: , Among them, , , , respectively represent the text vector, image vector, audio vector, and structure vector projected onto the common semantic space. , , , respectively represent the text vector, image vector, audio vector, and structure vector before being projected onto the common semantic space. and respectively represent the weight matrices. represents the sigmoid activation function. and respectively represent the bias terms.

[0010] Furthermore, the concatenated text vector, image vector, and audio vector are subjected to modality contrast and modality matching. The loss function for modality contrast and modality matching is: , where represents the loss of modality contrast and modality matching. represents the modality contrast loss. represents the modality matching loss; , where represents taking the expectation. respectively represent the hyperparameters for adjusting the loss weights. represents the cross-entropy loss. represents the true matching label between the text modality and the image modality. represents the similarity from the text modality to the image modality. represents the true matching label between the text modality and the audio modality. represents the similarity from the text modality to the audio modality. represents the true matching label between the image modality and the text modality. represents the similarity from the image modality to the text modality. represents the true matching label between the image modality and the audio modality. represents the similarity from the image modality to the audio modality. represents the true matching label between the audio modality and the text modality. represents the similarity from the audio modality to the text modality. represents the true matching label between the audio modality and the image modality. represents the similarity from the audio modality to the image modality; , Among them, is the true label, where y = 1 indicates a match and y = 0 indicates no match; is the matching probability predicted by the model.

[0011] The present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0012] The present invention also provides an electronic device, including a processor and a memory, the processor being interconnected with the memory, wherein the memory is used to store a computer program, the computer program includes computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0013] The beneficial effects brought by the technical solution provided by the present invention are as follows: The present invention proposes a learning video recommendation method based on modal semantic space alignment. By mapping multi-modal features such as text, images, audio, and structures of learning videos to a unified semantic space and combining with the learning behavior characteristics of users, high-quality user interest vectors and video feature vectors are constructed to fully understand the modal semantic structure of learning videos and achieve accurate matching and recommendation. The present invention solves the semantic alignment problem between multi-modal features through modal contrast learning and modal matching optimization; through the dynamic weighted fusion technology, the problem of modal feature loss in traditional methods is avoided, and the performance of the recommendation system and the user experience are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flowchart of the learning video recommendation method based on modal semantic space alignment according to an embodiment of the present invention; Figure 2 is a block diagram of an electronic device in an exemplary embodiment of Embodiment 1 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings.

[0016] The flowchart of the learning video recommendation method based on modal semantic space alignment according to an embodiment of the present invention is as shown in Figure 1 , and specifically includes the following steps: S1. Obtain the user's behavior information and learn the text information, image information, and audio information of the learning video. The user's behavior information, such as viewing records and click data, can reflect their interest in the learning video. The learning video contains multimodal information, and each source or form of information can be called a modality. In this invention, the multimodality of the learning video includes text, image, and audio, etc.

[0017] S2. Extract the user behavior vector, extract the text vector, image vector, and audio vector of the learning video; use a multi-layer graph neural network to extract the structural information between the learning videos to generate a structural vector, and obtain the vectors of different modalities of the learning video.

[0018] Specifically, use the RNN model to extract the user behavior vector , which is expressed by the formula: , , , Among them, represents the hidden state at time step t, represents the weight matrix input to the hidden layer, represents the input at time step t, represents the weight matrix from the previous hidden state to the current hidden state, represents the hidden state at time step t - 1, is the bias term, is the activation function, represents the hidden state at the last time step, represents the input at the last time step, represents the hidden state at time step T - 1, represents the user behavior vector, where the input is the user's behavior information.

[0019] Use the BERT model to extract the text vector of the learning video , the text information , where represents the length of the text after preprocessing. First, use a shallow convolutional network for window operation, and the process is as follows: , Among them, represents the text information after window operation, represents the non-linear function, represents the fully connected operation, represents the feature transformation function, represents the concatenation operation, represents the convolutional kernel, and W represents the mapping matrix. Indicates a convolution operation.

[0020] The obtained is fed into the BERT model, and text feature vectors are generated through multiple layers of Transformer encoders .

[0021] The ResNet model is used to extract image vectors , and the specific formula is as follows: , where represents the input of the (l + 1)-th layer, represents the activation function, represents the convolution kernel of the l-th layer, represents the input of the l-th layer, represents the bias of the l-th layer, represents the residual connection, which is used to capture multi-level features, and then global average pooling (GAP) is used to obtain the image feature vector : , where represents the vector for extracting the i-th image information, and N represents the number of image information.

[0022] The MFCC (Mel-frequency cepstral coefficients) feature extraction method is used to extract audio vectors , first, the audio signal is pre-emphasized to enhance the high-frequency part, and then the Fourier transform is performed on each frame of the audio signal to obtain the spectrum: , where is the frame length, is the original audio signal, is the audio signal after the Fourier transform, that is, the spectrum of the audio signal.

[0023] The spectrum of the audio signal is passed through a Mel filter bank to obtain the Mel spectrum: , where is the frequency; then the discrete cosine transform (DCT) is performed on the Mel spectrum to obtain the MFCC coefficients: , represents the energy output by the k-th Mel filter, K represents the total number of Mel filter banks, n represents the frequency index, and the MFCC coefficients are subjected to mean pooling to obtain the audio feature vector: .

[0024] The structure vector is extracted through a graph neural network. First, a graph is constructed , node V represents learning videos, edge E represents the relationship between videos, and the weight of the edge reflects the strength of the relationship. The types of edges include: video knowledge point overlap, user behavior association of watched videos, and video content similarity.

[0025] Based on different types of edges, define the edge weights. For the knowledge point overlap relationship, the edge weight is calculated as follows: , Among them, represents and the degree of knowledge point overlap between and respectively represent the knowledge point sets of the i-th video and the j-th video, represents and the number of knowledge points in the intersection of represents and the number of knowledge points in the union of

[0026] For user behavior association, use to represent the user behavior association weight of watching the i-th video and the j-th video, measuring the similarity of the i-th video and the j-th video in terms of user behavior.

[0027] For the content similarity relationship, the edge weight is calculated as follows: , Among them, represents the content similarity weight between the i-th video and the j-th video, and are the text feature vectors of the i-th video and the j-th video respectively; by calculating the cosine similarity of these two vectors, the similarity of video content can be evaluated.

[0028] Construct the adjacency matrix by integrating the above relationships: , Among them, represents the edge weight between the i-th video and the j-th video, is the weight parameter of the relationship type.

[0029] In the learning video relationship graph, different types of relationships (knowledge points, user behavior, content similarity) may contribute differently to the node features; introduce a relationship attention mechanism to dynamically assign weights to each relationship for calculating the importance of the relationship.

[0030] First, assign an initial feature vector , this vector can be the original features of the node, such as the metadata of the video (length, resolution, etc.), or obtained through a certain feature extraction method.

[0031] Next, a multi-layer perceptron (MLP) is used to calculate the relational attention between nodes. For each pair of nodes, their relational attention is calculated as follows: , where MLP represents the multi-layer perceptron, represents the feature representation of the i-th node at the l-th layer, represents the feature representation of the j-th neighbor node of the i-th node at the l-th layer, represents the relationship type between the i-th node and the j-th node, and the semicolon represents the vector concatenation operation.

[0032] To convert the attention scores into weights and ensure that these weights can reasonably represent the relative importance between nodes, normalization is required, which is achieved through the softmax function: , where, represents the attention of the i-th node and the j-th node under the relationship , represents the attention of the i-th node and the k-th node under the relationship , and the k-th node is a neighbor node of the i-th node. represents the set of neighbor nodes of the i-th node. Normalization ensures that the sum of the attention weights of all neighbor nodes is 1, enabling it to pay more attention to important neighbor nodes.

[0033] Using the calculated attention weights, the features of neighbor nodes are weighted and summed to aggregate information: , where, represents the feature representation of the i-th node at the l + 1-th layer, r represents the relationship, represents the neighbor nodes of the i-th node, represents the sigmoid activation function, represents the normalized attention weight of the i-th node and the j-th node under the relationship represents the adjacency matrix of the i-th node and the j-th node under the relationship represents the feature representation of the j-th neighbor node of the i-th node at the l-th layer, represents the weight matrix at the l-th layer for the relationship

[0034] After layers of graph convolution and relational attention mechanism, the final structural embedding vector of each video node is generated: , where represents the structure vector, represents the feature representation of the i-th node at the L-th layer, and L represents the number of layers of graph convolution. is a comprehensive representation of the local features, neighborhood information, and global structure relationship of the learning video, including the original features of the video and also incorporating its complex relationships in the graph structure, enabling a deeper understanding of the video content.

[0035] In the training process of the graph convolutional network, the loss function adopts MI maximization to fuse the local representation into the global representation, and more common local features can be captured from the local and global perspectives.

[0036] The concatenated vectors of two different modalities of the same video are combined into a positive sample pair, and the concatenated vectors of two different modalities of different videos are combined into a negative sample pair. For positive samples, by maximizing , the discriminator can correctly identify positive samples: , where N represents the number of positive samples, represents taking the expectation under the condition of , represents the positive sample input data and attachment information, is the -th local and global representations of the positive sample, is the shared semantic feature, such as the global context of the video, is the discriminator, which is used to determine whether the local and global representations match. For negative samples, by maximizing , the discriminator can correctly identify negative samples: , where M represents the number of positive samples, represents taking the expectation under the condition of , represents the negative sample input data and attachment information, is the -th local and global representations of the negative sample. A normalization factor is used to weight-average the losses of positive and negative samples to ensure the consistency of the scale of the loss function.

[0037] The final loss function in the training process of the graph convolutional network: , where represents the final loss function in the training process of the graph convolutional network. By optimizing , the model can learn the correlation between the local and global representations, thereby maximizing their mutual information.

[0038] S3. Use a multi-layer perceptron to project the extracted text vectors, image vectors, audio vectors, and structure vectors into a common semantic space. In the common semantic space, concatenate the structure vector with the text vector, image vector, and audio vector respectively.

[0039] The text vectors, image vectors, audio vectors, and structure vectors of the learning video exist in spaces of different dimensions. Directly mixing them will encounter compatibility problems. Project the feature vectors of different modalities into a common feature space so that these feature vectors can be uniformly processed and understood.

[0040] Map each modality vector of the learning video to the same hidden space using a MLP with one hidden layer: , where, , , , represent the text vector, image vector, audio vector, and structure vector projected into the common semantic space respectively, , , , represent the text vector, image vector, audio vector, and structure vector before being projected into the common semantic space respectively, and represent the weight matrices respectively, represents the sigmoid activation function, and represent the bias terms respectively.

[0041] Perform vector concatenation operations on the structure vector with the text vector, image vector, and audio vector of the learning video respectively to obtain: , where , , represent the concatenated text vector, image vector, and audio vector respectively.

[0042] S4. Perform modality contrast and modality matching on the concatenated text vector, image vector, and audio vector. By optimizing the semantic alignment between different modalities, obtain the multi-modal feature vector of the learning video. Here, the text modality corresponds to the concatenated text vector, the image modality corresponds to the concatenated image vector, and the audio modality corresponds to the concatenated audio vector.

[0043] The total loss function of modality contrast and modality matching is: , where, Represents the total loss of modality contrast and modality matching, Represents the modality contrast loss, Represents the modality matching loss.

[0044] , Among them, Represents taking the expectation, Respectively represent the hyperparameters for adjusting the loss weights, Represents the cross-entropy loss, Represents the true matching label between the text modality and the image modality, Represents the similarity from the text modality to the image modality, Represents the true matching label between the text modality and the audio modality, Represents the similarity from the text modality to the audio modality, Represents the true matching label between the image modality and the text modality, Represents the similarity from the image modality to the text modality, Represents the true matching label between the image modality and the audio modality, Represents the similarity from the image modality to the audio modality, Represents the true matching label between the audio modality and the text modality, Represents the similarity from the audio modality to the text modality, Represents the true matching label between the audio modality and the image modality, Represents the similarity from the audio modality to the image modality.

[0045] Among them, the similarity between two modalities is calculated by the following formula. Taking the similarity from the image modality to the text modality as an example, the other similarities are also calculated according to this formula: , Among them, Represents and The similarity calculation function of, Represents a learnable hyperparameter, Represents the text modality, Represents the image modality.

[0046] The similarity calculation function is as follows: , Among them, Represents the similarity between u and v, where u and v are any two modalities among the text modality, the image modality, and the audio modality.

[0047] , Among them, It is the true label, where y = 1 indicates a match and y = 0 indicates no match; is the matching probability predicted by the model.

[0048] S5. Fuse the multi-modal feature vectors of the learning videos, calculate the similarity between the user behavior vector and the fused multi-modal feature vectors of the learning videos through cosine similarity, and sort and recommend all the videos.

[0049] Concatenate the feature vectors of different modalities of the learning videos into a high-dimensional vector: , and match the user feature vector with the video feature vector Calculate the similarity. Usually, cosine similarity is used to calculate the similarity. Sort all the videos according to the calculated cosine similarity. The higher the similarity, the higher the ranking. Select the top N videos with the highest similarity and recommend them to the user.

[0050] In an exemplary embodiment, it includes a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, it implements the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0051] Please refer to Figure 2 , in an exemplary embodiment, it further includes an electronic device, including at least one processor, at least one memory, and at least one communication bus.

[0052] Among them, a computer program is stored on the memory, and the computer program includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through the communication bus and executes the above-mentioned learning video recommendation method based on modal semantic space alignment.

[0053] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A learning video recommendation method based on modal semantic space alignment, characterized in that, It includes the following steps: S1. Obtain the user's behavior information, and learn the text information, image information, and audio information of the video; S2. Extract the user behavior vector, and extract the text vector, image vector, and audio vector of the learned video; Use a multi-layer graph neural network to extract features from the structural information between the learned videos, generate a structural vector, and obtain vectors of different modalities of the learned videos; S3. Use a multi-layer perceptron to project the extracted text vector, image vector, audio vector, and structural vector into a common semantic space. In the common semantic space, splice the structural vector with the text vector, image vector, and audio vector respectively; S4. Perform modality comparison and modality matching on the spliced text vector, image vector, and audio vector, and obtain the multi-modal feature vector of the learned video by optimizing the semantic alignment between different modalities; S5. Fuse the multi-modal feature vectors of the learned videos, calculate the similarity between the user behavior vector and the fused multi-modal feature vector of the learned video through cosine similarity, and sort and recommend all videos.

2. The learning video recommendation method based on modal semantic space alignment according to claim 1, characterized in that, Use the RNN model to extract the user behavior vector , use the BERT model to extract the text vector of the learning video , use the ResNet model to extract the image vector , use the MFCC feature extraction method to extract the audio vector .

3. A learning video recommendation method based on modal semantic space alignment according to claim 1, characterized in that Structure vector-based graph Constructed with nodes V representing learning videos and edges E representing relationships between videos. The types of edges include: video knowledge point overlap, user behavior association of watched videos, and video content similarity; Based on different types of edges, define edge weights: ， Among them, represents and the degree of knowledge point overlap, and respectively represent the knowledge point sets of the i-th video and the j-th video, represents and the number of knowledge points in the intersection, represents and the number of knowledge points in the union; Represents the user behavior correlation weight of watching the i-th video and the j-th video, measuring the similarity of the i-th video and the j-th video in terms of user behavior; ， Among them, represents the content similarity weight between the i-th video and the j-th video, and are the text feature vectors of the i-th video and the j-th video respectively; Construct an adjacency matrix: ， Among them, represents the edge weight between the i-th video and the j-th video, is the weight parameter of the relationship type.

4. The learning video recommendation method based on modal semantic space alignment according to claim 3, characterized in that Use a multi-layer graph neural network to extract features from the structural information between the learned videos, generate a structural vector, expressed as: ， ， Among them, represents the structure vector, represents the feature representation of the \(i\)-th node at the \(L\)-th layer, where \(L\) represents the number of layers of graph convolution, represents the feature representation of the \(i\)-th node at the \((l + 1)\)-th layer, and \(r\) represents the relationship, represents the neighbor nodes of the \(i\)-th node, represents the sigmoid activation function, represents the relationship the normalized attention weight between the \(i\)-th node and the \(j\)-th node under the relationship, represents the relationship the adjacency matrix between the \(i\)-th node and the \(j\)-th node under the relationship, represents the feature representation of the \(j\)-th neighbor node of the \(i\)-th node at the \(l\)-th layer, represents the relationship the weight matrix at the \(l\)-th layer; ， Among them, represents the relationship between the attention of the i-th node and the j-th node under represents the relationship between the attention of the i-th node and the k-th node under, where the k-th node is a neighbor node of the i-th node represents the neighbor node of the i-th node; ， Among them, MLP represents a multi-layer perceptron, represents the feature representation of the $i$-th node in the $l$-th layer, represents the relationship type between the $i$-th node and the $j$-th node.

5. A learning video recommendation method based on modal semantic space alignment according to claim 1, characterized in that Use a multi-layer perceptron to project the extracted text vector, image vector, audio vector, and structural vector into a common semantic space, expressed as: ， Among them, , , , respectively represent the text vector, image vector, audio vector, and structure vector projected onto the common semantic space, , , , respectively represent the text vector, image vector, audio vector, and structure vector before being projected onto the common semantic space, and respectively represent the weight matrices, represents the sigmoid activation function, and respectively represent the bias terms.

6. A learning video recommendation method based on modal semantic space alignment according to claim 1, characterized in that, Perform modality comparison and modality matching on the spliced text vector, image vector, and audio vector. The loss function for modality comparison and modality matching is: ， Among them, represents the loss of modality contrast and modality matching, represents the modality contrast loss, represents the modality matching loss; ， Among them, represents taking the expectation, respectively represent the hyperparameters for adjusting the loss weights, represents the cross-entropy loss, represents the true matching labels between the text modality and the image modality, represents the similarity from the text modality to the image modality, represents the true matching labels between the text modality and the audio modality, represents the similarity from the text modality to the audio modality, represents the true matching labels between the image modality and the text modality, represents the similarity from the image modality to the text modality, represents the true matching labels between the image modality and the audio modality, represents the similarity from the image modality to the audio modality, represents the true matching labels between the audio modality and the text modality, represents the similarity from the audio modality to the text modality, represents the true matching labels between the audio modality and the image modality, represents the similarity from the audio modality to the image modality; ， Among them, is the true label, where y = 1 indicates a match and y = 0 indicates no match; is the matching probability predicted by the model.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method according to any one of claims 1-6.

8. An electronic device, characterized in that, It includes a processor and a memory, the processor is connected to the memory. Among them, the memory is used to store a computer program, the computer program includes computer-readable instructions, and the processor is configured to call the computer-readable instructions to execute the method according to any one of claims 1-6.

Images