Multi-modal semantic alignment method oriented to classroom teaching guidance

Abstract

The invention belongs to the technical field of cross-modal semantic alignment, and discloses a multi-modal semantic alignment method oriented to classroom teaching guidance. Semantic modeling is carried out for multi-modal data in the teaching process, semantics are transmitted through an intermediary modal to align data semantics in a cross-modal mode, and the semantic alignment effect of other modal data is enhanced by utilizing the intermediary modal data which is large in scale and easier to process in the teaching process. According to the method, a data prototype semantic space is further constructed, data of different modalities can be described in the unified prototype semantic space, and multi-modal data can be mapped into the prototype semantic space in two modes of prototype projection transformation and self-supervision category label generation. According to the method, efficient alignment and deep analysis of multi-modal data can be realized in a complex classroom environment, so that a teacher is helped to acquire classroom dynamics in real time, a teaching strategy is optimized, and improvement of teaching quality and classroom efficiency is promoted.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal semantic alignment and relates to a multi-modal semantic alignment method for classroom teaching guidance. Background Art

[0002] With the rapid development of information technology, more and more digital and intelligent tools have been introduced into the modern educational environment, which not only enriches the form of classroom teaching, but also provides new possibilities for personalized teaching and efficient classroom guidance. However, traditional teaching guidance methods often rely on teachers' experience and intuition, lack scientific analysis and intelligent auxiliary tools, and because this method relies on manual observation and analysis, it often appears subjective and inefficient when faced with massive data.

[0003] In current classroom teaching, there are multiple modal data resources, so the classroom teaching process involves the simultaneous processing of multiple modal information (such as voice, pictures, text, etc.), which makes the effective integration of information and semantic understanding face huge challenges. In recent years, the application research of multimodal technology in the field of education has gradually increased, providing new ideas for solving the above problems.

[0004] The multimodal semantic alignment method can extract and align relevant semantics from multi-source information, thereby helping teachers better understand and guide classroom teaching. This method can provide real-time feedback and decision support by comprehensively analyzing multimodal data such as language, vision, and behavior, thereby improving teaching effectiveness and student learning experience. However, since multimodal data is usually affected by complex scenarios such as its own noise, fusion mechanism, and training strategy, it will cause feature vectors to have problems such as feature loss and semantic offset, making it impossible to accurately represent the true intention of teaching behavior in classroom teaching scenarios. Summary of the invention

[0005] The purpose of this invention is to propose a multimodal semantic alignment method for classroom teaching guidance, so as to effectively integrate and utilize various modal data such as audio, pictures and text in the teaching process, jointly mine their semantics, and realize cross-modal semantic alignment of multimodal data, thereby promoting data representation learning and enhancing classroom teaching effectiveness.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] A multimodal semantic alignment method for classroom teaching guidance includes the following steps:

[0008] Step 1. Organize the multimodal teaching resources in the teaching process to form a multimodal data set;

[0009] Among them, multimodal datasets include pictures, audio, and text data;

[0010] Use text data as an intermediate modality to transmit semantics and align data semantics across modalities, thereby obtaining pre-aligned semantic representation vectors of multimodal data, and parameterize these semantic representation vectors;

[0011] Step 2. Construct a prototype semantic space for multimodal data, map the parameterized semantic representation vector of multimodal data into the prototype semantic space through prototype projection transformation and self-supervised category label generation, and design a semantic mapping loss function;

[0012] Step 3. Establish a cross-modal semantic alignment model based on contrastive learning and design a cross-modal semantic representation embedding loss function;

[0013] Step 4. Based on the loss functions constructed in steps 2 and 3, construct an overall loss function for training the cross-modal semantic alignment model, and optimize the model parameters of the cross-modal semantic alignment model by minimizing the overall loss function;

[0014] Step 5. Calculate the similarity of the semantic representation vectors of each modal data based on the cross-modal semantic alignment model, so that the image modality and audio modality data with the same semantics are paired, thereby achieving semantic alignment of multimodal data.

[0015] The present invention has the following advantages:

[0016] The present invention discloses a multimodal semantic alignment method for classroom teaching guidance. The method obtains a pre-aligned embedding representation vector method of modal data by using preprocessing technology; constructs a prototype semantic embedding representation strategy for multimodal data; designs a cross-modal semantic representation embedding loss function based on contrastive learning, and performs model optimization, and finally realizes cross-modal semantic alignment of multimodal data, thereby generating automated multimodal teaching resources and effectively guiding classroom teaching. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a processing flow chart of a multimodal semantic alignment method for classroom teaching guidance in an embodiment of the present invention;

[0018] Figure 2 A model structure diagram of a multimodal semantic alignment method for classroom teaching guidance in an embodiment of the present invention;

[0019] Figure 3 This is a diagram showing the effect of the multimodal semantic alignment method for classroom teaching guidance in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0021] Example 1

[0022] This embodiment 1 proposes a multimodal semantic alignment method for classroom teaching guidance. The method aims to adapt to modern intelligent teaching scenarios and design a method that can automatically associate different modal information to more effectively integrate and utilize various modal data in the teaching process, realize real-time dynamic multimodal teaching in the classroom, and enhance the effectiveness of classroom teaching.

[0023] The cross-modal data semantic alignment problem solved by the present invention is:

[0024] Assuming that multimodal data are known, these data exist in multiple modal types and can be divided into modalities a, b, and c. Modalities a and b, and modalities b and c are easy to establish semantic associations. How to achieve multimodal semantic alignment?

[0025] The present invention can semantically align data modalities a and c using data modality b as an intermediary. In this embodiment, data modality b is, for example, text data, data modality a is, for example, picture data, and data modality c is, for example, audio data.

[0026] The present invention generally includes five parts: semantic pre-alignment of cross-modal data, construction of prototype semantic space of multimodal data, design of cross-modal semantic representation embedding loss function, model training and semantic alignment of multimodal data in the teaching process.

[0027] like Figure 1 and 2 As shown, the multimodal semantic alignment method for classroom teaching guidance in this embodiment includes the following steps:

[0028] Step 1. Organize the multimodal teaching resources in the teaching process to form a multimodal data set. For the convenience of description, the more common text modal data, image modal data, and audio modal data in the teaching process are used as examples for description.

[0029] The multimodal data is processed using preprocessing techniques to obtain pre-aligned embedding representation vectors of the multimodal data.

[0030] The present invention utilizes the association between intermediate modal data and other modal data and takes the semantics of modal data that is larger in scale and easier to parse as an anchor point, thereby achieving semantic cognitive alignment of cross-modal data.

[0031] This method helps to construct semantic representations of various modal data and provide a more feasible solution for improving the effectiveness of multimodal teaching, thereby better meeting the needs of smart teaching in the new era.

[0032] First, the multimodal teaching resources in the teaching process are sorted out to form a multimodal data set. The image and audio data are embedded and represented using a pre-trained multimodal data representation model to obtain the semantic representation vector of each data.

[0033] Specifically, the pre-trained multimodal data representation models include contrastive language-image pre-training (CLIP) and contrastive language-audio pre-training (CLAP). For image data, the pre-training model CLIP is used to obtain the image semantic representation vector, while for audio data, the pre-training model CLAP is used to obtain the audio semantic representation vector.

[0034] Then, we select text modal data with a large amount of data that can be represented and processed by both CLIP and CLAP models as intermediate modal data, and reconstruct the vector representations of image modal data and audio modal data.

[0035] The process of reconstructing the vector representation of image modality data and audio modality data is as follows:

[0036] Where B and N represent the number of pictures and audio data respectively; are the semantic representation vectors of the 1st, 2nd, ..., Nth pictures respectively, Represent the semantic representation vectors of the 1st, 2nd, …, Mth audio respectively.

[0037] In order to facilitate the overall alignment of multimodal data in this invention, the text representation vectors obtained by the pre-trained models CLIP and CLAP are and Reconstruct the vector representation of image modality data and audio modality data to enhance the semantic relevance of image, audio modality and text modality in the data, and then obtain the representation embedding vector and The formula is as follows:

[0038]

[0039] Among them, sim() represents the similarity calculation function, which can be cosine similarity, τ 1 is the weight hyperparameter, Represents the semantic representation vector of the kth image, Represents the semantic representation vector of the k-th audio.

[0040] With the help of the pre-trained model, the semantic space of some multimodal data is pre-associated so that the i-th text data representation vector or Reconstruct a semantic representation vector of image modality data Or the semantic representation vector of audio modality data

[0041] In order to enhance the robustness of semantic space alignment of multimodal data and reduce numerical deviation and noise, the semantic representation vector of the reconstructed i-th image and i-th audio is and And the semantic representation vector of the i-th text data and Adding Gaussian perturbation, the formula is expressed as follows:

[0042]

[0043]

[0044] Among them, Normalize() is a vector standardization operation, and θ is a value sampled from a Gaussian distribution.

[0045] Next, the semantic representation vector of the i-th data of any modality is parameterized and transformed so that it can be better used as model input and participate in training as part of the model. The formal description is:

[0046]

[0047] in, is the text semantic representation vector, is the image semantic representation vector, is the audio semantic representation vector, f 1 (), f 2 () is a learnable transformation function, Respectively The result after parametric deformation.

[0048] In this embodiment, each transformation function can be set as a multi-layer perceptron.

[0049] The present invention performs semantic modeling for multimodal data in the teaching process, and can transmit semantics through intermediary modalities to align data semantics across modalities. In addition, the present invention proposes to use the larger-scale and easier-to-process intermediary modal data (text data in this embodiment) in the teaching process to enhance the semantic alignment effect of other modal data (audio and image data).

[0050] Step 2. Construct a prototype semantic space for multimodal data, map the parameterized semantic representation vector of the multimodal data into the prototype semantic space through prototype projection transformation and self-supervised category label generation, and design a semantic mapping loss function.

[0051] Step 2.1. Considering that multimodal teaching resources usually combine multiple media forms such as text, images, and audio to enhance students' understanding and interest, different resources will have similarities in the way of information presentation, such as using pictures and texts to explain complex concepts. Therefore, these data are similar in semantics and only differ in the form of presentation.

[0052] Therefore, the present invention designs a data prototype semantic space so that data of different modalities can be described in a unified prototype semantic space, and multimodal data are mapped into the prototype semantic space through two methods: prototype projection transformation and self-supervised category label generation.

[0053] The present invention describes the prototype semantic space as a set of numerical vectors z={z 1 ,z 2 ,…,z |z|} T The high-dimensional space formed, T represents the matrix transpose symbol, |Z| represents the total number of vectors in the set, z i The semantic distribution vector representing the prototype semantic space.

[0054] The semantic representation vector of the i-th text, picture or audio data Through the numerical transformation function, the specific mathematical form expressed in the prototype semantic space is described as:

[0055] Among them, C f () is a numerical transformation function based on modal data to prototype semantic space, which is used to transform the semantic representation vector set of input data Transformed to the prototype semantic space, is the collection size.

[0056] The numerical transformation function C of the prototype semantic space f () is represented by C, in order to make the transformation from data to prototype semantic space a linear change, so as to speed up the parameter training of the model designed by the present invention. Therefore, the prototype semantic representation vector of each data i is:

[0057]

[0058] in, Represents the semantic representation vector of the i-th data of any modality. The semantic distribution vector z of the prototype semantic space i Perform softmax normalization to get p i , to measure the semantic distribution of data in the prototype semantic space, the formula is as follows:

[0059]

[0060] Among them, zii Represents the prototype semantic distribution vector z i The jth element of ik Represents the prototype semantic space projection vector p i The kth element of 2 Represents the weight adjustment factor.

[0061] Step 2.2. Design a self-supervised pseudo-label generation mechanism, using the pseudo-label vector q i Represents the category of data i in the prototype semantic space to better identify the semantic category of multimodal data and facilitate the alignment of data in the prototype semantic space.

[0062] The present invention designs a self-supervised pseudo-label generation mechanism, using the pseudo-label vector q i To represent the category of data i in the prototype semantic space, the method can better identify the semantic categories of multimodal data and is conducive to the alignment of data in the prototype semantic space.

[0063] This scheme uses the Sinkhorn-Knopp algorithm to obtain pseudo labels for data. The Sinkhorn-Knopp algorithm regards the self-supervised classification of data in the prototype semantic space as an optimal transmission problem, which transforms the prototype semantic distribution vector z i The resulting matrix Z is considered the transmission cost matrix, and the transpose of the i-th row of the matrix Z corresponds to the vector z i .

[0064] In the present invention, the semantic distribution of data can be transmitted as a prototype semantic distribution. The algorithm is based on the matrix Z. The algorithm is iterated to form a transmission strategy matrix Q. The algorithm makes the matrix Q approximate a double random matrix. The matrix Q represents the category distribution of a multimodal data in the prototype semantic space. Finally, the data matrix Q obtained by the Sinkhorn-Knopp algorithm is used as a pseudo label by this algorithm. Among them, the transpose of the i-th row of the matrix Q corresponds to the pseudo label vector q i .

[0065] Taking into account the homology during data collection, the semantics of these data descriptions are the same, but the modal expressions are different. Therefore, these multimodal data have potential semantic consistency. To this end, the present invention designs a prototype semantic space so that the semantic representations of data in different modalities can be transformed into the same prototype semantic space. The present invention uses the prototype semantic space to obtain the projected representation of multimodal data in the prototype semantic space, so as to better align the semantics of data in different modalities.

[0066] Transformation function C based on modal data to prototype semantic space f () as a learnable parameter, and the calculated pseudo label vector q i Combined, construct the semantic mapping loss function L0 :

[0067]

[0068] in is the semantic pseudo label vector of the xt modality data The j-th element value of , that is, the value of the pseudo-label j corresponding to the ith data of any modality, xt is identified as the modality type such as text, picture or audio above.

[0069] L 0 It enables self-supervised cross-modal data semantic consistency mining based on the prototype semantic space of multimodal data.

[0070] Step 3. Establish a cross-modal semantic alignment model based on contrastive learning and design a cross-modal semantic representation embedding loss function.

[0071] Using contrastive learning to construct text representation vector pairing and audio-image pairing learning loss function L 1 and L 2 , the formula is as follows:

[0072]

[0073] Among them, τ 3 , τ 4 is a weight hyperparameter, B represents the total amount of data for model training at one time, Represents the parameterized representation vector of the jth audio modality data, Represents the parameterized representation vector of the jth data of the image modality, The parameterized representation vector representing the jth audio data after text semantic enhancement, The parameterized representation vector representing the jth piece of data in the image modality after text semantic enhancement.

[0074] Loss function L 1 It can ensure that the semantic representation vector of the parameterized i-th text data obtained based on the CLIP and CLAP pre-training models Mutual approximation; loss function L 2 It can ensure that the image modality semantic representation vector after parameterization of the i-th text data obtained based on the pre-trained model and audio modality semantic representation vector Approaching each other.

[0075] In order to further enhance the semantic alignment effect of the data, the regularization loss function L is added 3 :

[0076]

[0077] Step 4. Based on the loss functions constructed in steps 2 and 3, construct the overall loss function for training the cross-modal semantic alignment model, and use the gradient descent method to optimize the model parameters of the cross-modal semantic alignment model by minimizing the overall loss function.

[0078] Specifically, all the loss functions constructed in steps 2 and 3 above are combined together as the loss function L for training the cross-modal semantic alignment model based on contrastive learning. The calculation expression is as follows:

[0079] L=αL 0 +βL 1 +γL 2 +δL 3 ; Among them, α, β, γ, and δ are weight hyperparameters.

[0080] Step 5. Calculate the similarity of the semantic representation vectors of each modal data based on the cross-modal semantic alignment model, so that the image modality and audio modality data with the same semantics are paired, thereby achieving semantic alignment of multimodal data.

[0081] Based on step 4, the characterization vector of each modal data is transformed into 1 (), f 2 After the () function is transformed, the vector similarity is calculated, for example, using cosine similarity. In classroom teaching, this enables image modality data and audio modality data with the same semantics to be automatically paired, thereby helping teachers to advance the teaching process more efficiently. This can make the information presentation more comprehensive and three-dimensional, enhance students' understanding and interest, and guide them to absorb knowledge more deeply.

[0082] like Figure 3 As shown, during classroom teaching, when explaining "the sun shines on the incense burner and produces purple smoke, and you can see the waterfall hanging in the distance", the method of the present invention is used to match the "pictures" in the teaching resource library in real time, and the corresponding "audio" in the teaching resource library can also be played.

[0083] Of course, the multimodal data in this embodiment is not limited to audio data, image data, and text data, but can also be semantic alignment of the other three modal data; in addition, the amount of multimodal data is not limited to the three modal data shown in this embodiment, for example, it can also include four or more modal data, which will not be repeated here.

[0084] Several other common types of modal data include video modal data, geographic location modal data, sensor data, and other multimodal data suitable for specific subject content during teaching.

[0085] The method of the present invention uses artificial intelligence technology and multimodal information processing algorithms to fully integrate data from various modalities and jointly mine their semantics to achieve semantic alignment of multimodal data, thereby promoting data representation learning. Therefore, the method of the present invention can achieve efficient alignment and in-depth analysis of multimodal data in a complex classroom environment, thereby helping teachers to obtain classroom dynamics in real time, optimize teaching strategies, and promote improved teaching quality and classroom efficiency.

[0086] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A multimodal semantic alignment method for classroom teaching guidance, characterized in that: The steps include: Step 1. Organize the multimodal teaching resources in the teaching process to form a multimodal data set; Among them, multimodal datasets include pictures, audio, and text data; Use text data as an intermediate modality to transmit semantics and align data semantics across modalities, thereby obtaining pre-aligned semantic representation vectors of multimodal data, and parameterize these semantic representation vectors; Step 2. Construct a prototype semantic space for multimodal data, map the parameterized semantic representation vector of multimodal data into the prototype semantic space through prototype projection transformation and self-supervised category label generation, and design a semantic mapping loss function; Step 3. Establish a cross-modal semantic alignment model based on contrastive learning and design a cross-modal semantic representation embedding loss function; Step 4. Based on the loss functions constructed in steps 2 and 3, construct an overall loss function for training the cross-modal semantic alignment model, and optimize the model parameters of the cross-modal semantic alignment model by minimizing the overall loss function; Step 5. Calculate the similarity of the semantic representation vectors of each modal data based on the cross-modal semantic alignment model, so that the image modality and audio modality data with the same semantics are paired, thereby achieving semantic alignment of multimodal data.

2. The multimodal semantic alignment method for classroom teaching guidance according to claim 1 is characterized in that: The step 1 is specifically as follows: First, we organize the multimodal teaching resources in the teaching process to form a multimodal dataset. We use the pre-trained models CLIP and CLAP to embed the image and audio data into representations, respectively, to obtain the semantic representation vector of each modality data. Then, the text modal data that can be processed by the multimodal data representation pre-training model is used as the intermediate modal data to reconstruct the semantic representation vectors of the image modal data and the audio modal data; On this basis, each semantic representation vector is parameterized to serve as the input of the cross-modal semantic alignment model.

3. The multimodal semantic alignment method for classroom teaching guidance according to claim 2 is characterized in that: In step 1, the process of reconstructing the vector representation of the image modality data and the audio modality data is as follows: The semantic representation vectors of the i-th picture and the i-th audio obtained by CLIP and CLAP are represented as and The semantic representation vector of the i-th text data obtained by CLIP and CLAP is expressed as and The semantic vector representation of the image modal data and the audio modal data is reconstructed by integrating the semantic representation vector information of the image modal data and the audio modal data into the semantic representation vector of the i-th text data to obtain the semantic representation vector and The specific form is as follows: Where sim( ) is the similarity calculation function, τ1 is the weight hyperparameter, and N and M represent the number of images and audio data, respectively.

4. The multimodal semantic alignment method for classroom teaching guidance according to claim 3 is characterized in that: In step 1, the semantic representation vector of the reconstructed i-th picture and i-th audio is and And the semantic representation vector of the i-th text data and Adding Gaussian perturbation, the formula is expressed as follows: Where Normalize() is a vector normalization operation, and θ is a value sampled from a Gaussian distribution.

5. The multimodal semantic alignment method for classroom teaching guidance according to claim 4 is characterized in that: In step 1, the semantic representation vector of the i-th data of any modality is parameterized and transformed so that it can better serve as the input of the cross-modal semantic alignment model and participate in the training as part of the model. The formal description is: Among them, f1( ) and f2( ) are learnable transformation functions; Respectively The parametrically deformed form.

6. The multimodal semantic alignment method for classroom teaching guidance according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.

1. Construct a prototype semantic space for multimodal data. Describe the prototype semantic space Z as a set of numerical vectors Z = {z1, z2, …, z n } T The high-dimensional space formed; Where T represents the matrix transpose symbol, z i The semantic distribution vector representing the prototype semantic space; Transform the semantic representation vectors of data of different modalities into the same prototype semantic space, that is, the representation vectors of text, image, and audio data are expressed in the prototype semantic space again through numerical transformation functions; Using the prototype semantic space, we can obtain the projection representation of multimodal data in the prototype semantic space. Step 2.

2. Design a self-supervised pseudo-label generation mechanism, using the pseudo-label vector q i Represents the category of the i-th data of any modality in the prototype semantic space to identify the semantic category of multimodal data and facilitate the alignment of data in the prototype semantic space.

7. The multimodal semantic alignment method for classroom teaching guidance according to claim 6 is characterized in that: In step 2.1, the semantic representation vector of the i-th text, picture or audio data is Through the numerical transformation function, the specific mathematical form expressed in the prototype semantic space is described as: Among them, C f ( ) is a numerical transformation function based on modal data to prototype semantic space, which is used to transform the semantic representation set of input data Transform to the prototype semantic space Z, is the collection size; The numerical transformation function C of the prototype semantic space f ( ) is represented by c, which is in the form of a matrix, making the transformation from data to prototype semantic space a linear change. Therefore, the prototype semantic distribution vector z of the ith data of any modality is i for: in, Represents the semantic representation vector of the i-th data of any modality; For the prototype semantic distribution vector z i Perform softmax normalization to get p i , to measure the semantic distribution of data in the prototype semantic space, the formula is as follows: Among them, z ij Represents the prototype semantic distribution vector z i The j-th element value, p ik Represents the prototype semantic space projection vector p i The kth element value of , τ2 represents the weight adjustment factor.

8. The multimodal semantic alignment method for classroom teaching guidance according to claim 6 is characterized in that: In step 2.2, the Sinkhom-Knopp algorithm is used to obtain pseudo labels for the data; The Sinkhom-Knopp algorithm transforms the prototype semantic distribution vector z i The matrix Z composed of is regarded as the transmission cost matrix, and the transpose of the i-th row of the matrix Z corresponds to the vector z i ; The algorithm is based on the matrix Z, and through algorithm iteration, the transmission strategy matrix Q is formed; The Sinkhorn-Knopp algorithm makes the matrix Q approximate a doubly random matrix through iteration; Among them, the matrix Q represents the category distribution of a multimodal data in the prototype semantic space; the matrix Q finally obtained by the Sinkhorn-Knopp algorithm is used as a pseudo label, and the transpose of the i-th row of the matrix Q corresponds to the pseudo label vector q i .

9. The multimodal semantic alignment method for classroom teaching guidance according to claim 6, characterized in that: In step 2, the transformation function C based on multimodal data to the prototype semantic space is f ( ) is used as a learnable parameter and is compared with the calculated pseudo label vector q i Combined, the semantic mapping loss function L0 is constructed: in is the semantic pseudo label vector of the xt modality data The j-th element value of , that is, the value of the pseudo-label j corresponding to the ith data of any modality, xt is identified as text, picture or audio modality type.

10. The multimodal semantic alignment method for classroom teaching guidance according to claim 9, characterized in that: In step 3, the process of designing the cross-modal semantic representation embedding loss function is as follows: By using contrastive learning, a cross-modal semantic alignment model is established. The model makes the semantic representation vectors of data with similar semantics close to each other, and makes the semantic representation vectors of data with low semantic similarity far away from each other. Then, we construct the text representation vector pairing and audio-image pairing learning loss functions L1 and L2; The calculation expressions of L1 and L2 of the loss function are as follows: Among them, τ3, τ4 are weight hyperparameters, B represents the total amount of data for model training at one time, Represents the parameterized representation vector of the jth audio modality data, Represents the parameterized representation vector of the jth data of the image modality, The parameterized representation vector representing the jth audio data after text semantic enhancement, Represents the parameterized representation vector of the jth data in the image modality after text semantic enhancement; the loss function L1 ensures that the parameterized semantic representation vector of the i-th text data obtained based on the CLIP and CLAP pre-training models The loss function L2 ensures that the parameterized semantic representation vector of the image modality obtained based on the i-th text data obtained by the pre-training model and audio modality semantic representation vector approaching each other; In order to further enhance the semantic alignment effect of data, the regularization loss function L3 is added. The calculation expression of L3 is as follows: In step 4, all the loss functions designed in steps 2 and 3 above are combined together as the overall loss function L for cross-modal semantic alignment model training. The calculation expression is as follows: L=αL0+βL1+γL2+δL3; where α, β, γ, and δ are weight hyperparameters.

Images