A multi-modal semantic alignment method for classroom teaching guidance

Abstract

This invention belongs to the field of cross-modal semantic alignment technology and discloses a multimodal semantic alignment method for classroom teaching guidance. This invention performs semantic modeling on multimodal data in the teaching process, transmits semantics through intermediate modalities, and aligns data semantics across modalities. This invention leverages the large-scale and easily processed intermediate modal data in the teaching process to enhance the semantic alignment effect of other modalities. This invention also constructs a data prototype semantic space, enabling data of different modalities to be described in a unified prototype semantic space. Multimodal data can be mapped into the prototype semantic space through two methods: prototype projection transformation and self-supervised category label generation. This invention's method can achieve efficient alignment and in-depth analysis of multimodal data in complex classroom environments, thereby helping teachers obtain real-time classroom dynamics, optimize teaching strategies, and promote improved teaching quality and classroom efficiency.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal semantic alignment and relates to a multimodal semantic alignment method for guiding classroom teaching. Background Technology

[0002] With the rapid development of information technology, more and more digital and intelligent tools have been introduced into the modern education environment, which not only enriches the forms of classroom teaching but also provides new possibilities for personalized teaching and efficient classroom guidance. However, traditional teaching guidance methods often rely on teachers' experience and intuition, lacking scientific analysis and intelligent auxiliary tools. Because this method depends on manual observation and analysis, it often appears subjective and inefficient when faced with massive amounts of data.

[0003] In current classroom teaching, multiple modal data resources exist, thus requiring the simultaneous processing of various modal information (such as speech, images, and text). This poses a significant challenge to the effective integration of information and semantic understanding. In recent years, research on the application of multimodal technology in education has gradually increased, providing new ideas for solving these problems.

[0004] Multimodal semantic alignment methods can extract and align relevant semantics from multi-source information, thereby helping teachers better understand and guide classroom teaching. This approach provides real-time feedback and decision support by comprehensively analyzing multimodal data, including language, vision, and behavior data, thus improving teaching effectiveness and student learning experience. However, because multimodal data is often affected by inherent noise, fusion mechanisms, and complex training strategies, it can lead to problems such as feature loss and semantic shift in feature vectors, making it impossible to accurately represent the true intent of teaching behaviors in a classroom setting. Summary of the Invention

[0005] The purpose of this invention is to propose a multimodal semantic alignment method for classroom teaching guidance, so as to effectively integrate and utilize various modal data such as audio, images and text in the teaching process, jointly mine their semantics, realize cross-modal semantic alignment of multimodal data, thereby promoting data representation learning and enhancing classroom teaching effectiveness.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A multimodal semantic alignment method for guiding classroom teaching includes the following steps:

[0008] Step 1. Organize multimodal teaching resources from the teaching process to form a multimodal dataset;

[0009] The multimodal dataset includes image, audio, and text data;

[0010] Text data is used as an intermediary modality to convey semantics, thereby aligning data semantics across modalities to obtain pre-aligned semantic representation vectors of multimodal data, and these semantic representation vectors are then parameterized.

[0011] Step 2. Construct the prototype semantic space of multimodal data. Map the parameterized semantic representation vectors of multimodal data into the prototype semantic space through two methods: prototype projection transformation and self-supervised category label generation. Design a semantic mapping loss function.

[0012] Step 3. Establish a cross-modal semantic alignment model based on contrastive learning, and design a cross-modal semantic representation embedding loss function;

[0013] Step 4. Based on the loss functions constructed in Steps 2 and 3, construct the overall loss function for training the cross-modal semantic alignment model, and optimize the model parameters of the cross-modal semantic alignment model by minimizing the overall loss function;

[0014] Step 5. Calculate the semantic representation vector similarity of each modality data according to the cross-modal semantic alignment model, so that image modality and audio modality data with the same semantics are paired, thereby realizing the semantic alignment of multimodal data.

[0015] The present invention has the following advantages:

[0016] This invention describes a multimodal semantic alignment method for guiding classroom teaching. The method obtains a pre-aligned embedding representation vector method for modal data by using preprocessing techniques; constructs a prototype semantic embedding representation strategy for multimodal data; designs a cross-modal semantic representation embedding loss function based on contrastive learning and optimizes the model, and finally realizes cross-modal semantic alignment of multimodal data, thereby generating automated multimodal teaching resources and effectively guiding classroom teaching. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the multimodal semantic alignment method for classroom teaching guidance in this embodiment of the invention.

[0018] Figure 2 This is a model structure diagram of the multimodal semantic alignment method for classroom teaching guidance in this embodiment of the invention;

[0019] Figure 3 This diagram illustrates the effect of the multimodal semantic alignment method for classroom teaching guidance in this embodiment of the invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0021] Example 1

[0022] This embodiment 1 proposes a multimodal semantic alignment method for classroom teaching guidance. This method aims to adapt to modern intelligent teaching scenarios and design a method that can automatically associate different modal information in order to more effectively integrate and utilize various modal data in the teaching process, realize real-time dynamic multimodal teaching in the classroom, and enhance the effectiveness of classroom teaching.

[0023] The cross-modal data semantic alignment problem solved by this invention is as follows:

[0024] Suppose we have known multimodal data, which contains multiple modal types and can be divided into modalities a, b, and c. Modalities a and b, and modalities b and c are easily associated semantically. How can we achieve semantic alignment among these multimodal data?

[0025] This invention enables semantic alignment of data modalities a and c using data modality b as an intermediary. In this embodiment, data modality b is, for example, text data, data modality a is, for example, image data, and data modality c is, for example, audio data.

[0026] This invention generally comprises five parts: semantic pre-alignment of cross-modal data, construction of prototype semantic space for multimodal data, design of cross-modal semantic representation embedding loss function, model training, and semantic alignment of multimodal data during the teaching process.

[0027] like Figure 1 and 2 As shown, the multimodal semantic alignment method for classroom teaching guidance in this embodiment includes the following steps:

[0028] Step 1. Organize multimodal teaching resources in the teaching process to form a multimodal dataset. For ease of description, we will use text modal data, image modal data, and audio modal data, which are more common in the teaching process, as examples.

[0029] Preprocessing techniques are used to process multimodal data to obtain pre-aligned embedding representation vectors for the multimodal data.

[0030] This invention achieves semantic cognitive alignment across modal data by leveraging the correlation between intermediate modal data and other modal data, using the semantics of larger and easier-to-parse modal data as anchors.

[0031] This method helps to construct semantic representations of various modal data, providing a more feasible solution for improving the effectiveness of multimodal teaching, thereby better meeting the needs of smart teaching in the new era.

[0032] First, we organize the multimodal teaching resources in the teaching process to form a multimodal dataset. Then, we use a pre-trained multimodal data representation model to embed and represent the image and audio data to obtain the semantic representation vector of each data.

[0033] Specifically, the pre-trained multimodal data representation models include Contrastive Language-Image Pre-Training (CLIP) and Contrastive Language-Audio Pre-Training (CLAP). For image data, the pre-trained model CLIP is used to obtain image semantic representation vectors, while for audio data, the pre-trained model CLAP is used to obtain audio semantic representation vectors.

[0034] Then, text modal data with a large volume that can be represented by both CLIP and CLAP models is selected as intermediate modal data to reconstruct the vector representations of image and audio modal data.

[0035] The process of reconstructing the vector representations of image modal data and audio modal data is as follows:

[0036] Where B and N represent the number of images and audio data, respectively; These are the semantic representation vectors of the 1st, 2nd, ..., Nth images, respectively. These represent the semantic representation vectors of the 1st, 2nd, ..., Mth audio clips, respectively.

[0037] To facilitate the overall alignment of multimodal data in this invention, text representation vectors obtained using the pre-trained models CLIP and CLAP are used. and The vector representations of image and audio modal data are reconstructed to enhance the semantic relevance between image, audio, and text modalities in the data, thereby obtaining representation embedding vectors. and The formula is as follows:

[0038]

[0039] Where sim() represents the similarity calculation function, which can be cosine similarity, and τ1 is the weight hyperparameter. This represents the semantic representation vector of the k-th image. This represents the semantic representation vector of the k-th audio track.

[0040] By using a pre-trained model, the semantic space of some multimodal data is pre-associated, making the representation vector of the i-th text data more accurate. or Each reconstructs an image modality data semantic representation vector. Or the semantic representation vector of audio modality data

[0041] To enhance the robustness of semantic space alignment for multimodal data and reduce numerical bias and noise, the semantic representation vectors of the reconstructed i-th image and i-th audio are... and and the semantic representation vector of the i-th text data and Adding a Gaussian perturbation, the formula is expressed as follows:

[0042]

[0043]

[0044] Normalize() is a vector normalization operation, and θ is a value sampled from a Gaussian distribution.

[0045] Next, the semantic representation vector of the i-th data point in any modality is parametrically transformed to better serve as model input and participate in training as part of the model. This can be formally described as follows:

[0046]

[0047] in, This is a text semantic representation vector. This is the semantic representation vector of the image. Let f1() and f2() be the audio semantic representation vectors, and f1() and f2() be learnable transformation functions. They represent The result after parametric deformation.

[0048] In this embodiment, each transformation function can be set as a multilayer perceptron.

[0049] This invention focuses on semantic modeling of multimodal data in the teaching process, enabling semantic alignment across modalities by transmitting semantics through intermediary modalities. Furthermore, this invention proposes leveraging the larger-scale and easier-to-process intermediary modal data (text data in this embodiment) from the teaching process to enhance the semantic alignment of other modal data (audio and image data).

[0050] Step 2. Construct the prototype semantic space of multimodal data. Map the parameterized semantic representation vectors of multimodal data into the prototype semantic space through two methods: prototype projection transformation and self-supervised category label generation. Design a semantic mapping loss function.

[0051] Step 2.1. Considering that multimodal teaching resources typically combine various media formats such as text, images, and audio to enhance students' understanding and interest, different resources may have similarities in their information presentation methods, such as using a combination of text and graphics to explain complex concepts. Therefore, these data are semantically similar, differing only in their presentation.

[0052] Therefore, this invention designs a data prototype semantic space, which allows data of different modalities to be described in a unified prototype semantic space. Multimodal data is mapped into the prototype semantic space through two methods: prototype projection transformation and self-supervised category label generation.

[0053] This invention describes the prototype semantic space as a set of numerical vectors z = {z1, z2, ..., z...} |z|} T The resulting high-dimensional space, T denotes the matrix transpose, |Z| denotes the total number of vectors in the set, z i The semantic distribution vector representing the prototype semantic space.

[0054] The semantic representation vector of the i-th text, image, or audio data The specific mathematical form, derived from the numerical transformation function and re-represented using the prototype semantic space, is described as follows:

[0055] Among them, C f () is a numerical transformation function based on modal data to prototype semantic space, used to transform the set of semantic representation vectors of input data. Transform to the prototype semantic space The size of the set.

[0056] The numerical transformation function C of the prototype semantic space f ( ) is denoted by C. To ensure that the transformation from data to the prototype semantic space is linear, thus accelerating the parameter training of the model designed in this invention, the prototype semantic representation vector for each data i is:

[0057]

[0058] in, Let z represent the semantic representation vector of the i-th data point in any modality. The semantic distribution vector z in the prototype semantic space. i The softmax normalized representation is used to obtain p i The formula is as follows: This measures the semantic distribution of data in the prototype semantic space.

[0059]

[0060] Among them, z ii Represents the prototype semantic distribution vector z iThe j-th element, p ik p represents the prototype semantic space projection vector i The k-th element, τ2, represents the weight adjustment factor.

[0061] Step 2.2. Design a self-supervised pseudo-label generation mechanism, utilizing the pseudo-label vector q i This indicates the category of data i in the prototype semantic space, which helps to better identify the semantic category of multimodal data and facilitates the alignment of data in the prototype semantic space.

[0062] This invention designs a self-supervised pseudo-label generation mechanism, utilizing the pseudo-label vector q i This is used to represent the category of data i in the prototype semantic space, enabling the method to better identify the semantic category of multimodal data, which is beneficial for the alignment of data in the prototype semantic space.

[0063] This design uses the Sinkhorn-Knopp algorithm to obtain pseudo-labels for the data. The Sinkhorn-Knopp algorithm treats self-supervised classification of data in the prototype semantic space as an optimal transfer problem, which uses the prototype semantic distribution vector z... i The matrix Z formed by the arrangement is considered as the transmission cost matrix, and the transpose of the i-th row of matrix Z corresponds to the vector z. i .

[0064] In this invention, the semantic distribution of data can be transferred to a prototype semantic distribution. The algorithm is based on matrix Z and iterates to form a transfer strategy matrix Q. The algorithm approximates matrix Q with a doubly random matrix, representing a category distribution of multimodal data in the prototype semantic space. Finally, the data matrix Q obtained using the Sinkhorn-Knopp algorithm is used as a pseudo-label in this algorithm. The transpose of the i-th row of matrix Q corresponds to the pseudo-label vector q. i .

[0065] Considering the homology of data acquisition, these data describe the same semantics, differing only in modal representation. Therefore, these multimodal data possess potential semantic consistency. To address this, this invention designs a prototype semantic space, enabling the semantic representations of data from different modalities to be transformed into the same prototype semantic space. This invention utilizes the prototype semantic space to obtain the projected representation of multimodal data within that space, allowing for better alignment of the semantics of data from different modalities.

[0066] The transformation function C from modal data to prototype semantic space f () is used as a learnable parameter, and is used in conjunction with the calculated pseudo-label vector q. i Combined, construct the semantic mapping loss function L0:

[0067]

[0068] in The semantic pseudo-label vector of xt modal data The j-th element value is the pseudo-label j value corresponding to the i-th data in any modality, and xt identifies the modality type such as text, image, or audio in the preceding text.

[0069] L0 enables self-supervised cross-modal semantic consistency mining based on the prototype semantic space.

[0070] Step 3. Establish a cross-modal semantic alignment model based on contrastive learning and design a cross-modal semantic representation embedding loss function.

[0071] Contrastive learning is used to construct text representation vector pairing and audio-image pairing learning loss functions L1 and L2, as follows:

[0072]

[0073] Where τ3 and τ4 are weight hyperparameters, and B represents the total amount of data used in one training iteration of the model. This represents the parameterized representation vector of the j-th data point in the audio modality. This represents the parameterized representation vector of the j-th data point in the image modality. This represents the parameterized representation vector of the j-th data point in the audio modality after text semantic enhancement. The parameterized representation vector of the j-th data point in the image modality after text semantic enhancement.

[0074] The L1 loss function can guarantee that the semantic representation vector of the i-th text data obtained by the CLIP and CLAP pre-trained models is parameterized. Mutual approximation; the L2 loss function can guarantee that the image modal semantic representation vector after parameterization of the i-th text data obtained based on the pre-trained model is close enough. and audio modal semantic representation vector They are getting closer to each other.

[0075] To further enhance the semantic alignment of the data, a regularization loss function L3 is added:

[0076]

[0077] Step 4. Based on the loss functions constructed in Steps 2 and 3, construct the overall loss function for training the cross-modal semantic alignment model, and use the gradient descent method to optimize the model parameters of the cross-modal semantic alignment model by minimizing the overall loss function.

[0078] Specifically, all the loss functions constructed in steps 2 and 3 above are combined together as the training loss function L for the cross-modal semantic alignment model based on contrastive learning. The calculation expression is as follows:

[0079] L = αL0 + βL1 + γL2 + δL3; where α, β, γ, and δ are weight hyperparameters.

[0080] Step 5. Calculate the semantic representation vector similarity of each modality data according to the cross-modal semantic alignment model, so that image modality and audio modality data with the same semantics are paired, thereby realizing the semantic alignment of multimodal data.

[0081] Building upon step 4, the representation vectors of each modality are transformed using the functions f1() and f2(), and vector similarity is calculated, for example, using cosine similarity. In classroom teaching, this allows image and audio modality data with the same semantics to be automatically paired, thus helping teachers advance the teaching process more efficiently. This results in a more comprehensive and three-dimensional presentation of information, enhancing students' understanding and interest, and guiding them to absorb knowledge more deeply.

[0082] like Figure 3 As shown, during classroom teaching, when explaining "The sun shines on the incense burner, producing purple smoke; from afar, the waterfall hangs before the river," the method of this invention was used to match the "images" in the teaching resource library in real time, and the corresponding "audio" in the teaching resource library could also be played.

[0083] Of course, the multimodal data in this embodiment is not limited to audio data, image data, and text data, but can also be semantically aligned with other three types of data; in addition, the amount of multimodal data is not limited to the three types of data shown in this embodiment, but can include four or more types of data, which will not be elaborated here.

[0084] Other common types of modal data include video modal data, geographic location modal data, and sensor data, which are multimodal data suitable for teaching specific subject content.

[0085] The method of this invention, through artificial intelligence technology and multimodal information processing algorithms, aims to fully integrate data from various modalities and jointly mine their semantics to achieve semantic alignment of multimodal data, thereby promoting data representation learning. Therefore, the method of this invention can achieve efficient alignment and in-depth analysis of multimodal data in complex classroom environments, thereby helping teachers to obtain real-time classroom dynamics, optimize teaching strategies, and promote the improvement of teaching quality and classroom efficiency.

[0086] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A multi-modal semantic alignment method for classroom teaching guidance, characterized in that, The method comprises the following steps: Step 1. Organize the multi-modal teaching resources in the teaching process to form a multi-modal data set; The multi-modal data set comprises picture, audio and text data; The text data is used as an intermediate modality to deliver semantics, so as to align the data semantics across modalities, thereby obtaining pre-aligned semantic representation vectors of the multi-modal data, and the semantic representation vectors are subjected to parameterization processing; Step 2. Construct a prototype semantic space of the multi-modal data, map the parameterized semantic representation vectors of the multi-modal data into the prototype semantic space through prototype projection transformation and self-supervised category label generation, and design a semantic mapping loss function; Step 3. Establish a cross-modal semantic alignment model based on contrast learning, and design a cross-modal semantic representation embedding loss function; Step 4. Based on the loss functions constructed in steps 2 and 3, construct a total loss function for training the cross-modal semantic alignment model, and optimize the model parameters of the cross-modal semantic alignment model by minimizing the total loss function; Step 5. According to the model of the cross-modal semantic alignment model, the similarity of the semantic representation vectors of each modality data is calculated, so that the picture modality and the audio modality data with the same semantics are paired, thereby realizing the semantic alignment of the multi-modal data; The step 2 is specifically: Step 2.

1. Constructing a prototype semantic space of multi-modal data, the prototype semantic space is described as a high-dimensional space formed by a set of numerical vectors is described as a high-dimensional space formed by a set of numerical vectors wherein, denotes the matrix transpose symbol, denotes a semantic distribution vector of a prototype semantic space; The semantic representation vectors of data of different modalities are all transformed into the same prototype semantic space, that is, the representation vectors of text, picture and audio data are represented by a numerical transformation function and a prototype semantic space; Using the prototype semantic space, the projection representation of the multi-modal data in the prototype semantic space is obtained; Step 2.

2. Designing a self-supervised pseudo-label generation mechanism to utilize pseudo-label vectors representing the first data in the prototype semantic space, to identify the semantic categories of the multi-modal data, facilitating the alignment of data in the prototype semantic space; In step 2, the transformation function from the multi-modal data to the prototype semantic space is based on As learnable parameters, the pseudo-label vector obtained by calculation In combination, the semantic mapping loss function is constructed : ； wherein is a semantic pseudo label vector of the modal data the first element value of the semantic pseudo label vector, i.e. the pseudo label corresponding to the first piece of data of the arbitrary modal value, identifies the modal type as text, picture or audio.

2. The multi-modal semantic alignment method for classroom teaching guidance according to claim 1, wherein The step 1 is specifically: Firstly, multi-modal teaching resources in the teaching process are organized to form a multi-modal data set; pre-trained models CLIP and CLAP are used to embed and represent picture and audio data respectively, so as to obtain semantic representation vectors of each modality data; Then, the text modality data which can be processed by the multi-modal data representation pre-training model is used as an intermediate modality data to reconstruct the semantic representation vectors of the picture modality data and the audio modality data; On this basis, the semantic representation vectors are subjected to parameterization processing to serve as inputs of the cross-modal semantic alignment model.

3. The multi-modal semantic alignment method for classroom teaching guidance according to claim 2, wherein In the step 1, the process of reconstructing the vector representation of the picture modality data and the audio modality data is as follows: The semantic representation vectors of the first ​​​​​​​ The picture modal data and audio modal data semantic vector representation are reconstructed, and the reconstruction mode is completed by integrating the picture modal data and audio modal data semantic representation vector information into the semantic representation vector of the first text data, to obtain the semantic representation vector and , and the specific form is as follows: ； ； wherein is a similarity computation function, is a weight hyperparameter, and Npand Narespectively represent the number of pictures and audio data.

4. The multi-modal semantic alignment method for classroom teaching guidance according to claim 3, wherein In step 1, the reconstructed first... The first picture and the first Semantic representation vector of audio segments and and the Semantic representation vector of text data and Adding a Gaussian perturbation, the formula is expressed as follows: ， ； ， ； wherein, is a standardization operation for vectors, is a number sampled from a Gaussian distribution.

5. The multi-modal semantic alignment method for classroom teaching guidance according to claim 4, wherein In step 1, the arbitrary mode number The semantic representation vectors of each data point are parametrically deformed to better serve as input to the cross-modal semantic alignment model and participate in training as part of the model. This can be formally described as follows: ， ， ， ； wherein , is a learnable transformation function; , , , respectively represent , , , the form after parameterized morphing.

6. The multi-modal semantic alignment method for classroom teaching guidance according to claim 1, wherein In step 2.1, the first semantic representation vector of the text, picture or audio data The specific mathematical form of the numerical transformation function and the re-expression in the original semantic space are described as follows: ； wherein, is a numerical transformation function to the prototype semantic space based on modal data, for transforming a semantic representation set of input data to the prototype semantic space , is the set size; Numerical transformation function of the prototype semantic space With denotes a matrix, so that the transformation of data to the prototype semantic space is a linear change, and thus the prototype semantic distribution vector of the i-th data of any modality is: ​ ； wherein, represents the first data semantic representation vector; prototype semantic distribution vector carried out normalized representation to measure the semantic distribution of data in the prototype semantic space, the formula is as follows: ； in, Represents the prototype semantic distribution vector The Each element value Represents the projection vector of the prototype semantic space The Each element value This represents the weighting adjustment factor.

7. The multi-modal semantic alignment method for classroom teaching guidance according to claim 1, wherein In the step 2.2, the Sinkhorn-Knopp algorithm is used to obtain the pseudo label of the data; The Sinkhorn-Knopp algorithm uses prototype semantic distribution vectors. The matrix formed Consider it as a transmission cost matrix, matrix The The transpose of a row corresponds to a vector The algorithm is based on matrices. Through algorithm iteration, a transmission strategy matrix is ​​formed. ; The Sinkhorn-Knopp algorithm approximates a doubly stochastic matrix by iteratively forcing a matrix wherein the matrix represents the class distribution of the multi-modal data in the prototype semantic space; finally, the matrix obtained by the Sinkhorn-Knopp algorithm is used as the pseudo label, and the transpose of the first row of the matrix corresponds to the pseudo label vector .

8. The multi-modal semantic alignment method for classroom teaching guidance according to claim 1, characterized in that, In step 3, the process of designing a cross-modal semantic representation embedding loss function is as follows: By using the idea of contrastive learning, a cross-modal semantic alignment model is established. The model makes the semantic representation vectors of data with similar semantics approach each other, while the semantic representation vectors of data with low semantic similarity move away from each other; Then, construct the text representation vector pair and the audio-picture pair learning loss function and ; The loss function is and The computational expression is as follows: ； ； in, , It is a weight hyperparameter. This represents the total amount of data used in a single training iteration of the model. Indicates the audio modality number 1 The parameterized representation vector of each data point Indicates the first image modality The parameterized representation vector of each data point This represents the audio modality after text semantic enhancement. The parameterized representation vector of each data point This represents the image modality after text semantic enhancement. The parameterized representation vector of each data point; loss function Guarantee the first [model] obtained based on CLIP and CLAP pre-trained models Semantic representation vectors after parameterization of text data , Mutual approximation; loss function Guarantee the first obtained based on the pre-trained model Image modal semantic representation vector after parameterization of text data and audio modal semantic representation vector They are getting closer to each other; To further enhance the semantic alignment effect of data, increase the regular term loss function , The calculation expression is as follows: ； In step 4, all the loss functions designed in steps 2 and 3 above are combined together as the overall loss function for training the cross-modal semantic alignment model The calculation expression is as follows: ; wherein, , , , are weight hyperparameters.

Images