Multi-modal model training method and device

Abstract

The invention discloses a multi-modal model training method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining a training sample set of a multi-modal model, wherein the training sample set comprises video data and text data; training the multi-modal model by using the training sample set, and adjusting and optimizing parameters of the multi-modal model by minimizing at least two of the first loss function, the second loss function and the third loss function to obtain the trained multi-modal model, the first loss function is constructed according to the global features of the text data and the global features of the video data; the second loss function is constructed according to the local features of the text data and the local features of the video data; the third loss function is constructed according to the global features of the text data and the local features of the video data. According to the embodiment, fine-grained and comprehensive multi-modal model training is realized, and the multi-modal model is endowed with finer-grained and deeper understanding ability.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a training method and device for a multimodal model. Background Art

[0002] A multimodal model is a deep learning model that can effectively integrate data of different modalities (such as images, text, sound, etc.) to achieve more efficient information understanding and intelligent recognition. The multimodal models used in existing video retrieval scenarios are usually trained based on the similarity between the global features of video modality data and text modality data. Specifically, the parameters of the multimodal model are tuned by minimizing the global loss function of each modality data.

[0003] In the process of implementing the present invention, the inventors found that the prior art has the following problems:

[0004] The accuracy of existing video retrieval based on multimodal models is not high, and the main reason for its low accuracy is that the multimodal model is relatively weak in understanding fine-grained semantics and cannot have a fine-grained and in-depth understanding of the received multimodal data information. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a method and device for training a multimodal model, which realizes a fine-grained and comprehensive training of a multimodal model. On the premise of maintaining overall semantic consistency, the first loss function, the second loss function and the third loss function are used to improve the robustness of the multimodal model and its ability to understand local information, giving the multimodal model a more fine-grained and in-depth understanding ability.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a method for training a multimodal model is provided, comprising:

[0007] Acquire a training sample set of a multimodal model, wherein the training sample set includes video data and text data;

[0008] The multimodal model is trained using the training sample set, and the parameters of the multimodal model are adjusted and optimized by minimizing at least two of a first loss function, a second loss function, and a third loss function to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; and the third loss function is constructed based on the global features of the text data and the local features of the video data.

[0009] Optionally, the second loss function is constructed in the following manner: obtaining local semantic features of the text data and local semantic features of the video data; constructing the similarity between the local semantic features of the text data and the local semantic features of the video data; determining the local cross-modal features of the text data based on the similarity and the local semantic features of the video data; and obtaining the second loss function based on the local cross-modal features and the local semantic features of the text data.

[0010] Optionally, the third loss function is constructed in the following manner: obtaining the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the local momentum features of the transformed video after performing a spatiotemporal transformation on the video data; constructing a first cosine similarity between the global semantic features of the text data and the local semantic features of the video data, and a second cosine similarity between the global momentum features of the text data and the local momentum features of the transformed video; and obtaining the third loss function based on the first cosine similarity and the second cosine similarity.

[0011] Optionally, obtaining local semantic features of the text data and local semantic features of the video data includes: performing word segmentation on the text data to obtain a text sequence; performing feature extraction on the text sequence using a text encoder to obtain local semantic features of the text data; performing video frame extraction and segmentation on the video data to obtain a video sequence; and performing feature extraction on the video sequence using a video encoder to obtain local semantic features of the video data.

[0012] Optionally, the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the local momentum features of the transformed video after the video data is subjected to a spatiotemporal transformation, are obtained, including: using a text encoder to perform feature extraction on the overall semantic information of the text data to obtain the global semantic features of the text data; extracting and segmenting video frames of the video data to obtain a video sequence; using a video encoder to perform feature extraction on the video sequence to obtain the local semantic features of the video data; using a momentum text encoder to perform feature extraction on the overall semantic information of the text data to obtain the global momentum features of the text data; performing a spatiotemporal transformation on data frames in the video data to obtain a transformed video; extracting and segmenting video frames of the transformed video to obtain a transformed video sequence; using a momentum video encoder to perform feature extraction on the changing video sequence to obtain the local momentum features of the transformed video.

[0013] Optionally, the momentum video encoder is constructed based on the structure and parameters of the video encoder, and the momentum text encoder is constructed based on the structure and parameters of the text encoder.

[0014] Optionally, the spatiotemporal transformation includes a flip transformation and a cropping transformation of the video data.

[0015] According to a second aspect of an embodiment of the present invention, a method for performing video retrieval based on a multimodal model is provided, comprising:

[0016] Obtain text data and video datasets to be retrieved;

[0017] The text data and the video data set are input into a pre-built multimodal model, and the text data and each video data in the video data set are matched and retrieved using the multimodal model to obtain target video data output by the multimodal model that matches the text data; wherein the multimodal model is trained according to any one of the methods provided in the first aspect of the embodiments of the present invention.

[0018] According to a third aspect of an embodiment of the present invention, there is provided a training device for a multimodal model, comprising:

[0019] A sample acquisition module, used to acquire a training sample set of a multimodal model, wherein the training sample set includes video data and text data;

[0020] A model training module is used to train the multimodal model using the training sample set, and adjust and optimize the parameters of the multimodal model by minimizing at least two of a first loss function, a second loss function, and a third loss function to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; and the third loss function is constructed based on the global features of the text data and the local features of the video data.

[0021] Optionally, the training device of the multimodal model also includes a second loss function construction module, which is used to: obtain local semantic features of the text data and local semantic features of the video data; construct the similarity between the local semantic features of the text data and the local semantic features of the video data; determine the local cross-modal features of the text data based on the similarity and the local semantic features of the video data; and obtain the second loss function based on the local cross-modal features and the local semantic features of the text data.

[0022] Optionally, the training device of the multimodal model also includes a third loss function construction module, which is used to: obtain the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the local momentum features of the transformed video after the video data is transformed in time and space; construct a first cosine similarity between the global semantic features of the text data and the local semantic features of the video data, and a second cosine similarity between the global momentum features of the text data and the local momentum features of the transformed video; and obtain the third loss function based on the first cosine similarity and the second cosine similarity.

[0023] Optionally, the second loss function construction module is also used to: perform word segmentation on the text data to obtain a text sequence; use a text encoder to extract features from the text sequence to obtain local semantic features of the text data; extract and segment video frames from the video data to obtain a video sequence; use a video encoder to extract features from the video sequence to obtain local semantic features of the video data.

[0024] Optionally, the third loss function construction module is also used to: use a text encoder to extract features from the overall semantic information of the text data to obtain global semantic features of the text data; extract and segment video frames from the video data to obtain a video sequence; use a video encoder to extract features from the video sequence to obtain local semantic features of the video data; use a momentum text encoder to extract features from the overall semantic information of the text data to obtain global momentum features of the text data; perform spatiotemporal transformation on data frames in the video data to obtain a transformed video; extract and segment video frames from the transformed video to obtain a transformed video sequence; use a momentum video encoder to extract features from the changing video sequence to obtain local momentum features of the transformed video.

[0025] Optionally, the momentum video encoder is constructed based on the structure and parameters of the video encoder, and the momentum text encoder is constructed based on the structure and parameters of the text encoder.

[0026] Optionally, the spatiotemporal transformation includes a flip transformation and a cropping transformation of the video data.

[0027] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic training device of a multimodal model, comprising:

[0028] one or more processors;

[0029] a storage device for storing one or more programs,

[0030] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first aspect of the embodiment of the present invention.

[0031] According to a fifth aspect of an embodiment of the present invention, there is provided a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method provided in the first aspect of the embodiment of the present invention is implemented.

[0032] According to a sixth aspect of an embodiment of the present invention, there is provided a computer program product, including a computer program, wherein when the computer program is executed by a processor, the method provided in the first aspect of the embodiment of the present invention is implemented.

[0033] An embodiment of the invention has the following advantages or beneficial effects: by obtaining a training sample set of a multimodal model, the training sample set includes video data and text data; using the training sample set to train the multimodal model, by minimizing at least two of the first loss function, minimizing the second loss function and minimizing the third loss function, adjusting and optimizing the parameters of the multimodal model, and obtaining a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; the third loss function is a technical solution constructed based on the global features of the text data and the local features of the video data, thereby realizing a fine-grained and comprehensive training of the multimodal model, and under the premise of maintaining overall semantic consistency, utilizing the first loss function, the second loss function and the third loss function to enhance the robustness of the multimodal model and its ability to understand local information, thereby giving the multimodal model a more fine-grained and in-depth understanding capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0035] Figure 1 is a schematic diagram of the main process of the training method of the multimodal model according to an embodiment of the present invention;

[0036] Figure 2 is a flowchart of a method for constructing a second loss function of a multimodal model according to an embodiment of the present invention;

[0037] Figure 3 is a flowchart of a method for constructing a third loss function of a multimodal model according to an embodiment of the present invention;

[0038] Figure 4 Schematic diagram of the construction principle of the loss function of the multimodal model of an embodiment of the present invention;

[0039] Figure 5is a schematic diagram of main modules of a training device for a multimodal model according to an embodiment of the present invention;

[0040] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0041] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that in the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] The accuracy of existing video retrieval based on multimodal models is not high, and the main reason for its low accuracy is that the multimodal model is relatively weak in understanding fine-grained semantics, and cannot have a fine-grained and in-depth understanding of the received multimodal data information, and cannot meet practical applications well.

[0045] In order to solve the above-mentioned problems existing in the prior art, the present invention proposes a training method for a multimodal model, which realizes a fine-grained and comprehensive training of a multimodal model. On the premise of maintaining overall semantic consistency, the first loss function, the second loss function and the third loss function are used to improve the robustness of the multimodal model and the ability to understand local information, thereby giving the multimodal model a more fine-grained and in-depth understanding ability.

[0046] Figure 1 is a schematic diagram of the main process of the training method of the multimodal model according to an embodiment of the present invention, such as Figure 1 As shown, the training method of the multimodal model in the embodiment of the present invention includes the following steps S101 to S103.

[0047] Step S101: Obtain a training sample set of a multimodal model, where the training sample set includes video data and text data.

[0048] Specifically, for the training task of the multimodal model, the training sample set is purposefully constructed according to the application scenario of the multimodal model. For example, in the video question-answering scenario, it is necessary to construct video-type data and text-type question data, as well as the answer results to the text-type question data based on the content of the video-type data, to form a training sample set; in the video retrieval scenario, video-type data that matches the text-type data is required to form a training sample set. Different application scenarios correspond to different training sample sets, but in general, the data types in the training sample set mainly include video data and text data.

[0049] It can be understood that the multimodal model also supports types such as speech and pictures. For application scenarios that include types such as speech and pictures, the training sample set can also be data of types such as speech and pictures. The embodiment of the present invention mainly takes the video-text type of modality as an example, so the sample set involved mainly includes video data and text data.

[0050] Furthermore, when the training system receives a training task for a multimodal model, the storage address of the training sample set is obtained by parsing the training task, and the storage address is accessed to obtain the training sample set for training the multimodal model.

[0051] Step S102, using the training sample set to train the multimodal model, by minimizing a first loss function, minimizing a second loss function and minimizing at least two of a third loss function, adjusting and optimizing the parameters of the multimodal model, to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; the third loss function is constructed based on the global features of the text data and the local features of the video data.

[0052] Specifically, the embodiments of the present invention aim to improve the fine-grained understanding ability of the multimodal model, as well as the ability of the multimodal model to resist interference and maintain stability, and mainly construct the second loss function and the third loss function, wherein the second loss function focuses on the alignment of word-granularity semantic information of video data and text data, and the third loss function focuses on improving the robustness of the multimodal model.

[0053] The specific second loss function is constructed based on the local features of the text data and the local features of the video data. From the perspective of specific application scenarios, it can be understood as performing fine-grained splitting of the understanding results output by the multimodal model after receiving the training samples, such as word segmentation of text data and frame splitting of video-type data, to obtain fine-grained understanding results; the true value results corresponding to the training samples are also synchronously split to obtain fine-grained true value results; and the second loss function of an embodiment of the present invention is constructed by minimizing the difference between the understanding results and the true value results.

[0054] The specific third loss function is constructed based on the global features of text data and the local features of video data. From the perspective of specific scenarios, it can be understood as first outputting a first understanding result through the multimodal model after receiving a training sample, then performing image scaling processing on the video type data in the training sample and inputting it into the multimodal model to obtain a second understanding result output by the multimodal model. The third loss function is constructed by minimizing the difference between the global semantics of the text in the first understanding result, the second understanding result, and the true value result corresponding to the training sample.

[0055] Users can choose to use it according to business needs. Of course, it is preferred to use these two newly constructed loss functions together with the first loss function in the training of the multimodal model, so as to achieve the goal of simultaneously improving the robustness of the multimodal model and its ability to understand local information while maintaining overall semantic consistency, giving the multimodal model a more fine-grained and in-depth understanding ability.

[0056] Figure 2 is a flow chart of a method for constructing a second loss function of a multimodal model according to an embodiment of the present invention. As another embodiment of the present invention, Figure 2 As shown, the second loss function is constructed in the following way:

[0057] Step S201: Acquire local semantic features of the text data and local semantic features of the video data.

[0058] Specifically, an embodiment of the present invention pre-constructs serialization components for text type data and video type data, inputs text data and video data into the text serialization component and the video serialization component respectively, obtains text sequence and video sequence through the text serialization component and the video serialization component, and then uses an encoder to extract features from the text sequence and the video sequence to obtain local semantic features of the text data and local semantic features of the video data.

[0059] According to one embodiment of the present invention, local semantic features of the text data and local semantic features of the video data are obtained, including: performing word segmentation on the text data to obtain a text sequence; performing feature extraction on the text sequence using a text encoder to obtain local semantic features of the text data; extracting and segmenting video frames of the video data to obtain a video sequence; and performing feature extraction on the video sequence using a video encoder to obtain local semantic features of the video data.

[0060] Specifically, the embodiment of the present invention also provides a specific serialization method. For text data, the text data is converted into a text sequence composed of M words through a word segmentation algorithm; for video data, according to a preset extraction rule (such as average extraction or extraction according to the scene image frames involved in the video content), a number of image frames are extracted from the video data, and then each extracted image frame is segmented into non-overlapping frame segments, and all frame segments are integrated into a video sequence according to the image frames. It can be understood that in integrating the image frame segments, it is preferred to orderly combine the image frames into a video sequence according to the order of the image frames in the video data and the positions of the image frame segments in the corresponding image frames.

[0061] Furthermore, the serialized text sequence is input into a predefined text encoder to obtain the local semantic features of the text data {t1, t2, …, t M}, where t i Represents the feature of the i-th word in the sequence. The serialized video sequence is input into a predefined video encoder to obtain the local semantic features of the video data {v1, v2, …, v N}, where v i is the feature representation of the i-th segment.

[0062] Step S202: constructing the similarity between the local semantic features of the text data and the local semantic features of the video data.

[0063] Specifically, for different types of data of the multimodal model, the different types of data need to be matched and aligned, and the matching alignment can be achieved by comparing the similarity between the local semantic features of the above text data and the local semantic features of the video data.

[0064] The embodiment of the present invention adopts a method based on the attention mechanism to calculate the similarity between them, taking the jth text feature t j and the kth video feature v k For example, the similarity between the two j,k Calculated by the following formula:

[0065]

[0066] Among them, Q and K are learnable weight matrices, which are used to map text features and video features to the same feature space; and the denominator is the scaling factor for normalization to ensure the stability of similarity calculation, and d represents the dimension of the feature vector. The softmax function here ensures the normalization of all pair similarities so that their sum is 1. Specifically, when calculating similarity, for the jth text feature t j , we need to calculate the corresponding video segments v1, v2, …, v N The similarity.

[0067] Step S203: Determine the local cross-modal features of the text data according to the similarity and the local semantic features of the video data.

[0068] Specifically, based on the similarity between the local semantic features of the text data and the local semantic features of the video data calculated above, the local cross-modal feature representation of the text data is further derived. j , whose local cross-modal feature representation c j It can be calculated by the following formula:

[0069]

[0070] Among them, V is a learnable weight matrix used to transform the original video features into the cross-modal fusion space. Through this operation, we can aggregate the local semantic feature vectors of each video data according to the similarity weight, thereby constructing a local semantic feature t that reflects the text data. j Local cross-modal feature representation of interactive information with video modality c j .

[0071] Step S204: Obtain the second loss function according to the local cross-modal features and the local semantic features of the text data.

[0072] Specifically, in order to achieve local matching and alignment of local semantic features of text data and local semantic features of video data, the embodiment of the present invention proposes a second loss function (Local-to-Local Loss) for local alignment, denoted as L l2l The specific second loss L l2l The definition is as follows:

[0073]

[0074] Where M is the number of local semantic features of text data in the training batch, t j is the vector representation of the local semantic features of the jth text data, c jis the cross-modal feature representation corresponding to the j-th text feature, and τ is a temperature parameter used to control the smoothness of the distribution.

[0075] The embodiment of the present invention calculates the similarity between the local semantic features of text data and the local semantic features of video data based on the cross-attention mechanism, and then calculates the local cross-modal features of the text data based on the similarity and the local semantic features of the video data, and then constructs a second loss function, which provides a loss function that focuses on fine-grained semantic alignment for the training of multimodal models, and helps improve the ability of multimodal models in fine-grained semantic understanding.

[0076] Figure 3 is a flow chart of a method for constructing a third loss function of a multimodal model of an embodiment of the present invention. As another embodiment of the present invention, Figure 3 As shown, the third loss function is constructed in the following way:

[0077] Step S301, obtaining the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the transformed video local momentum features after performing a spatiotemporal transformation on the video data.

[0078] According to an embodiment of the present invention, the spatiotemporal transformation includes a flip transformation and a cropping transformation of the video data.

[0079] Specifically, the text data is understood, summarized and concluded as a whole to obtain global semantics, and the text encoder is used to extract features from the global semantics to obtain global semantic features. The local semantic features of video data are similar to the above, and the text encoder is used to extract features from the serialized video sequence.

[0080] In addition, in order to improve the robustness of the multimodal model, the embodiment of the present invention utilizes the local semantic equivariance of video data under spatiotemporal transformation (that is, when the video data undergoes spatiotemporal transformation, the correspondence between the local segment and the text data will also change accordingly), based on the spatiotemporal transformation of the video data (the spatiotemporal transformation of the embodiment of the present invention mainly refers to the flipping transformation and cropping transformation of the video data), to construct feature matching alignment at the global text and local video levels. And in order to smooth and accelerate the training process of the multimodal model, the embodiment of the present invention introduces global momentum features corresponding to the global semantic features of the text data and local momentum features of the video after the spatiotemporal transformation of the video corresponding to the local semantic features of the video data during the training stage.

[0081] According to another embodiment of the present invention, the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the local momentum features of the transformed video after the video data is subjected to a spatiotemporal transformation, are obtained, including: using a text encoder to perform feature extraction on the overall semantic information of the text data to obtain the global semantic features of the text data; extracting and segmenting video frames of the video data to obtain a video sequence; using a video encoder to perform feature extraction on the video sequence to obtain the local semantic features of the video data; using a momentum text encoder to perform feature extraction on the overall semantic information of the text data to obtain the global momentum features of the text data; performing a spatiotemporal transformation on the data frames in the video data to obtain a transformed video; extracting and segmenting the video frames of the transformed video to obtain a transformed video sequence; using a momentum video encoder to perform feature extraction on the changing video sequence to obtain the local momentum features of the transformed video.

[0082] Specifically, in the embodiment of the present invention, a text encoder is used to extract features from the overall semantic information of text data, and the overall semantic information of the text data is input into the text encoder to obtain a global semantic feature t representing the entire semantics of the text data. cls The method for obtaining the local semantic features of the video data is similar to the method for obtaining the local semantic features of the video data when constructing the second loss function. The local semantic features of the video data {v1, v2, …, v N}.

[0083] In addition, the present invention introduces a momentum text encoder corresponding to the text encoder and a momentum video encoder corresponding to the video encoder, which are mainly used to control the degree of influence of new observations on the current estimate and the degree of retention of past estimates. The overall semantic information of the text data is input into the momentum text encoder to obtain the global momentum feature of the text data. The data frames in the video data are subjected to a spatiotemporal transformation, specifically, at least one of a flip transformation and a crop transformation can be performed to obtain a transformed video, and then the transformed video is subjected to a similar extraction and segmentation method as that used in the above construction of the second loss function to obtain a changed video sequence, and finally the changed video sequence is input into a momentum video encoder to obtain a transformed video local momentum feature output by the momentum video encoder. in is the momentum feature representation of the i-th segment.

[0084] According to still another embodiment of the present invention, the momentum video encoder is constructed based on the structure and parameters of the video encoder, and the momentum text encoder is constructed based on the structure and parameters of the text encoder.

[0085] Specifically, the momentum video encoder of the embodiment of the present invention has the same structure as the video encoder, for example, both are deep learning Transformer structures based on the self-attention mechanism, and only the parameters are different. The video encoder updates the parameters according to the gradient descent method, and the parameters of the momentum video encoder are updated on the basis of the parameters of the video encoder, and the gradient direction in the previous period of time is also considered, and the contribution of the previous gradient direction is controlled by the momentum parameter to alleviate the oscillation and deviation from the optimal solution caused by the direction change, and accelerate the convergence speed of network training. Correspondingly, the momentum text encoder also has the same composition structure as the text encoder, and only the parameters are different. The text encoder updates the parameters according to the gradient descent method, and the parameters of the momentum text encoder are updated on the basis of the parameters of the text encoder, and the gradient direction in the previous period of time is also considered, and the contribution of the previous gradient direction is controlled by the momentum parameter to alleviate the oscillation and deviation from the optimal solution caused by the direction change, and accelerate the convergence speed of network training.

[0086] Step S302: construct a first cosine similarity between the global semantic features of the text data and the local semantic features of the video data, and a second cosine similarity between the global momentum features of the text data and the local momentum features of the transformed video.

[0087] Specifically, calculate the global semantic features t of the text data cld The first cosine similarity A between the local semantic features of the video data generated by the video data V. Specifically, the local semantic features v of the kth video data k and the global semantic features of text data cls The first cosine similarity calculation formula between is defined as:

[0088]

[0089] Similarly, the global momentum feature of the text data is calculated and transform video Corresponding transformed video local momentum features The second cosine similarity between Specific k-th transformation video local momentum feature Global momentum features with text data The second cosine similarity calculation formula between is defined as:

[0090]

[0091] Step S303: Obtain the third loss function according to the first cosine similarity and the second cosine similarity.

[0092] Specifically, there is a close correlation between the semantic content described by text data and video data, which manifests itself as local semantic equivariance when the video undergoes spatiotemporal transformation, that is, the semantic content of the video region should remain consistent after transformation. Based on this observation, we define a third loss function L for global-local cross-modal semantic alignment g2l , which is used to measure the consistency between the transformed video features after spatiotemporal transformation and the original video features. Specifically, we expect that A should be consistent with Therefore, the third loss function can be expressed as:

[0093]

[0094] Where N is the number of local semantic features of the video data in the training batch.

[0095] Figure 4 Schematic diagram of the construction principle of the loss function of the multimodal model of the embodiment of the present invention. It mainly involves video data and text data. For video data, the original video data is first serialized to obtain a video sequence, and a video encoder is used to extract features from the video sequence to obtain local semantic features of the video data {v1, v2, ..., v N At the same time, the original video data is transformed and serialized in time and space to obtain a transformed video sequence, and the momentum video encoder is used to extract features of the transformed video sequence to obtain the local momentum features of the transformed video. For text data, we first serialize the text data to get a text sequence. To facilitate storage and management and save resources, we also add the overall semantic information of the text data to the text sequence. We use the text encoder to extract features from the text sequence containing the overall semantic information to get the global semantic features of the text data and the local semantic features of the text data. cls ,t1,t2,…,t M At the same time, the momentum text encoder is used to extract the overall semantic information of the text data to obtain the global momentum feature of the text data.

[0096] Furthermore, according to the local semantic features of the text data {t1, t2, …, t M} and the local semantic features of the video data {v1,v2,…,v N}, construct the second loss function. According to the global semantic features of text data t cls , local semantic features of video data {v1,v2,…,v N}、Global momentum features of text data And the local momentum features of the transformed video after the spatiotemporal transformation of the video data Construct the third loss function.

[0097] In addition, for the construction of the first loss function, the overall semantic information of the video data can be input into the video encoder to obtain the global semantic features v of the video data output by the video encoder. cls , combined with the global semantic features of the above text data t cls , and get the first loss function.

[0098] The embodiment of the present invention utilizes minimizing the first loss function, minimizing the second loss function, and minimizing the third loss function to train the multimodal model in a fine-grained and comprehensive manner, thereby improving the robustness of the multimodal model and its ability to understand local information.

[0099] Based on the above-mentioned multimodal model training method, an embodiment of the present invention also provides a method for video retrieval based on a multimodal model, comprising: obtaining text data and a video data set to be retrieved; inputting the text data and the video data set into a pre-constructed multimodal model, using the multimodal model to match and retrieve the text data and each video data in the video data set, and obtaining target video data output by the multimodal model that matches the text data; wherein the multimodal model is trained according to any of the above-mentioned methods for the multimodal model.

[0100] It should be noted that when using the above-mentioned trained multimodal model for specific application reasoning, it is no longer necessary to perform spatiotemporal transformation on the video data, and accordingly, there is no need to introduce a momentum video encoder and a momentum text encoder. It is only necessary to directly input the video data and the text data into the video encoder and the text encoder to obtain the corresponding feature representations. These feature representations extracted by the trained multimodal model can capture the key information of the video and the text, and can be used for video retrieval. In addition, it can be understood that the trained multimodal model of the embodiment of the present invention can also be applied to other scenarios, such as video description generation scenarios (input a video and output a paragraph describing the video content), video question and answer scenarios (input a video and a question, and output the answer to the question based on the content of the video), etc.

[0101] Figure 5 Schematic diagram of the main modules of the training device of the multimodal model according to an embodiment of the present invention. Figure 5 As shown, the multimodal model training device 500 mainly includes a sample acquisition module 501 and a model training module 502.

[0102] A sample acquisition module 501 is used to acquire a training sample set of a multimodal model, wherein the training sample set includes video data and text data;

[0103] The model training module 502 is used to train the multimodal model using the training sample set, and adjust and optimize the parameters of the multimodal model by minimizing at least two of the first loss function, the second loss function, and the third loss function to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; and the third loss function is constructed based on the global features of the text data and the local features of the video data.

[0104] According to one embodiment of the present invention, the training device 500 of the multimodal model also includes a second loss function construction module (not shown in the figure), which is used to: obtain the local semantic features of the text data and the local semantic features of the video data; construct the similarity between the local semantic features of the text data and the local semantic features of the video data; determine the local cross-modal features of the text data based on the similarity and the local semantic features of the video data; and obtain the second loss function based on the local cross-modal features and the local semantic features of the text data.

[0105] According to another embodiment of the present invention, the training device 500 of the multimodal model also includes a third loss function construction module (not shown in the figure), which is used to: obtain the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the local momentum features of the transformed video after the video data is subjected to a spatiotemporal transformation; construct a first cosine similarity between the global semantic features of the text data and the local semantic features of the video data, and a second cosine similarity between the global momentum features of the text data and the local momentum features of the transformed video; and obtain the third loss function based on the first cosine similarity and the second cosine similarity.

[0106] According to another embodiment of the present invention, the second loss function construction module (not shown in the figure) is also used to: perform word segmentation on the text data to obtain a text sequence; perform feature extraction on the text sequence using a text encoder to obtain local semantic features of the text data; perform video frame extraction and segmentation on the video data to obtain a video sequence; perform feature extraction on the video sequence using a video encoder to obtain local semantic features of the video data.

[0107] According to another embodiment of the present invention, the third loss function construction module (not shown in the figure) is also used to: use a text encoder to extract features from the overall semantic information of the text data to obtain global semantic features of the text data; extract and segment video frames from the video data to obtain a video sequence; use a video encoder to extract features from the video sequence to obtain local semantic features of the video data; use a momentum text encoder to extract features from the overall semantic information of the text data to obtain global momentum features of the text data; perform spatiotemporal transformation on data frames in the video data to obtain a transformed video; extract and segment video frames from the transformed video to obtain a transformed video sequence; use a momentum video encoder to extract features from the changing video sequence to obtain local momentum features of the transformed video.

[0108] According to another embodiment of the present invention, the momentum video encoder is constructed based on the structure and parameters of the video encoder, and the momentum text encoder is constructed based on the structure and parameters of the text encoder.

[0109] According to yet another embodiment of the present invention, the spatiotemporal transformation includes a flip transformation and a cropping transformation on the video data.

[0110] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention can be applied.

[0111] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0112] Users can use terminal devices 601, 602, 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications, such as model training applications, etc. (only as examples), can be installed on terminal devices 601, 602, 603.

[0113] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0114] Server 605 can be a server that provides various services, such as a background management server that provides support for the multimodal model performed by the user using the terminal devices 601, 602, and 603 (only as an example). The background management server can obtain a training sample set of the multimodal model, wherein the training sample set includes video data and text data; the multimodal model is trained using the training sample set, and the parameters of the multimodal model are adjusted and optimized by minimizing at least two of the first loss function, the second loss function, and the third loss function, to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; the third loss function is constructed based on the global features of the text data and the local features of the video data, and the processing result (such as the multimodal model, etc. - only as an example) is fed back to the terminal device.

[0115] It should be noted that the training method of the multimodal model provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the training device of the multimodal model is generally arranged in the server 605 .

[0116] It should be understood that Figure 6 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0117] Reference below Figure 7 , Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 7 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0118] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0119] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0120] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0121] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0122] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0123] The units involved in the embodiments of the present invention may be implemented in software or hardware. The units described may also be arranged in a processor, for example, it may be described as follows: a processor includes: a sample acquisition module and a model training module.

[0124] Among them, the names of these modules do not constitute a limitation of the modules themselves in some cases. For example, the sample acquisition module can also be described as "a module for acquiring a training sample set of a multimodal model, wherein the training sample set includes video data and text data."

[0125] On the other hand, the present invention also provides a computer-readable medium, which may be included in the device described in the embodiment; or it may exist independently without being assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by a device, the device includes: obtaining a training sample set of a multimodal model, the training sample set including video data and text data; using the training sample set to train the multimodal model, by minimizing at least two of the first loss function, minimizing the second loss function, and minimizing the third loss function, adjusting and optimizing the parameters of the multimodal model to obtain the trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; the third loss function is constructed based on the global features of the text data and the local features of the video data.

[0126] According to the technical solution of the embodiment of the present invention, it has the following advantages or beneficial effects: by obtaining a training sample set of a multimodal model, the training sample set includes video data and text data; using the training sample set to train the multimodal model, by minimizing at least two of the first loss function, minimizing the second loss function and minimizing the third loss function, adjusting and optimizing the parameters of the multimodal model, and obtaining a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; the third loss function is constructed based on the global features of the text data and the local features of the video data. The technical solution realizes a fine-grained and comprehensive training of the multimodal model, and on the premise of maintaining overall semantic consistency, the first loss function, the second loss function and the third loss function are used to improve the robustness of the multimodal model and the ability to understand local information, giving the multimodal model a more fine-grained and in-depth understanding ability.

[0127] The specific implementation methods described herein do not constitute limitations on the scope of protection of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for training a multimodal model, characterized in that: include: Acquire a training sample set of a multimodal model, wherein the training sample set includes video data and text data; The multimodal model is trained using the training sample set, and the parameters of the multimodal model are adjusted and optimized by minimizing at least two of a first loss function, a second loss function, and a third loss function to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; and the third loss function is constructed based on the global features of the text data and the local features of the video data.

2. The method according to claim 1, characterized in that The second loss function is constructed in the following way: Acquire local semantic features of the text data and local semantic features of the video data; Constructing the similarity between the local semantic features of the text data and the local semantic features of the video data; Determining local cross-modal features of the text data according to the similarity and the local semantic features of the video data; The second loss function is obtained according to the local cross-modal features and the local semantic features of the text data.

3. The method according to claim 1, characterized in that: The third loss function is constructed in the following way: Acquire the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the transformed video local momentum features after performing spatiotemporal transformation on the video data; Constructing a first cosine similarity between the global semantic feature of the text data and the local semantic feature of the video data, and a second cosine similarity between the global momentum feature of the text data and the local momentum feature of the transformed video; The third loss function is obtained according to the first cosine similarity and the second cosine similarity.

4. The method according to claim 2, characterized in that: Acquiring the local semantic features of the text data and the local semantic features of the video data includes: Segmenting the text data to obtain a text sequence; extracting features from the text sequence using a text encoder to obtain local semantic features of the text data; The video data is subjected to video frame extraction and segmentation to obtain a video sequence; and a video encoder is used to perform feature extraction on the video sequence to obtain local semantic features of the video data.

5. The method according to claim 3, characterized in that: Acquiring the global semantic features of the text data, the local semantic features of the video data, the global momentum features of the text data, and the transformed video local momentum features after performing spatiotemporal transformation on the video data, including: Using a text encoder to extract features from the overall semantic information of the text data to obtain global semantic features of the text data; Extracting and segmenting video frames from video data to obtain a video sequence; extracting features from the video sequence using a video encoder to obtain local semantic features of the video data; Using a momentum text encoder to extract features from the overall semantic information of the text data to obtain global momentum features of the text data; The data frames in the video data are transformed in time and space to obtain a transformed video; the video frames of the transformed video are extracted and segmented to obtain a transformed video sequence; and the momentum video encoder is used to extract features of the changed video sequence to obtain local momentum features of the transformed video.

6. The method according to claim 5, characterized in that The momentum video encoder is constructed based on the structure and parameters of the video encoder, and the momentum text encoder is constructed based on the structure and parameters of the text encoder.

7. The method according to claim 3, characterized in that The spatiotemporal transformation includes a flip transformation and a crop transformation on the video data.

8. A method for video retrieval based on a multimodal model, characterized in that: include: Obtain text data and video datasets to be retrieved; The text data and the video data set are input into a pre-built multimodal model, and the text data and each video data in the video data set are matched and retrieved using the multimodal model to obtain target video data output by the multimodal model that matches the text data; wherein the multimodal model is trained according to any one of the methods described in claims 1 to 7.

9. A multimodal model training device, characterized in that: include: A sample acquisition module, used to acquire a training sample set of a multimodal model, wherein the training sample set includes video data and text data; A model training module is used to train the multimodal model using the training sample set, and adjust and optimize the parameters of the multimodal model by minimizing at least two of a first loss function, a second loss function, and a third loss function to obtain a trained multimodal model, wherein the first loss function is constructed based on the global features of the text data and the global features of the video data; the second loss function is constructed based on the local features of the text data and the local features of the video data; and the third loss function is constructed based on the global features of the text data and the local features of the video data.

10. A mobile electronic device terminal, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

11. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images