Training method and device of visual text pre-training model, equipment and medium

Abstract

The embodiment of the invention provides a training method and device of a visual text pre-training model, equipment and a medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of inputting an obtained sample video and an obtained sample text into an initial multi-modal processing model, performing segmentation processing on the sample video to obtain a plurality of initial video frames, and extracting text features from the sample text; performing space-time importance evaluation on each initial pixel block in each initial video frame to obtain space-time information; determining a masked video frame based on the spatio-temporal information, and performing feature reconstruction processing on a part in a masked state in the masked video frame based on the text features to obtain a sample reconstruction result; and calculating a loss value according to a sample reconstruction result, and adjusting model parameters of the initial multi-modal processing model according to the loss value to obtain a trained target multi-modal processing model. According to the invention, the ability of the multi-modal processing model obtained by training to understand the multi-modal information can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, equipment, and medium for a visual text pre-training model. Background Art

[0002] The Vision-Language Pretraining Model is a multimodal processing model designed to simultaneously understand and process visual and textual information. Pretraining on large-scale vision-text paired datasets, the Vision-Language Pretraining Model learns the alignment between visual and textual information and constructs cross-modal representations, enabling it to be applied to various downstream tasks.

[0003] In related technologies, in order to enable the multimodal processing model to better understand the important parts of the video, it is often necessary to train the multimodal model by blocking part of the area of ​​a video frame in the current video. However, considering only a single video frame makes it impossible to effectively understand the important parts of the entire video, resulting in the multimodal processing model finally trained having poor understanding ability of multimodal information. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose a training method, device, equipment and medium for a visual text pre-training model, aiming to improve the ability of the trained multimodal processing model to understand multimodal information.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a training method for a visual text pre-training model, comprising:

[0006] Obtaining a sample video and a sample text for describing the sample video;

[0007] Inputting the sample video and sample text into the initial multimodal processing model, segmenting the sample video to obtain multiple initial video frames, and extracting text features from the sample text;

[0008] Performing a spatial importance evaluation on each initial pixel block in each initial video frame to obtain spatial information, and performing a temporal importance evaluation on the initial pixel block to obtain temporal information. For the same initial pixel block, the spatial information and temporal information are superimposed to obtain the corresponding spatiotemporal information.

[0009] Determine the masked video frame based on the spatiotemporal information, and perform feature reconstruction on the masked portion of the masked video frame based on the text features to obtain a sample reconstruction result;

[0010] The loss value is calculated according to the sample reconstruction result, and the model parameters of the initial multimodal processing model are adjusted according to the loss value to obtain the trained target multimodal processing model.

[0011] In some embodiments, the initial video frame includes a plurality of initial pixel blocks;

[0012] The spatial importance of each initial pixel block in each initial video frame is evaluated to obtain spatial information, including:

[0013] Performing linear processing on each initial pixel block in the current initial video frame to obtain a corresponding embedded pixel block;

[0014] Obtain global identification information of the sample video;

[0015] Mapping the global identification information to the current initial video frame to obtain current identification information corresponding to the current initial video frame;

[0016] A dot product operation is performed on the current identification information and the embedded pixel block to obtain a dot product result, and the dot product result is normalized based on the feature dimension of the initial pixel block to obtain the corresponding spatial information.

[0017] In some embodiments, determining the masked video frame based on the spatiotemporal information includes:

[0018] For each spatiotemporal information in the same initial video frame, the initial pixel blocks corresponding to the spatiotemporal information within a second preset range are selected as masked pixel blocks in a masked state, and the initial pixel blocks corresponding to the spatiotemporal information not within the second preset range are selected as non-masked pixel blocks in a non-masked state;

[0019] According to the initial spatial relative positions of the initial pixel blocks, the masked pixel blocks and the non-masked pixel blocks are spliced ​​to obtain a masked video frame.

[0020] In some embodiments, performing temporal importance evaluation on the initial pixel block to obtain temporal information includes:

[0021] Obtaining adjacent initial video frames of the current initial video frame, and predicting an initial key position of the current initial video frame based on the adjacent initial video frames;

[0022] For each initial pixel block in the current initial video frame, if the distance between any initial pixel block and the initial key position is within a first preset range, determining corresponding time information according to a preset weight;

[0023] If the distance between any initial pixel block and the initial key position is not within the first preset range, the corresponding time information is determined according to the preset decay rate.

[0024] In some embodiments, the sample reconstruction result includes a visual reconstruction result and a textual reconstruction result;

[0025] Based on the text features, feature reconstruction processing is performed on the masked portion of the masked video frame to obtain a sample reconstruction result, including:

[0026] Obtain adjacent masked video frames corresponding to adjacent initial video frames, and predict an updated key position of the current masked video frame based on the adjacent masked video frames;

[0027] Determining update time information corresponding to each masked pixel block and non-masked pixel block in the masked video frame based on the update key position;

[0028] Perform feature extraction on the non-masked pixel blocks to obtain non-masked features, and perform feature reconstruction on the masked portion of the masked video frame based on the update time information, non-masked features, and text features to obtain a visual reconstruction result.

[0029] The sample text is randomly occluded to obtain the occluded text, and the occluded text is reconstructed to obtain the text reconstruction result.

[0030] In some embodiments, the adjacent masked video frames are determined by the following steps, including:

[0031] Performing spatial importance evaluation on each adjacent initial pixel block in adjacent initial video frames to obtain adjacent spatial information, and performing temporal importance evaluation on the initial pixel blocks to obtain adjacent temporal information;

[0032] For the same adjacent initial pixel blocks, adjacent spatial information and adjacent temporal information are superimposed to obtain corresponding adjacent spatiotemporal information, and adjacent masked video frames are determined based on the adjacent spatiotemporal information.

[0033] In some embodiments, the loss values ​​include a video-text matching loss value, a video-text contrastive learning loss value, a visual modeling loss value, and a text modeling loss value;

[0034] Calculate the loss value based on the sample reconstruction results, including:

[0035] Determining a video text matching loss value based on the masked pixel block and the corresponding preset text mask;

[0036] Determine the video text contrastive learning loss value based on text features and visual reconstruction results;

[0037] Obtaining a verification visual reconstruction result, and determining a visual modeling loss value based on the visual reconstruction result and the verification visual reconstruction result;

[0038] Based on the text features and text reconstruction results, the text modeling loss value is determined.

[0039] In some embodiments, adjusting the model parameters of the initial multimodal processing model according to the loss value to obtain a trained target multimodal processing model includes:

[0040] Adjust the model parameters of the initial multimodal processing model according to the video text matching loss value to obtain the trained target multimodal processing model; or,

[0041] Adjusting the model parameters of the initial multimodal processing model according to the video-text contrastive learning loss value to obtain a trained target multimodal processing model; or,

[0042] Adjusting the model parameters of the initial multimodal processing model according to the visual modeling loss value to obtain a trained target multimodal processing model; or,

[0043] Adjust the model parameters of the initial multimodal processing model according to the text modeling loss value to obtain the trained target multimodal processing model; or,

[0044] According to the video-text matching loss value, the video-text contrastive learning loss value, the visual modeling loss value and the text modeling loss value, the model parameters of the initial multimodal processing model are jointly adjusted to obtain the trained target multimodal processing model.

[0045] To achieve the above objectives, a second aspect of an embodiment of the present application provides a training device for a visual text pre-training model, comprising:

[0046] An acquisition module, used to acquire a sample video and a sample text for describing the sample video;

[0047] The video text processing module is used to input the sample video and sample text into the initial multimodal processing model, segment the sample video to obtain multiple initial video frames, and extract text features from the sample text;

[0048] a spatiotemporal information determination module configured to perform a spatial importance assessment on each initial pixel block in each initial video frame to obtain spatial information, and to perform a temporal importance assessment on the initial pixel block to obtain temporal information; and for the same initial pixel block, superimposing the spatial information and temporal information to obtain corresponding spatiotemporal information;

[0049] A sample reconstruction module is used to determine the masked video frame based on the spatiotemporal information, and perform feature reconstruction processing on the masked portion of the masked video frame based on the text features to obtain a sample reconstruction result;

[0050] The target training module is used to calculate the loss value based on the sample reconstruction result, and adjust the model parameters of the initial multimodal processing model according to the loss value to obtain the trained target multimodal processing model.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the training method of the visual text pre-training model of the above-mentioned first aspect.

[0052] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training method of the visual text pre-training model of the above-mentioned first aspect.

[0053] The training method, device, equipment and medium of the visual text pre-training model proposed in this application obtain sample videos and sample texts used to describe the sample videos; input the sample videos and sample texts into the initial multimodal processing model, segment the sample videos to obtain multiple initial video frames, and extract text features from the sample texts; perform spatial importance evaluation on each initial pixel block in each initial video frame to obtain spatial information, and perform temporal importance evaluation on the initial pixel block to obtain temporal information; for the same initial pixel block, superimpose the spatial information and temporal information to obtain the corresponding spatiotemporal information; and then, use the spatial importance evaluation and temporal importance evaluation to obtain the corresponding spatiotemporal information. The spatiotemporal information obtained under the comprehensive evaluation of the quality assessment guides the initial multimodal processing module to better understand the background and important object features in the sample video; then, the masked video frame is determined based on the spatiotemporal information, and the feature reconstruction processing is performed on the masked part of the masked video frame based on the text features to obtain the sample reconstruction result. This process greatly enhances the model's ability to understand the sample video content, thereby ensuring the consistency of the masking strategy and the reconstruction target during the model training process; then, the loss value is calculated according to the sample reconstruction result, and the model parameters of the initial multimodal processing model are adjusted according to the loss value to obtain the trained target multimodal processing model. The present application can improve the trained target multimodal processing model's ability to understand multimodal information, especially the video text information. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of an optional implementation environment of a training device for a visual text pre-training model provided in an embodiment of the present application;

[0055] Figure 2 This is an optional flowchart of the training method of the visual text pre-training model provided in the embodiment of the present application;

[0056] Figure 3 This is an optional sample schematic diagram of the training method of the visual text pre-training model provided in the embodiment of the present application;

[0057] Figure 4 This is an optional training flow diagram of the training method of the visual text pre-training model provided in the embodiment of the present application;

[0058] Figure 5 yes Figure 2 An optional implementation flowchart of step 103;

[0059] Figure 6 yes Figure 2 Step 103 in another optional implementation flow chart;

[0060] Figure 7 yes Figure 2 An optional implementation flowchart of step 104;

[0061] Figure 8 yes Figure 2 Step 104 in another optional implementation flowchart;

[0062] Figure 9 yes Figure 8 An optional implementation flowchart of step (104.2.1);

[0063] Figure 10 yes Figure 2 An optional implementation flowchart of step 105;

[0064] Figure 11 yes Figure 2 Step 105 in another optional implementation flow chart;

[0065] Figure 12 This is a schematic diagram of an optional visual experiment result of the training method of the visual text pre-training model provided in an embodiment of the present application;

[0066] Figure 13 This is a schematic diagram of an optional device module of the training device for the visual text pre-training model provided in an embodiment of the present application;

[0067] Figure 14 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0069] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0071] First, let’s analyze some of the terms used in this application:

[0072] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or digital computer-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0073] Machine Learning (ML) is a branch of AI that studies how to enable computer systems to automatically improve and optimize their performance using experience (usually data). In other words, machine learning enables computers to "learn" from data without being explicitly programmed to perform specific tasks.

[0074] Natural Language Processing (NLP): NLP uses computers to process, understand, and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is commonly used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis, and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0075] Next, the technical background related to the embodiments of the present application is introduced:

[0076] The visual text pre-training model is a multimodal processing model for simultaneously understanding and processing visual and textual information. The visual text pre-training model is pre-trained on a large-scale visual-text pairing dataset to learn the alignment relationship between visual and textual information and construct a cross-modal representation. This allows the visual text pre-training model to be applied to various downstream tasks, including but not limited to video retrieval, video question-answering, and video subtitle generation. The specific settings can be determined based on actual conditions and are not limited in the present application.

[0077] In related technologies, in order to enable the multimodal processing model to better understand the important parts of the video, it is often necessary to train the multimodal model by blocking part of the area of ​​a video frame in the current video. However, considering only a single video frame makes it impossible to effectively understand the important parts of the entire video, resulting in the multimodal processing model finally trained having poor understanding ability of multimodal information.

[0078] Based on this, the embodiments of the present application provide a training method, apparatus, equipment and medium for a visual text pre-training model, aiming to improve the trained multimodal processing model's ability to understand multimodal information, especially video text information.

[0079] For example, Figure 1 As shown, Figure 1This is a schematic diagram of an optional implementation environment of a training device for a visual text pre-training model provided in an embodiment of the present application, wherein the implementation environment includes a client 11 and a server 12, wherein the client 11 and the server 12 are connected via a communication network, and the training device for the visual text pre-training model (for ease of description, it can also be referred to as a "training device") is deployed on the server 12 end. The client 11 can input a sample video for training and a sample text for describing the sample video to the server 12; the server 12 obtains the sample video and sample text sent by the client 11; then, the sample video and sample text are input into the initial multimodal processing model, the sample video is segmented to obtain multiple initial video frames, and text features are extracted from the sample text; then, the spatial importance of each initial pixel block in each initial video frame is evaluated to obtain spatial information, and the temporal importance of the initial pixel block is evaluated to obtain temporal information. For the same initial pixel block, the spatial information and temporal information are superimposed to obtain the corresponding spatiotemporal information; then, the masked video frame is determined based on the spatiotemporal information, and the part of the masked video frame that is in a masked state is feature reconstructed based on the text features to obtain a sample reconstruction result; finally, the loss value is calculated based on the sample reconstruction result, and the model parameters of the initial multimodal processing model are adjusted based on the loss value to obtain the trained target multimodal processing model. In this way, when training the multimodal processing model, the embodiment of the present application not only considers the importance of local areas within a single video frame (spatial importance), but also introduces temporal dynamic information across frames (temporal importance). The superposition of the two forms a more comprehensive spatiotemporal information under the spatiotemporal importance evaluation mechanism, and determines the masked area of ​​the current video frame based on the spatiotemporal information to guide feature reconstruction, thereby improving the multimodal processing model's ability to understand video-text.

[0080] Among them, the server 12 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In addition, the server 12 can also be a node server in the blockchain network. The client 11 can be a mobile phone, a computer, an intelligent voice interaction device, a smart wearable device, a smart home appliance, a car terminal, etc., but is not limited to this. The client 11 and the server 12 can be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited here.

[0081] It should be noted that in the embodiments of the present application, when it comes to information related to user characteristics such as user basic information or user identity, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained first. After clearly obtaining the user's separate permission or consent, the necessary data for the normal operation of the embodiments of the present application will be obtained. For example, before obtaining sample videos and sample texts used to describe the sample videos, the embodiments of the present application will first obtain the authorization or consent of relevant personnel, otherwise the sample videos and sample texts that cannot be applied to the embodiments of the present application will be obtained. In addition, the other relevant data obtained by the training device of the present application are all authorized data, which will not be repeated here.

[0082] In the embodiment of the present application, the training device will be described from the perspective of the training device, which can be integrated into a computer device, such as a server. Figure 2 As shown, Figure 2 This is an optional flowchart of the training method of the visual text pre-training model provided in the embodiment of the present application. Figure 2 The method may include but is not limited to the following steps 101 to 105. When the training device executes the training method of the visual text pre-training model, the specific process is as follows. It should be noted that this embodiment is Figure 2 The order of step 101 to step 105 is not specifically limited, and the order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0083] Step 101: Obtain a sample video and a sample text for describing the sample video.

[0084] Step 101 is described in detail below.

[0085] In some embodiments, "sample video" and "sample text" refer to data pairs used to train a multimodal processing model so that the multimodal processing model learns how to match visual information (video) with corresponding semantic information (text description) as basic material.

[0086] The sample video refers to the video clip input into the initial multimodal processing model. The sample video can contain a variety of content, such as movie clips, TV program clips, and user-generated content. During the training process, the sample video is segmented into multiple frames (initial video frames), and then each frame is further segmented into multiple pixel blocks (initial pixel blocks). These initial pixel blocks are the basic units for spatial and temporal importance assessment, which provide an important basis for subsequent mask processing.

[0087] The sample text is used to describe the text information of the corresponding sample video. The sample text can be a detailed description or summary of the sample video, or a label related to the sample video. The language of the sample text can be Chinese, English, or other languages ​​that the model can understand. This is not limited in the embodiments of this application. The role of the sample text is to help the model understand the content of the sample video. Specifically, the corresponding text features will be extracted through the text encoder. The text features will then participate in the reconstruction process of the obscured part of the video frame, enhancing the model's cross-modal understanding ability.

[0088] Further, if Figure 3 As shown, Figure 3 This is an optional sample diagram of the training method of the visual text pre-training model provided in the embodiment of the present application. A sample video v includes multiple initial video frames {v1, v2, v3...v T}, the sample text z is the description corresponding to the sample video v, Figure 3 The sample text in the example is: Senior couple looking through binoculars on sailboat together.

[0089] Furthermore, the sample video and sample text can be obtained from public data sets, such as the Microsoft Research Video to Text (MSRVTT) and the Densely Annotated VIdeo Description MOment Dataset (DiDeMo). Public data sets contain a large number of annotated videos and their corresponding text descriptions. Alternatively, you can record the sample video yourself and obtain the corresponding sample text in the form of manual annotation or artificial intelligence annotation. If the artificial intelligence annotation method is adopted, you can use automatic speech recognition technology to generate subtitles for the sample video as sample text describing the sample video, or intelligently identify the content of the sample video and generate the corresponding sample text. There are many ways to obtain sample videos and corresponding sample texts, which can be adaptively adjusted according to actual conditions. This is only an example and does not limit the embodiments of the present application.

[0090] In step 102 , the sample video and the sample text are input into an initial multimodal processing model, the sample video is segmented to obtain a plurality of initial video frames, and text features are extracted from the sample text.

[0091] Step 102 is described in detail below.

[0092] Among them, the multimodal processing model is a visual text pre-training model. The untrained or poorly trained multimodal processing model is called the "initial multimodal processing model". For the sake of convenience, the multimodal processing model can also be referred to as the "model". The multimodal processing model is trained based on the Masked AutoEncoder (MAE) technology, in which MAE first masks some pixel blocks in the initial video frame, and then uses the remaining visible pixel blocks to reconstruct the occluded pixel blocks, thereby forcing the multimodal processing model to learn the information between each pixel block. Furthermore, the downstream application scenarios of the multimodal processing model include but are not limited to: video subtitle generation, cross-modal retrieval, action recognition and behavior analysis, video question answering, video anomaly detection and analysis, etc.

[0093] like Figure 4 As shown, Figure 4 This is an optional training flow diagram of the training method of the visual text pre-training model provided in the embodiment of the present application. First, a sample video x={x1, x2, x3...x t ...x T} and the corresponding sample text z, where x t represents the tth frame among the T initial video frames.

[0094] Furthermore, a tokenizer is used to block each initial video frame and convert it into T*M visual tokens. Each initial video frame includes T groups of visual tokens, and each group includes M (1, 2...i...M) visual tokens. In the fields of computer vision and multimodal learning, a "visual token" refers to a basic processing unit extracted from a video frame. In the embodiments of this application, the visual token is the initial pixel block. Next, a text encoder is used to encode the text tokens and extract the text features y. In the field of natural language processing (NLP), a "text token" refers to the smallest unit in a text, which can be a word, a punctuation mark, a number, or any other meaningful string of characters.

[0095] The role of the Tokenizer is to convert high-dimensional sample videos into more compact visual tokens for efficient subsequent compression and representation, thereby effectively reducing the computational requirements during model training and inference, and improving the efficiency of model generation and inference. Of course, other tools can also be used to process each initial video frame, and this embodiment of the application does not limit this.

[0096] Step 103 , performing spatial importance evaluation on each initial pixel block in each initial video frame to obtain spatial information, and performing temporal importance evaluation on the initial pixel block to obtain temporal information. For the same initial pixel block, the spatial information and temporal information are superimposed to obtain corresponding spatiotemporal information.

[0097] Step 103 is described in detail below.

[0098] Spatial importance assessment refers to the quantitative evaluation of the importance of each initial pixel block in each initial video frame. This process involves calculating the importance of the initial pixel block relative to the initial video frame, such as whether it contains significant objects or features. Temporal importance assessment refers to the evaluation of the importance of the initial pixel block at the same position over time between consecutive video frames. Its purpose is to identify those initial pixel blocks that have changed significantly over time or contain key action information.

[0099] Furthermore, the spatiotemporal information obtained through the combined evaluation of spatial and temporal importance assessments guides the initial multimodal processing module to better understand the background and important object features in the sample video. This process greatly enhances the model's ability to understand the sample video content, thereby ensuring the consistency of the masking strategy and reconstruction goals during the model training process. This helps improve the final trained target multimodal processing model's ability to understand multimodal information (video-text), enabling it to perform well in a wider range of application scenarios.

[0100] In some embodiments, as Figure 5 As shown, Figure 5 yes Figure 2 Step 103 in the flowchart is an optional implementation, wherein spatial importance evaluation is performed on each initial pixel block in each initial video frame to obtain spatial information, including the following steps:

[0101] (103.1.1) performing linear processing on each initial pixel block in the current initial video frame to obtain a corresponding embedded pixel block;

[0102] (103.1.2) Obtaining global identification information of the sample video;

[0103] (103.1.3) Mapping the global identification information to the current initial video frame to obtain current identification information corresponding to the current initial video frame;

[0104] (103.1.4) Perform a dot product operation on the current identification information and the embedded pixel block to obtain a dot product result, and normalize the dot product result based on the feature dimension of the initial pixel block to obtain the corresponding spatial information.

[0105] Steps (103.1.1) to (103.1.4) are described in detail below.

[0106] In some embodiments, the following <1> , for each visual token (initial pixel block) in each initial video frame, spatial importance evaluation is performed to obtain spatial information, where the spatial information can be a spatial importance score

[0107]

[0108] Where [cls] is the global identification information. Usually, the global identification information can be determined by the first initial video frame, which represents the global information of the entire sample video; Φ represents the linear layer, which linearly processes each initial pixel block in the current initial video frame to obtain the corresponding embedded pixel block. Φ Q ([cls]) represents mapping the global video information to the current initial video, such as mapping the global video information to the query space, thereby obtaining the current identification information Φ corresponding to the current initial video frame Q ([cls]); D represents the feature dimension; “·” represents the dot product operation; the softmax function is a commonly used activation function in machine learning. It is mainly used to convert a K-dimensional vector containing arbitrary real numbers into another K-dimensional vector. Each element of the output vector lies in the interval (0, 1), and the sum of all elements is equal to 1 to achieve normalization.

[0109] Furthermore, the embodiments of the present application accurately quantify the relative spatial importance of each initial pixel block in the corresponding initial video frame, so that the model focuses on areas containing key information, avoids wasting computing resources on areas containing fewer features or features that are not very helpful, and thus improves the accuracy and efficiency of the model's feature extraction during the training process.

[0110] In some embodiments, as Figure 6 As shown, Figure 6 yes Figure 2 Step 103 in FIG. 1 is another optional implementation flowchart, in which temporal importance evaluation is performed on the initial pixel block to obtain temporal information, comprising the following steps:

[0111] (103.2.1) Obtaining adjacent initial video frames of the current initial video frame, and predicting the initial key position of the current initial video frame based on the adjacent initial video frames;

[0112] (103.2.2) For each initial pixel block in the current initial video frame, if the distance between any initial pixel block and the initial key position is within a first preset range, determine the corresponding time information according to the preset weight;

[0113] (103.2.3) If the distance between any initial pixel block and the initial key position is not within the first preset range, the corresponding time information is determined according to the preset attenuation rate.

[0114] The following describes steps (103.2.1) to (103.2.3) in detail.

[0115] In some embodiments, after completing the spatial importance assessment for each initial pixel block, a temporal importance assessment is also performed on each initial pixel block to obtain temporal information, where the temporal information can be a temporal importance score. Furthermore, leveraging the rich temporal information and context provided by adjacent initial video frames, the initial multimodal processing model can more accurately identify important actions or significantly changed regions in the current initial video frame in the temporal dimension.

[0116] First, if Figure 4 As shown, the current initial video frame (x t ), and x t The adjacent initial video frames (x t-1 ) is input into a lightweight network (Predictor) to obtain the initial key position (p t ), and the initial key positions corresponding to the adjacent initial video frames (p t-1 ). The initial key position refers to the position containing important actions or significant changes in the current initial video frame, which is predicted with the help of adjacent initial video frames.

[0117] Furthermore, the lightweight network can be a mobile network (MobileNet), a squeeze network (SqueezeNet), a shuffle network (ShuffleNet), a ghost network (GhostNet), etc. Of course, the lightweight network for processing adjacent video frames can be specifically selected according to actual conditions. This is only an example and does not mean that the embodiments of the present application are not limited to this.

[0118] Then, as follows <2> 、 <3> As shown, the temporal importance score is obtained by evaluating the temporal importance of each visual token in each initial video frame.

[0119]

[0120] in, Indicates splicing in the feature dimension; η is the preset weight, β is the preset attenuation rate, and the specific values ​​of the preset weight and the preset attenuation rate can be adjusted according to the actual situation; σ is the preset distance threshold, and the first preset range can be determined based on σ; d = dist(p, x t，i) represents the distance from the i-th visual token in the current initial video frame to the corresponding initial key position.

[0121] Furthermore, for the same initial pixel block, the spatial information and time information are superimposed to obtain the spatiotemporal information (spatiotemporal importance score) of the initial pixel block comprehensively determined in the spatial dimension and the temporal dimension.

[0122] Step 104 : determining a masked video frame based on the spatiotemporal information, and performing feature reconstruction processing on a masked portion of the masked video frame based on the text feature to obtain a sample reconstruction result.

[0123] Step 104 is described in detail below.

[0124] The sample reconstruction results include both visual and text reconstruction. Furthermore, a masked video frame refers to the result of masking selected initial pixel blocks after determining their spatiotemporal importance. Text features extracted from the sample text are used to assist in feature reconstruction of the masked portions, thereby leveraging cross-modal information to help the model better understand the sample video and ultimately achieve a visual reconstruction result after feature reconstruction.

[0125] Furthermore, the integration of spatiotemporal information in spatial and temporal dimensions enhances the model's ability to determine key information, enabling it to more comprehensively understand and process visual-textual information even when faced with unfamiliar input data, thereby improving the overall performance and reliability of the trained target multimodal processing model.

[0126] In some embodiments, as Figure 7 As shown, Figure 7 yes Figure 2 An optional implementation flowchart of step 104 in FIG. 100 is provided for determining a masked video frame based on spatiotemporal information, including the following steps:

[0127] (104.1.1) For each spatiotemporal information item in the same initial video frame, select the initial pixel blocks corresponding to the spatiotemporal information item within a second preset range as masked pixel blocks in a masked state, and select the initial pixel blocks corresponding to the spatiotemporal information item outside the second preset range as non-masked pixel blocks in a non-masked state;

[0128] (104.1.2) According to the initial spatial relative positions of the initial pixel blocks, the masked pixel blocks and the non-masked pixel blocks are spliced ​​to obtain a masked video frame.

[0129] The following describes steps (104.1.1) to (104.1.2) in detail.

[0130] In some embodiments, the initial pixel block is masked according to the spatiotemporal importance score to obtain the masked pixel block x inv and the non-masked pixel block x vis Specifically, for each spatiotemporal importance score obtained for the current initial video frame, they are sorted in descending order of the scores, and the first r% (within the second preset range) of the initial pixel blocks are selected and set to 0, which are invisible visual tokens (masked pixel blocks); the remaining (1-r)% of the initial pixel blocks are retained, which are visible visual tokens (non-masked pixel blocks). The specific value of can be set according to the actual situation, or adjusted in real time with the help of artificial intelligence equipment.

[0131] The initial spatial relative position refers to the relative position of each initial pixel block in the original video frame. Specifically, each initial pixel block has a fixed coordinate or index, representing its specific position in the corresponding initial video frame. It should be noted that masking the video frame does not change the content of the original video frame; it only sets the initial pixel blocks to different display states (including masked and unmasked) to facilitate the training of the initial multimodal processing model.

[0132] In some embodiments, as Figure 8 As shown, Figure 8 yes Figure 2 Another optional implementation flowchart of step 104 in FIG. 1 is to perform feature reconstruction processing on the portion in the masked video frame that is in a masked state based on the text feature to obtain a sample reconstruction result, including the following steps:

[0133] (104.2.1) Obtain adjacent masked video frames corresponding to adjacent initial video frames, and predict updated key positions of the current masked video frame based on the adjacent masked video frames;

[0134] (104.2.2) Determine update time information corresponding to each masked pixel block and non-masked pixel block in the masked video frame based on the update key position;

[0135] (104.2.3) Performing feature extraction on the non-masked pixel blocks to obtain non-masked features, and performing feature reconstruction on the masked portion of the masked video frame based on the update time information, the non-masked features, and the text features to obtain a visual reconstruction result;

[0136] (104.2.4) Performing random occlusion processing on the sample text to obtain occluded text, and performing reconstruction processing on the occluded text to obtain a text reconstruction result.

[0137] The following describes steps (104.2.1) to (104.2.4) in detail.

[0138] In some embodiments, as Figure 4 As shown, the input is into the visual encoder (Video Encoder), and Video Encoder is used to encode x vis Perform feature extraction to obtain non-masked features So that the spatiotemporal decoder can be based on non-masking features The text features and update time information are used to reconstruct the features of the masked part in the masked video.

[0139] Further, as follows <4> As shown, the spatiotemporal decoder first determines the update key position of the current masked video frame based on the adjacent masked video frames. The specific implementation method is the same as that of Formula <2> Similar, no further elaboration;

[0140]

[0141] in, Indicates the updated key position of the current masked video frame; Indicates the updated key position of adjacent masked video frames.

[0142] Furthermore, based on the update key position of the current masked video frame, the update time information corresponding to each masked pixel block and non-masked pixel block in the masked video frame is determined. Specific implementation method and formula <3> Similar, no further details.

[0143] Further, as follows <5> As shown, the text features are combined with the mask pixel block x inv Perform cross-modal enhancement and combine with non-masked features After splicing, the masked video frames are weightedly enhanced using the calculated update time information. Finally, the position code pos is introduced to distinguish the spatial position of each visual token, and feature reconstruction is completed through the multilayer perceptron (MLP) network to obtain the visual reconstruction result:

[0144]

[0145] Among them, pos inv Represents the position code of the masked pixel block, pos vis Represents the position code of the non-occluded pixel block.

[0146] Furthermore, the text tokens in the sample text are randomly masked to obtain the masked text; then, the masked text is encoded using the text encoder to obtain the text encoding features; then, the text encoding features are decoded using the text decoder (Cross-modalDecoder) to obtain the text reconstruction result.

[0147] In some embodiments, as Figure 9 As shown, Figure 9 yes Figure 8 In an optional implementation flowchart of step (104.2.1), adjacent masked video frames are determined by the following steps:

[0148] (A.1) performing spatial importance evaluation on each adjacent initial pixel block in adjacent initial video frames to obtain adjacent spatial information, and performing temporal importance evaluation on the initial pixel blocks to obtain adjacent temporal information;

[0149] (A.2) For the same adjacent initial pixel blocks, the adjacent spatial information and the adjacent temporal information are superimposed to obtain corresponding adjacent spatiotemporal information, and adjacent masked video frames are determined based on the adjacent spatiotemporal information.

[0150] The adjacent masked video frames refer to masked video frames corresponding to adjacent initial video frames. Adjacent masked video frames also include masked pixel blocks in a masked state and unmasked pixel blocks in an unmasked state. The specific method for determining adjacent masked video frames is similar to that for the current masked video frame and will not be further described here.

[0151] Step 105 , calculating a loss value based on the sample reconstruction result, and adjusting the model parameters of the initial multimodal processing model based on the loss value to obtain a trained target multimodal processing model.

[0152] Step 105 is described in detail below.

[0153] In some embodiments, a loss value is calculated based on the sample reconstruction results, and the parameters of the initial multimodal processing model are optimized and adjusted based on this loss value, allowing the model to continuously learn and improve its understanding and expression of video content by minimizing reconstruction errors. In particular, in a framework that combines spatiotemporal information for masking and uses text features for feature reconstruction, the model's ability to identify key areas in the video is improved, thereby improving the model's ability to grasp cross-modal semantic alignment. This end-to-end training method effectively improves the model's generalization and robustness in downstream tasks such as video understanding, action recognition, and multimodal retrieval, enhancing its application potential in complex scenarios.

[0154] In some embodiments, as Figure 10 As shown, Figure 10 yes Figure 2 An optional implementation flowchart of step 105 in FIG. 1 is provided for calculating the loss value based on the sample reconstruction result, including the following steps:

[0155] (105.1.1) Determine a video text matching loss value based on the masked pixel block and the corresponding preset text mask;

[0156] (105.1.2) Determine the video-text contrastive learning loss value based on text features and visual reconstruction results;

[0157] (105.1.3) Obtain the verification visual reconstruction result, and determine the visual modeling loss value based on the visual reconstruction result and the verification visual reconstruction result;

[0158] (105.1.4) Determine the text modeling loss value based on the text features and text reconstruction results.

[0159] Steps (105.1.1) to (105.1.4) are described in detail below.

[0160] In some embodiments, the loss value includes a video text matching loss value L VTM , video text contrastive learning loss value L VTC , visual modeling loss value L MVM And the text modeling loss value L MLM Among them, L VTM and L VTC The cross-modal feature alignment capability used to train the model is achieved by using L MVM Evaluate the accuracy of video feature reconstruction using L MLM To evaluate the accuracy of text reconstruction.

[0161] Furthermore, based on the formula <6> Determine the video text matching loss value, based on the formula <7> Determine the video text matching loss value, based on the formula <8> Determine the video text matching loss value, based on the formula <9> Determine the video text matching loss value:

[0162]

[0163]

[0164] Among them, f(x i ) is used to verify the visual reconstruction results, and f is the teacher model; Indicates the masking feature corresponding to the masked pixel block; y′ i It is 0 or 1, 0 represents a negative sample, 1 represents a positive sample; N represents the total number of samples; N inv Indicates the number of masked pixel blocks; N int Indicates the number of non-occluded pixel blocks.

[0165] In some embodiments, as Figure 11 As shown, Figure 11 yes Figure 2 Step 105 in the embodiment of the present invention is another optional implementation flowchart, which adjusts the model parameters of the initial multimodal processing model according to the loss value to obtain the trained target multimodal processing model, including the following steps:

[0166] (105.2.1) Adjusting the model parameters of the initial multimodal processing model based on the video-text matching loss value to obtain a trained target multimodal processing model;

[0167] (105.2.2) Alternatively, the model parameters of the initial multimodal processing model are adjusted according to the video-text contrastive learning loss value to obtain a trained target multimodal processing model;

[0168] (105.2.3) Alternatively, adjusting model parameters of the initial multimodal processing model according to the visual modeling loss value to obtain a trained target multimodal processing model;

[0169] (105.2.4) Alternatively, the model parameters of the initial multimodal processing model are adjusted according to the text modeling loss value to obtain a trained target multimodal processing model;

[0170] (105.2.5) Alternatively, the model parameters of the initial multimodal processing model are jointly adjusted based on the video-text matching loss value, the video-text contrastive learning loss value, the visual modeling loss value, and the text modeling loss value to obtain the trained target multimodal processing model.

[0171] Steps (105.2.1) to (105.2.5) are described in detail below.

[0172] In some embodiments, the video-to-text matching loss is used to measure the similarity or difference between video features and corresponding text descriptions. The goal is to enable the model to learn to accurately align video content with the corresponding text description. The video-to-text matching loss is suitable for tasks that require precise matching of video and text, such as video subtitle generation and video retrieval. By minimizing the video-to-text matching loss, the model can establish a closer connection between video and text, improving cross-modal understanding capabilities.

[0173] In some embodiments, the video-text contrastive learning loss is designed to enhance the model's discriminative ability by using positive pairs (correct video-text pairs) and negative pairs (incorrect video-text pairs). This encourages the model to assign higher similarity scores to positive pairs and lower scores to negative pairs. This helps improve the model's generalization capabilities on large datasets, especially in the presence of large amounts of unlabeled data, helping the model better understand and distinguish subtle differences between different modalities, thereby improving the accuracy of classification and retrieval tasks.

[0174] In some embodiments, the visual modeling loss focuses on optimizing the model's understanding and representation of visual information within video frames, such as spatial feature extraction and action recognition. This loss enhances the model's ability to analyze video content, enabling it to more accurately capture salient areas and dynamic changes in the video.

[0175] In some embodiments, the text modeling loss focuses on optimizing the model's ability to understand and encode text descriptions. By optimizing the text modeling loss, the model can better understand and represent textual information, which is crucial for tasks requiring precise text descriptions, such as video subtitle generation and question-answering systems. Furthermore, good text reconstruction can enhance the model's cross-modal understanding capabilities.

[0176] Furthermore, the embodiment of the present application can use only one or more of the four loss values ​​for training; or, based on the following formula <10> Determine the total loss value L and use the total loss value to jointly train the model:

[0177] L=L VTC +L VTM +L MVM +L MLM <10>

[0178] The adjusted model parameters include, but are not limited to, convolutional layer parameters (e.g., weights and biases), fully connected layer parameters (e.g., weight matrices and bias vectors), normalization layer parameters (e.g., scaling factors and offset factors), and attention mechanism parameters. It should be noted that the adjusted model parameters can be set based on actual conditions, and this embodiment of the application does not impose any restrictions on this.

[0179] Furthermore, when the preset training conditions are met, the trained target multimodal processing model is obtained. The preset training conditions may be that the number of training rounds reaches a preset value, or the training time reaches a preset value, or the loss value reaches a desired range. The preset training conditions can be set according to actual conditions, and the embodiments of the present application do not limit this.

[0180] In order to better illustrate the beneficial effects of the embodiments of the present application, two types of experimental results are given below:

[0181] (1) Visual experiment results

[0182] like Figure 12 As shown, Figure 12This is an optional visual experimental result diagram of the training method of the visual text pre-training model provided in the embodiment of the present application. For each initial video frame (Frames) in a given sample video, as the training rounds increase (from the initialization epoch to the last epoch), it can be seen that the method (spatiotemporal mask) adopted in the embodiment of the present application is better than the traditional method (temporal masking) in generating masked video frames that block important action areas (the action currently being performed by the little girl), thereby forcing the model to focus on and understand important visual parts. It is understandable that the target multimodal processing model obtained by training with the masked video frames generated by the method proposed in the embodiment of the present application will have a stronger ability to understand multimodal information in actual use.

[0183] (2) Data Experiment Results

[0184] As shown in Table 1 below, Table 1 is a schematic diagram of the results of an optional data experiment for the training method of the visual text pre-training model provided in the embodiment of this application. The embodiment of this application also conducted experiments on video-text retrieval on three public datasets. It can be seen that compared with traditional methods, the performance of STR(Ours) in the embodiment of this application on all three public datasets is significantly improved. In addition, compared with models trained on more data, the performance of the proposed method is equally significant. Among them, the larger the value, the better the effect.

[0185] Table 1

[0186]

[0187] The experiments involved three public datasets: the Microsoft Research Video-to-Text Dataset (MSRVTT), the Descriptive Video Clip Dataset (Didemo), and the ActivityNet Dataset. R1, R5, and R10 are metrics representing the top-1, top-5, and top-10 accuracy, respectively.

[0188] The traditional methods and models involved in the experiment include: Clipped Bidirectional Encoder Representations from Transformers (ClipBERT), Frozen Model (Frozen), Alignment and Representation Learning for Pre-training and Fine-tuning (ALPRO), Bridge Transformer (BridgeFormer), Singularity Model (Singularity), Language And Vision Embedding Deep Neural Network (LAVENDER), Video Image Object Language Embedding v2 (VIOLETv2), High Temporal Alignment Attention Model (HiTeA), Video Indicating Natural Language Understanding (VINDLU), Unified Multimodal Transformer (UMT), Highly Efficient Retrieval Optimization Model (Highly Efficient Retrieval Optimization, HERO), Video Image Object Language Embedding (VIOLET), All-in-One Model, Clip-to-Clip Model (Clip4Clip), Extended Contrastive Learning Image Pretraining (X-CLIP).

[0189] like Figure 13 As shown, Figure 13 201 to 205:

[0190] An acquisition module 201 is used to acquire a sample video and a sample text for describing the sample video;

[0191] The video text processing module 202 is used to input the sample video and sample text into the initial multimodal processing model, segment the sample video to obtain multiple initial video frames, and extract text features from the sample text;

[0192] The spatiotemporal information determination module 203 is configured to perform a spatial importance assessment on each initial pixel block in each initial video frame to obtain spatial information, and perform a temporal importance assessment on the initial pixel block to obtain temporal information. For the same initial pixel block, the spatial information and temporal information are superimposed to obtain corresponding spatiotemporal information.

[0193] The sample reconstruction module 204 is configured to determine the masked video frame based on the spatiotemporal information, and perform feature reconstruction processing on the masked portion of the masked video frame based on the text features to obtain a sample reconstruction result.

[0194] The target training module 205 is used to calculate the loss value according to the sample reconstruction result, and adjust the model parameters of the initial multimodal processing model according to the loss value to obtain the trained target multimodal processing model.

[0195] The training method, device, equipment and medium of the visual text pre-training model proposed in this application obtain sample videos and sample texts used to describe the sample videos; input the sample videos and sample texts into the initial multimodal processing model, segment the sample videos to obtain multiple initial video frames, and extract text features from the sample texts; perform spatial importance evaluation on each initial pixel block in each initial video frame to obtain spatial information, and perform temporal importance evaluation on the initial pixel block to obtain temporal information; for the same initial pixel block, superimpose the spatial information and temporal information to obtain the corresponding spatiotemporal information; and then, use the spatial importance evaluation and temporal importance evaluation to obtain the corresponding spatiotemporal information. The spatiotemporal information obtained under the comprehensive evaluation of the quality assessment guides the initial multimodal processing module to better understand the background and important object features in the sample video; then, the masked video frame is determined based on the spatiotemporal information, and the feature reconstruction processing is performed on the masked part of the masked video frame based on the text features to obtain the sample reconstruction result. This process greatly enhances the model's ability to understand the sample video content, thereby ensuring the consistency of the masking strategy and the reconstruction target during the model training process; then, the loss value is calculated according to the sample reconstruction result, and the model parameters of the initial multimodal processing model are adjusted according to the loss value to obtain the trained target multimodal processing model. The present application can improve the trained target multimodal processing model's ability to understand multimodal information, especially the video text information.

[0196] The specific implementation of the training device of the visual text pre-training model is basically the same as the specific embodiment of the training method of the above-mentioned visual text pre-training model, and will not be repeated here.

[0197] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described training method for the visual text pre-training model. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.

[0198] like Figure 14 As shown, Figure 14 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0199] The processor 301 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0200] The memory 302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302, and the processor 301 calls and executes the training method of the visual text pre-training model of the embodiments of this application;

[0201] Input / output interface 303, used to implement information input and output;

[0202] Communication interface 304, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0203] bus 305 , which transmits information between the various components of the device (e.g., processor 301 , memory 302 , input / output interface 303 , and communication interface 304 );

[0204] The processor 301 , the memory 302 , the input / output interface 303 and the communication interface 304 are connected to each other in communication within the device via the bus 305 .

[0205] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the training method of the above-mentioned visual text pre-training model.

[0206] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0207] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0208] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0209] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0210] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0211] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0212] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0216] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0217] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A training method for a visual text pre-training model, characterized in that: include: Obtaining a sample video and a sample text for describing the sample video; Inputting the sample video and the sample text into an initial multimodal processing model, segmenting the sample video to obtain a plurality of initial video frames, and extracting text features from the sample text; Performing a spatial importance evaluation on each initial pixel block in each of the initial video frames to obtain spatial information, and performing a temporal importance evaluation on the initial pixel block to obtain temporal information, and for the same initial pixel block, superimposing the spatial information and the temporal information to obtain corresponding spatiotemporal information; Determine a masked video frame based on the spatiotemporal information, and perform feature reconstruction processing on a masked portion of the masked video frame based on the text features to obtain a sample reconstruction result; A loss value is calculated according to the sample reconstruction result, and model parameters of the initial multimodal processing model are adjusted according to the loss value to obtain a trained target multimodal processing model.

2. The training method of the visual text pre-training model according to claim 1, characterized in that The initial video frame includes a plurality of initial pixel blocks; The performing spatial importance evaluation on each initial pixel block in each initial video frame to obtain spatial information includes: Performing linear processing on each of the initial pixel blocks in the current initial video frame to obtain a corresponding embedded pixel block; Obtaining global identification information of the sample video; Mapping the global identification information to the current initial video frame to obtain current identification information corresponding to the current initial video frame; A dot product operation is performed on the current identification information and the embedded pixel block to obtain a dot product result, and the dot product result is normalized based on the feature dimension of the initial pixel block to obtain the corresponding spatial information.

3. The training method of the visual text pre-training model according to claim 1, characterized in that The determining of the masked video frame based on the spatiotemporal information includes: For each of the spatiotemporal information in the same initial video frame, the initial pixel blocks corresponding to the spatiotemporal information within a second preset range are selected as masked pixel blocks in a masked state, and the initial pixel blocks corresponding to the spatiotemporal information not within the second preset range are selected as non-masked pixel blocks in a non-masked state; The masked pixel blocks and the non-masked pixel blocks are spliced ​​together according to the initial spatial relative positions of the initial pixel blocks to obtain the masked video frame.

4. The training method of the visual text pre-training model according to claim 3, characterized in that: The performing time importance evaluation on the initial pixel block to obtain time information includes: Obtaining the adjacent initial video frames of the current initial video frame, and predicting the initial key position of the current initial video frame based on the adjacent initial video frames; For each of the initial pixel blocks in the current initial video frame, if the distance between any of the initial pixel blocks and the initial key position is within a first preset range, determining the corresponding time information according to a preset weight; If the distance between any of the initial pixel blocks and the initial key position is not within a first preset range, the corresponding time information is determined according to a preset decay rate.

5. The training method of the visual text pre-training model according to claim 4, characterized in that: The sample reconstruction result includes a visual reconstruction result and a text reconstruction result; The performing feature reconstruction processing on the portion in the masked state in the masked video frame based on the text feature to obtain a sample reconstruction result includes: Acquire adjacent masked video frames corresponding to the adjacent initial video frames, and predict an updated key position of the current masked video frame based on the adjacent masked video frames; Determining update time information corresponding to each of the masked pixel blocks and the non-masked pixel blocks in the masked video frame based on the update key position; performing feature extraction processing on the non-masked pixel block to obtain a non-masked feature, and performing feature reconstruction processing on a portion of the masked video frame that is in a masked state according to the update time information, the non-masked feature, and the text feature to obtain the visual reconstruction result; The sample text is randomly occluded to obtain occluded text, and the occluded text is reconstructed to obtain the text reconstruction result.

6. The training method of the visual text pre-training model according to claim 5, characterized in that: The adjacent masked video frames are determined by the following steps, which include: Performing spatial importance evaluation on each adjacent initial pixel block in the adjacent initial video frame to obtain adjacent spatial information, and performing temporal importance evaluation on the initial pixel block to obtain adjacent temporal information; For the same adjacent initial pixel blocks, the adjacent spatial information and the adjacent temporal information are superimposed to obtain corresponding adjacent spatiotemporal information, and the adjacent masked video frames are determined based on the adjacent spatiotemporal information.

7. The training method of the visual text pre-training model according to claim 5, characterized in that: The loss values ​​include video-text matching loss value, video-text contrastive learning loss value, visual modeling loss value and text modeling loss value; The calculating the loss value according to the sample reconstruction result includes: Determining the video text matching loss value based on the masked pixel block and the corresponding preset text mask; Determining the video-text contrastive learning loss value based on the text features and the visual reconstruction result; Obtaining a verification visual reconstruction result, and determining the visual modeling loss value based on the visual reconstruction result and the verification visual reconstruction result; The text modeling loss value is determined based on the text features and the text reconstruction result.

8. The training method of the visual text pre-training model according to claim 7, characterized in that: The adjusting the model parameters of the initial multimodal processing model according to the loss value to obtain a trained target multimodal processing model includes: Adjusting the model parameters of the initial multimodal processing model according to the video-text matching loss value to obtain a trained target multimodal processing model; or, Adjusting the model parameters of the initial multimodal processing model according to the video-text contrastive learning loss value to obtain a trained target multimodal processing model; or, Adjusting the model parameters of the initial multimodal processing model according to the visual modeling loss value to obtain a trained target multimodal processing model; or, Adjusting the model parameters of the initial multimodal processing model according to the text modeling loss value to obtain a trained target multimodal processing model; or According to the video-text matching loss value, the video-text comparative learning loss value, the visual modeling loss value and the text modeling loss value, the model parameters of the initial multimodal processing model are jointly adjusted to obtain a trained target multimodal processing model.

9. A training device for a visual text pre-training model, characterized in that: include: An acquisition module, used to acquire a sample video and a sample text for describing the sample video; a video-text processing module, configured to input the sample video and the sample text into an initial multimodal processing model, segment the sample video to obtain a plurality of initial video frames, and extract text features from the sample text; a spatiotemporal information determination module, configured to perform a spatial importance assessment on each initial pixel block in each of the initial video frames to obtain spatial information, and perform a temporal importance assessment on the initial pixel block to obtain temporal information, and for the same initial pixel block, superimpose the spatial information and the temporal information to obtain corresponding spatiotemporal information; a sample reconstruction module, configured to determine a masked video frame based on the spatiotemporal information, and perform feature reconstruction processing on a masked portion of the masked video frame based on the text features to obtain a sample reconstruction result; The target training module is used to calculate the loss value according to the sample reconstruction result, and adjust the model parameters of the initial multimodal processing model according to the loss value to obtain the trained target multimodal processing model.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the visual text pre-training model according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the training method of the visual text pre-training model according to any one of claims 1 to 8 is implemented.

Images