Video understanding method and device, model training method and device, computer equipment and storage medium

By introducing binary classification and comparison loss data into the video understanding model, combining encoding network and sorting processing components, the problem of insufficient understanding of image frame order and hierarchy in video content understanding is solved, and the integrity of video file content is improved and accuracy.

CN119068378APending Publication Date: 2024-12-03BEIJING INSTITUTE FOR GENERAL ARTIFICIAL INTELLIGENCE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310632623.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the existing video understanding technology, the integrity and integrity of video content are difficult to improve in the process of visual recognition and classification, especially the lack of understanding of image frame sequence and video file hierarchy.

Method used

A video understanding model training method is proposed. The target video understanding model is trained through binary classification loss data to predict the frame sequence ability of image text pairs, and the model is trained by comparing the loss data to predict whether image text pairs belong to the same unit segment, combining the encoding network and Sorting processing components to improve the integrity of video file content understanding.

Benefits of technology

It realizes an understanding of the overall and timing structure of the video file, improves the completeness and accuracy of the understanding of video content, and can perceive and understand the narrative structure of the image frame sequence in the video file.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068378A_ABST
    Figure CN119068378A_ABST
Patent Text Reader

Abstract

The invention discloses a video understanding and model training method and device, computer equipment and a storage medium, and the method comprises the steps: training a target video understanding model through binary classification loss data in a training process of the target video understanding model, and predicting the capability of any two image texts for a frame sequence; a target video understanding model is trained by comparing loss data to predict the ability of whether any two image text pairs belong to the same unit segment, and after the target video understanding model is trained, multiple image text pairs corresponding to a to-be-understood video file are input into the target video understanding model for content understanding. Obtaining the video content of the to-be-understood video file; through the target video understanding model, the video file can be understood on the whole, the narrative structure of the image frame sequence can be understood on the time sequence structure, and the integrity of understanding the content of the video file is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video understanding technology, and in particular to a video understanding, model training method, device, computer equipment and storage medium. The present invention can be used in video search, smart home, online education and other fields. Background Art

[0002] Video understanding is a topic that has attracted much attention from both academia and industry in the current field of computer vision research. With the rapid development of the video industry, how to use artificial intelligence technology to better understand video content is becoming increasingly important.

[0003] In the related art, the task of understanding the content of a video file is regarded as a visual recognition and classification task (Visual Recognition and Classifications).

[0004] However, the integrity of video content obtained by visual recognition and classification in related technologies needs to be improved. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present invention proposes a video understanding, model training method, device, computer equipment and storage medium.

[0006] The present invention provides a video understanding method, the method comprising:

[0007] Acquire a plurality of image-text pairs corresponding to the video file to be understood; wherein the image-text pairs include image frames and texts corresponding to the image frames;

[0008] The multiple image-text pairs are input into a target video understanding model for content understanding to obtain the video content of the video file to be understood; wherein the loss data of the target video understanding model during the training process includes binary classification loss data and contrast loss data, the binary classification loss data is used to train the target video understanding model's ability to predict the frame order of any two image-text pairs, and the contrast loss data is used to train the target video understanding model's ability to predict whether the any two image-text pairs belong to the same unit segment.

[0009] In one embodiment, the arbitrary two image-text pairs include a first image-text pair and a second image-text pair; and the training method of the target video understanding model includes:

[0010] Inputting the first image-text pair and the second image-text pair into an initial video understanding model, determining predicted probability distribution data of a frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment;

[0011] Determine the binary classification loss data based on the true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data;

[0012] Determine the contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments;

[0013] The parameters of the initial video understanding model are updated based on the binary classification loss data and the contrast loss data to obtain the target video understanding model.

[0014] In one embodiment, the initial video understanding model includes an encoding network and a sorting processing component; the method for determining the predicted probability distribution data includes:

[0015] Performing feature extraction and feature fusion on the image frame and the text in the first image-text pair through the encoding network to obtain a first fusion feature of the first image-text pair;

[0016] Performing feature extraction and feature fusion on the image frame and the text in the second image-text pair through the encoding network to obtain a second fused feature of the second image-text pair;

[0017] The first fusion feature and the second fusion feature are input into the sorting processing component for binary classification processing to obtain the predicted probability distribution data.

[0018] In one embodiment, the initial video understanding model further includes a clustering processing component; and the method for determining the first feature distribution data includes:

[0019] The first fused feature of the first image-text pair and the second fused feature of the second image-text pair are input into the clustering processing component for contrast loss learning to obtain first feature distribution data between the first image-text pair and the second image-text pair.

[0020] In one embodiment, the method for determining the second characteristic distribution data includes:

[0021] determining a third fusion feature of a third image-text pair;

[0022] The second fused feature and the third fused feature are input into the cluster processing component to perform contrast loss learning, so as to obtain second feature distribution data between the second image-text pair and the third image-text pair.

[0023] In one embodiment, the encoding network includes a first encoding network and a second encoding network; and the first fusion feature and the second fusion feature are determined by:

[0024] Extracting features from the image frame and the text in the first image-text pair through the first encoding network to obtain first image features and first text features;

[0025] Extracting features from the image frame and the text in the second image-text pair through the first encoding network to obtain second image features and second text features;

[0026] Inputting the first image feature, the first text feature, the second image feature, and the second text feature into the second encoding network for cross-fusion and feature extraction, to obtain a correlation feature classification head of the first image, a correlation feature classification head of the first text, a correlation feature classification head of the second image, and a correlation feature classification head of the second text;

[0027] Merging the associated feature classification head of the first image and the associated feature classification head of the first text to obtain the first fusion feature;

[0028] The second fusion feature is obtained by merging the associated feature classification head of the second image and the associated feature classification head of the second text.

[0029] In one embodiment, the first encoding network includes an image encoding branch and a text encoding branch obtained by joint training; the first encoding network extracts features of the image frame and the text in the first image-text pair to obtain first image features and first text features, including:

[0030] Extracting features from the image frames in the first image-text pair by using the image encoding branch to obtain the first image features;

[0031] The text in the first image-text pair is subjected to feature extraction by the text encoding branch to obtain the first text feature.

[0032] In one embodiment, the unit segment is a scene; the first image-text pair and the second image-text pair belong to the same scene; and the method for determining the contrast loss data includes:

[0033] The scene contrast loss data is determined based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different scenes.

[0034] In one embodiment, the unit segment is a storyboard; the first image-text pair and the second image-text pair belong to the same storyboard; and the method for determining the contrast loss data includes:

[0035] The mirror contrast loss data is determined based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different mirrors.

[0036] In one embodiment, the unit segmentation is a scene and a storyboard; the model training process includes a first training stage and a second training stage, and the first training stage is earlier than the second training stage; the method for determining the contrast loss data includes:

[0037] In the first training stage, determining the frame contrast loss data based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair; wherein the first image-text pair and the second image-text pair belong to the same frame, and the second image-text pair and the third image-text pair belong to different frames;

[0038] In the second training stage, scene contrast loss data is determined based on the third feature distribution data between the fourth image-text pair and the fifth image-text pair, and the fourth feature distribution data between the fifth image-text pair and the sixth image-text pair; wherein the fourth image-text pair and the fifth image-text pair belong to the same scene; and the fifth image-text pair and the sixth image-text pair belong to different scenes.

[0039] The present invention provides a model training method, the method comprising:

[0040] Inputting a first image-text pair and a second image-text pair into an initial video understanding model, determining predicted probability distribution data of a frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment;

[0041] Determine binary classification loss data based on true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data;

[0042] Determine contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments;

[0043] The parameters of the initial video understanding model are updated based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

[0044] The present invention provides a video understanding device, the device comprising:

[0045] An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to the video file to be understood; wherein the image-text pairs include image frames and texts corresponding to the image frames;

[0046] A video file understanding module is used to input the multiple image-text pairs into a target video understanding model for content understanding to obtain the video content of the video file to be understood; wherein the loss data of the target video understanding model during the training process includes binary classification loss data and contrast loss data, the binary classification loss data is used to train the target video understanding model's ability to predict the frame order of any two image-text pairs, and the contrast loss data is used to train the target video understanding model's ability to predict whether the any two image-text pairs belong to the same unit segment.

[0047] The present invention provides a model training device, comprising:

[0048] An image-text pair input module, used to input a first image-text pair and a second image-text pair into an initial video understanding model, and determine predicted probability distribution data of a frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment;

[0049] A classification loss determination module, configured to determine binary classification loss data based on the true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data;

[0050] a contrast loss determination module, configured to determine contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments;

[0051] A model parameter updating module is used to update the parameters of the initial video understanding model based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

[0052] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0053] The present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0054] According to the above embodiment of the present invention, during the training process of the target video understanding model, the target video understanding model is trained with binary classification loss data to predict the frame order of any two image-text pairs, and is trained with contrast loss data to predict whether any two image-text pairs belong to the same unit segment. After the target video understanding model is trained, multiple image-text pairs corresponding to the video file to be understood are input into the target video understanding model for content understanding to obtain the video content of the video file to be understood. Since the target video understanding model has the ability to predict the frame order of any two image-text pairs and the ability to predict whether any two image-text pairs belong to the same unit segment, through these two capabilities, the target video understanding model can not only understand the video file as a whole, but also understand the narrative structure of the image frame sequence in terms of temporal structure, thereby improving the completeness of the understanding of the video file content.

[0055] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A flowchart of a video understanding method provided according to an embodiment of the specification;

[0057] Figure 2 A flowchart of a method for training a target video understanding model provided according to an implementation manner of the specification;

[0058] Figure 3a A framework diagram of an initial video understanding model provided according to an embodiment of the specification;

[0059] Figure 3b A flowchart of a method for determining predicted probability distribution data according to an embodiment of the specification;

[0060] Figure 4 A framework diagram of an initial video understanding model provided according to another embodiment of the specification;

[0061] Figure 5 A framework diagram of a method for determining second characteristic distribution data provided according to an embodiment of the specification;

[0062] Figure 6a A framework diagram of an initial video understanding model provided according to another embodiment of the specification;

[0063] Figure 6b A flowchart of a target video understanding model training method provided according to an implementation method of the specification;

[0064] Figure 7a A flowchart of a target video understanding model training method provided according to an implementation method of the specification;

[0065] Figure 7b A framework diagram of an initial video understanding model provided according to another embodiment of the specification;

[0066] Figure 8 A flowchart of a method for determining comparative loss data according to an embodiment of the specification;

[0067] Fig. 9 A flowchart of a model training method provided according to an embodiment of the specification;

[0068] Fig.10 A framework diagram of a video understanding device provided according to an embodiment of the specification;

[0069] Fig.11 A framework diagram of a model training device provided according to an implementation method of the specification. DETAILED DESCRIPTION

[0070] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0071] Video understanding is an artificial intelligence technology that uses deep neural networks to automatically parse and understand video content. By analyzing and learning videos, video understanding can identify information such as objects, actions, scenes, and emotions in videos, thereby improving the accuracy and efficiency of applications such as video retrieval, classification, labeling, and generation. Video features are automatically learned and extracted through deep neural network models composed of multiple layers of neurons. These models can learn the characteristics and laws of videos through a large amount of training data and apply them to video understanding tasks. Video understanding has a wide range of applications, including video classification, object detection, behavior recognition, sentiment analysis, video generation, etc. For example, deep learning video understanding can be used to implement applications such as intelligent monitoring, autonomous driving, and video recommendation, improving people's production and living efficiency.

[0072] People and even young children can understand and reason about videos through the visual and auditory information included in the video files, and form corresponding visual narratives. In the related art, the video language model (Video-LanguageModels) mainly understands videos based on dense features extracted from vision and language. However, after analysis by the inventor, it is found that, on the one hand, since each image frame in the video file can be regarded as a functional unit for constructing the narrative ability of the video file, if unreasonable exchange occurs between the image frames, the integrity and integrity of the video file may be broken. It can be seen that the order of image frames is crucial to the understanding integrity of the video file. On the other hand, the video file is divided into hierarchical structures during the production process, such as the hierarchical structure of the scene-storyboard-frame of the movie video file, the hierarchical structure of the grid-frame of the animation video file, and the hierarchical structure of the storyboard-frame of the short video file. The hierarchical structure of the video file is the basic narrative structure of the video file, and it can be seen that the hierarchical structure of the video file is closely related to the understanding of the content of the video file.

[0073] Therefore, unlike the visual recognition and classification tasks that are mainly concerned in the related art, the inventor proposes to split the training task of the video understanding model into: the task of predicting the order of any two image frames and the task of predicting whether any two image frames belong to the same unit segment (such as scene, storyboard). Therefore, the video understanding model in the implementation of this specification has the ability to understand the hierarchical structure of video files, and can perceive and understand the narrative structure of image frame sequences, realize the overall understanding of video files, and realize the visual narrative reasoning ability based on video files.

[0074] Based on this, an embodiment of the present specification provides a video understanding method, by obtaining multiple image-text pairs corresponding to a video file to be understood; wherein the image-text pairs include image frames and texts corresponding to the image frames; and inputting the multiple image-text pairs into a target video understanding model for content understanding to obtain the video content of the video file to be understood; wherein the loss data of the target video understanding model during the training process includes binary classification loss data and contrast loss data, the binary classification loss data is used to train the target video understanding model's ability to predict the frame order of any two image-text pairs, and the contrast loss data is used to train the target video understanding model's ability to predict whether any two image-text pairs belong to the same unit segment.

[0075] The target video understanding model in the above video understanding method has the ability to predict the frame order of any two image-text pairs and the ability to predict whether any two image-text pairs belong to the same unit segment. Through these two capabilities, the target video understanding model can not only understand the video file as a whole, but also understand the narrative structure of the image frame sequence in terms of temporal structure, thereby improving the completeness of the understanding of the content of the video file.

[0076] Furthermore, the target video understanding model can be directly deployed on the client or server for use, or it can be applied to downstream rich video understanding tasks after small sample learning, such as Video Question Answering (VideoQA), Video-GroundedDialogueGenerations, and Fine-GrainedActionPredictions.

[0077] The video understanding method provided in the embodiments of this specification can be applied to the client or server of the video understanding application system. The client can be an electronic device with network access capability. Specifically, for example, the client can be a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a smart wearable device, a shopping guide terminal, a television, a smart speaker, a microphone, etc. Among them, smart wearable devices include but are not limited to smart bracelets, smart watches, smart glasses, smart helmets, smart necklaces, etc. Alternatively, the client can also be software that can run in an electronic device. The server can be an electronic device with certain computing and processing capabilities. It can have a network communication module, a processor, and a memory, etc. Of course, the server can also refer to software running in an electronic device. The server can also be a distributed server, which can be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the embodiments of the specification. For example, it can be a new form of "server" based on quantum computing.

[0078] This specification provides a video understanding method. Figure 1 , the video understanding method may include the following steps:

[0079] S110: Acquire multiple image-text pairs corresponding to the video file to be understood.

[0080] Among them, the image-text pair includes an image frame and the text corresponding to the image frame. The video file can be a short video file, a long video file such as a movie or TV program, or an animation video file. The video file includes a series of image frames, and the image frames correspond to text. The text can be subtitle text, glyph text, non-subtitle text obtained from the image frame using text recognition, or text features of the image frame. An image-text pair can be formed based on the image frame and the text corresponding to the image frame. It should be noted that the text corresponding to the image frame can be any text content such as dialogue, narration, character introduction, etc. In some special cases, the text corresponding to the image frame can also be blank.

[0081] In some cases, in order to improve the accuracy and comprehensiveness of video content understanding, and the video files involve image frames of image type and text data of text type, two different modal data, image frames and their corresponding texts, are jointly processed in the video understanding process.

[0082] Specifically, in some embodiments, when it is necessary to understand the content of a video file, extraction can be performed based on the video file to obtain multiple image-text pairs corresponding to the video file. In other embodiments, the video file can be extracted in advance to obtain multiple image-text pairs corresponding to the video file and store them. When it is necessary to understand the content of the video file, the multiple image-text pairs corresponding to the video file can be directly obtained.

[0083] S120: Input multiple image-text pairs into a target video understanding model to perform content understanding, and obtain video content of the video file to be understood.

[0084] Among them, the loss data of the target video understanding model in the training process includes binary classification loss data and contrast loss data. The binary classification loss data is used to train the target video understanding model to predict the frame order of any two image text pairs. The contrast loss data is used to train the target video understanding model to predict whether any two image text pairs belong to the same unit segment. The frame order between any two image text pairs is certain, that is, any two image text pairs have true labels. The binary classification loss data can be the error data between the model prediction result and the true label calculated by the binary classification loss function (such as the cross entropy loss function and the Hinge loss function), and the model parameters are updated by back propagation so that the target video understanding model can predict the frame order between any two image text pairs. The role of the contrast loss data is to minimize the distance between two image text pairs from the same unit segment and maximize the distance between two image text pairs from different unit segments. The model parameters are adjusted by contrast loss data so that the target video understanding model can predict whether any two image text pairs belong to the same unit segment.

[0085] Among them, the unit segmentation can be a division unit of a video file, and the unit segmentation can be a video that can be used to describe a simple or brief plot. The unit segmentation can be a scene, a storyboard, or a grid of an animation video. Specifically, during the production process of a video file, it will be split or decomposed according to the background, location, etc. of the time, and multiple elements of the same level or different levels will be obtained to determine the scene, character action, and shot switching of each shot. Exemplarily, the video file can be a movie video file, and the video segmentation can be any of the important elements such as scenes and storyboards in the film production process. The scene can be the place where an event occurs in a movie, which can be indoors or outdoors, a real place, or a fictional place. The storyboard can be a continuous sequence of pictures obtained through a shot in the production of a movie video file, and the production of the storyboard can consider factors such as scripts, storyboards, scene design, and character design. Exemplarily, the video file can be an animation video file, and the video segmentation can be a storyboard or storyboard in the animation production process, which can also be called a grid. In an animation video file, the video segmentation needs to be determined by considering the content in the script and storyboard.

[0086] Among them, the target video understanding model can be a model that uses deep learning technology to analyze and understand video files. It should be noted that the target video understanding model in this embodiment is different from the video understanding model that uses visual recognition and classification in the related art. Considering the basic hierarchical structure and visual semantic order of the video file, the target video understanding model can be a video understanding model for visual narrative reasoning and overall movie understanding. Exemplarily, the first image-text pair can be an image-subtitle pair, and the target video understanding model may include a text encoding network (Language Encoder) for encoding subtitles, an image encoding network (Vision Encoder) for encoding image frames, and an image-text joint encoder (Video-Dialogue Transformer Encoder) based on a cross-attention mechanism.

[0087] Specifically, during the training process of the target video understanding model, the target video understanding model is trained using paired and contrastive learning methods to predict the correct order between multiple image-text pairs and the hierarchical structure of the video file. On the one hand, the model parameters are updated by backpropagation of binary classification loss data, so that the target video understanding model has the ability to predict whether any two image-text pairs belong to the same unit segment. On the other hand, the model parameters are adjusted using contrastive loss data to enable the target video understanding model to have the ability to predict whether any two image-text pairs belong to the same unit segment. Ultimately, the target video understanding model has the ability to decode the visual narrative structure of the movie and handle potential confusion in video data. After completing the model training, multiple image-text pairs corresponding to the video file to be understood can be input into the target video understanding model for feature extraction, and content understanding is performed based on the extracted features to obtain the video content of the video file to be understood.

[0088] It should be noted that the hierarchical structure can be determined based on the unit segmentation divided by the production process of the video file. For example, when the video file is produced, it is divided into multiple scenes, and the hierarchical structure is scene-image frame (Scene+rame). For example, when the video file is produced, it is divided into multiple storyboards, and the hierarchical structure is storyboard-image frame (Shot+Frame). For example, when the video file is produced, it is divided into multiple scenes, and each scene is divided into multiple storyboards, and the hierarchical structure is scene-storyboard-image frame (Scene+Shot+Frame).

[0089] In the above-mentioned video understanding method, during the training process of the target video understanding model, the target video understanding model is trained with binary classification loss data to predict the frame order of any two image-text pairs, and the target video understanding model is trained with contrast loss data to predict whether any two image-text pairs belong to the same unit segment. After the target video understanding model is trained, multiple image-text pairs corresponding to the video file to be understood are input into the target video understanding model for content understanding to obtain the video content of the video file to be understood. Since the target video understanding model has the ability to predict the frame order of any two image-text pairs and the ability to predict whether any two image-text pairs belong to the same unit segment, through these two capabilities, the target video understanding model can not only understand the video file as a whole, but also understand the narrative structure of the image frame sequence in terms of temporal structure, thereby improving the completeness of the understanding of the video file content.

[0090] In some implementations, any two image-text pairs include a first image-text pair and a second image-text pair. Figure 2 , the training method of the target video understanding model can include the following steps:

[0091] S210. Input the first image-text pair and the second image-text pair into an initial video understanding model to determine predicted probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair.

[0092] Among them, the first image-text pair and the second image-text pair belong to the same unit segment. In this embodiment, the training process of the initial video understanding model includes two training tasks: the ability to predict the frame order of any two image-text pairs and the ability to predict whether any two image-text pairs belong to the same unit segment. Therefore, in order to optimize and adjust the parameters of the initial video understanding model, a suitable loss function can be selected to improve the performance and accuracy of the model.

[0093] Specifically, the initial video understanding model is trained using any two image-text pairs from the same unit segment. First, any two image-text pairs including a first image-text pair and a second image-text pair are input into the initial video understanding model for feature extraction to obtain image-text features of the first image-text pair and image-text features of the second image-text pair. Frame sequence prediction is performed based on the image-text features of the first image-text pair and the image-text features of the second image-text pair to obtain predicted probability distribution data of the frame sequence between the first image-text pair and the second image-text pair. Distance distribution data (such as Euclidean distance) between the image-text features of the first image-text pair and the image-text features of the second image-text pair is extracted to obtain first feature distribution data between the first image-text pair and the second image-text pair.

[0094] S220, determining binary classification loss data based on the true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data.

[0095] The frame sequence between the first image-text pair and the second image-text pair corresponds to a real label, and the real label corresponds to real probability distribution data.

[0096] Specifically, in order to improve the frame order arrangement ability of the initial video understanding model, the gap between the prediction results of the initial video understanding model and the true label is measured by the cross entropy loss function. Therefore, after converting the frame order between the first image-text pair and the second image-text pair into a binary classification problem, the true probability distribution data and the predicted probability distribution data of the frame order between the first image-text pair and the second image-text pair are substituted into the cross entropy loss function to obtain the binary classification loss data, so as to minimize the gap between the model prediction results and the true label, and help the model learn better classification rules.

[0097] Exemplarily, the binary classification loss data is calculated using the following formula:

[0098]

[0099] Where Pi is the first image-text pair; Pj is the second image-text pair; L cls (Pi, Pj) is the binary classification loss data; CrossEntropy is the cross entropy loss function; is the predicted probability distribution data, O∈{0,1} is the real probability distribution data, which is used to represent the real sorting categories of Pi and Pj.

[0100] S230 , determining contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair.

[0101] Among them, the second image-text pair and the third image-text pair belong to different unit segments. The first feature distribution data can reflect the abstract features of the similarity between the first image-text pair and the second image-text pair belonging to the same unit segment, and the second feature distribution data can reflect the abstract features of the similarity between the second image-text pair and the third image-text pair belonging to different unit segments. During the model training process, the contrast loss function calculates the distance between the first image-text pair and the second image-text pair of the same unit segment and minimizes it, and also calculates the distance between the first image-text pair and the third image-text pair of different unit segments and maximizes it. Specifically, the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair are substituted into the contrast loss function to obtain the contrast loss data.

[0102] Exemplarily, the contrast loss data is calculated using the following formula:

[0103]

[0104] Where Pi is the first image-text pair; Pj is the second image-text pair; is the third image-text pair; To compare the loss data; is the first characteristic distribution data, is the second feature distribution data; n is the number of negative samples in the negative sample set. It can be understood that is one of the negative examples.

[0105] It should be noted that the feature distribution data between the first image-text pair and the third image-text pair may also be determined, and the contrast loss data may be determined using the feature distribution data between the first image-text pair and the third image-text pair and the first feature distribution data between the first image-text pair and the second image-text pair, wherein the first image-text pair and the third image-text pair belong to different unit segments.

[0106] S240, updating the parameters of the initial video understanding model based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

[0107] Specifically, as mentioned above, the model parameters are adjusted through binary classification loss data so that the target video understanding model can predict the frame order between any two image text pairs, and the model parameters are adjusted through contrast loss data so that the target video understanding model can predict whether any two image text pairs belong to the same unit segment. After the parameters are updated, a new round of model training is started until the model stop training conditions are met, and the target video understanding model is obtained.

[0108] In the above implementation, the initial video understanding model is trained using the first image-text pair and the second image-text pair, and the binary classification loss data is determined based on the real probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data, and the contrast loss data is determined based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair, and the model is trained to predict the frame sequence between any two image-text pairs and to predict whether any two image-text pairs belong to the same unit segment. This enables the target video understanding model to understand the video file as a whole and the narrative structure of the image frame sequence in terms of the temporal structure, thereby improving the completeness of the understanding of the video file content.

[0109] In some embodiments, see Figure 3a The initial video understanding model includes an encoding network 302 and a sorting processing component 304. Figure 3b , the method for determining the predicted probability distribution data may include the following steps:

[0110] S310: Perform feature extraction and feature fusion on the image frame and text in the first image-text pair through an encoding network to obtain a first fused feature of the first image-text pair.

[0111] The first fusion feature is used to represent the overall features of the image frame and the corresponding text in the first image-text pair. The first fusion feature can be represented by a vector or other forms. The first fusion feature can be directly obtained by splicing the image frame features and the text features, or by extracting the image frame features and the text features based on a cross-attention mechanism.

[0112] In some cases, as mentioned above, two types of data, image frames and their corresponding texts, can provide more comprehensive information. Therefore, feature extraction and feature fusion are performed based on the image frame and text in the first image-text pair to obtain the fusion features of the first image-text pair.

[0113] Specifically, the image frame in the first image-text pair is encoded through the encoding network to obtain image features of the image frame, and the text corresponding to the image frame is encoded through the encoding network to obtain text features of the text. Encoding can be performed based on the image features of the image frame and the text features of the text to obtain fusion features of the first image-text pair.

[0114] S320: Perform feature extraction and feature fusion on the image frame and text in the second image-text pair through the encoding network to obtain a second fused feature of the second image-text pair.

[0115] The second fused feature is used to represent the overall features of the image frame and the corresponding text in the second image-text pair. The second fused feature can be represented by a vector or other forms. The second fused feature can be directly obtained by splicing the image frame features and the text features, or by extracting the image frame features and the text features based on a cross-attention mechanism.

[0116] In some cases, as mentioned above, two types of data, image frames and their corresponding texts, can provide more comprehensive information. Therefore, feature extraction and feature fusion are performed based on the image frames and texts in the second image-text pair to obtain fused features of the second image-text pair.

[0117] Specifically, the image frames in the second image-text pair are encoded through the encoding network to obtain image features of the image frames, and the text corresponding to the image frames is encoded through the encoding network to obtain text features of the text. Encoding can be performed based on the image features of the image frames and the text features of the text to obtain fusion features of the second image-text pair.

[0118] S330: Input the first fusion feature and the second fusion feature into a sorting processing component for binary classification processing to obtain predicted probability distribution data.

[0119] There is a frame order between the first image-text pair and the second image-text pair, for example, the first image-text pair is arranged before the second image-text, or the first image-text pair is arranged after the second image-text. The sorting processing component may be a pairwise classification component (Pairwise Classification Heads).

[0120] Specifically, the frame sorting problem between the first image-text pair and the second image-text pair is converted into a binary classification problem, and the first fused feature of the first image-text pair and the second fused feature of the second image-text pair are input into the sorting processing component for binary classification processing to obtain predicted probability distribution data. The predicted probability distribution data reflects the probability distribution of the frame sequence between the first image-text pair and the second image-text pair being divided into two categories, indicating the possibility that the first image-text pair is arranged before the second image-text, and the possibility that the first image-text pair is arranged after the second image-text.

[0121] In the above implementation, by fusing and sorting the information of the two modalities of image frames and their corresponding texts, the frame sorting problem between the first image-text pair and the second image-text pair is converted into a binary classification problem, and the model is trained to predict the frame order between any two image-text pairs.

[0122] In some embodiments, see Figure 4 , the initial video understanding model also includes a cluster processing component 402. Among them, the cluster processing component can be a cluster head (Cluster Heads) that performs cluster processing on the first image-text pair and the second image-text pair. Specifically, during the model training process, through contrast loss learning, the model can better learn the similarities between image-text pairs in the same unit segment, thereby improving the classification accuracy and generalization ability of the model. Therefore, the first fused feature of the first image-text pair and the second fused feature of the second image-text pair are input into the cluster processing component for contrast loss learning, and the first feature distribution data between the first image-text pair and the second image-text pair is obtained, so that the first image-text pair and the second image-text pair are closer in the feature space.

[0123] In some embodiments, see Figure 5 The method for determining the second feature distribution data may include: determining a third fusion feature of a third image-text pair; inputting the second fusion feature and the third fusion feature into a clustering processing component for contrast loss learning to obtain the second feature distribution data between the second image-text pair and the third image-text pair.

[0124] Specifically, during the model training process, contrastive loss learning can allow the model to better learn the differences between image-text pairs of different unit segments, thereby improving the classification accuracy and generalization ability of the model. Therefore, a third image-text pair from a different unit segment from the second image-text pair is obtained, and feature extraction and fusion are performed on the image frame and text in the third image-text pair to obtain the third fused feature of the third image-text pair. The second fused feature of the second image-text pair and the third fused feature of the third image-text pair are input into the clustering processing component for contrastive loss learning to obtain the second feature distribution data between the second image-text pair and the third image-text pair, so that the second image-text pair and the third image-text pair are more dispersed in the feature space.

[0125] In the above implementation, by fusing and sorting the information of the two modalities of the image frame and its corresponding text, the model is trained by contrastive learning to predict whether any two image-text pairs belong to the same unit segment.

[0126] In some embodiments, see Figure 6a , the coding network includes a first coding network 602 and a second coding network 604. Figure 6b The method for determining the first fusion feature and the second fusion feature may include the following steps:

[0127] S610: Extract features of an image frame and text in a first image-text pair through a first encoding network to obtain first image features and first text features.

[0128] Specifically, the image frame and the text in the first image-text pair are respectively recorded as the first image and the first text. The first encoding network includes an image encoding network and a text encoding network. The first image is input into the image encoding network to obtain a first image feature of the first image. The first text is input into the text encoding network to obtain a first text feature of the first text. The first image feature of the first image and the first text feature of the first text can form a first feature pair.

[0129] S620: Perform feature extraction on the image frame and text in the second image-text pair through the first encoding network to obtain second image features and second text features.

[0130] Specifically, the image frame and the text in the second image-text pair are respectively recorded as the second image and the second text. The second image is input into the image encoding network to obtain the second image feature of the second image. The second text is input into the text encoding network to obtain the second text feature of the second text. The second image feature of the second image and the second text feature of the second text can form a second feature pair.

[0131] S630, input the first image feature, the first text feature, the second image feature, and the second text feature into the second encoding network for cross-fusion and feature extraction, to obtain the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text.

[0132] Specifically, the first feature pair and the second feature pair are input into the second encoding network, and the first feature pair and the second feature pair are cross-fused and feature extracted through the second encoding network to extract the temporal features between the first feature pair and the second feature pair, so that they have each other's feature information, and obtain the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text.

[0133] Exemplarily, the second encoding network may use a Transformer encoder, which may capture the complexity and temporal correlation between the first feature pair and the second feature pair, thereby achieving more accurate prediction and generation. Exemplarily, the second encoding network may use an encoder with a cross-attention mechanism.

[0134] S640: Merge the associated feature classification head of the first image and the associated feature classification head of the first text to obtain a first fusion feature.

[0135] S650: Merge the associated feature classification head of the second image and the associated feature classification head of the second text to obtain a second fusion feature.

[0136] Specifically, the associated feature classification head of the first image and the associated feature classification head of the first text are spliced ​​to obtain the first fusion feature. The associated feature classification head of the second image and the associated feature classification head of the second text are spliced ​​to obtain the second fusion feature.

[0137] In this embodiment, the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text are obtained through the first encoding network and the second encoding network, so as to more effectively and accurately extract the information contained in the video file, so as to better assist the subsequent model training process.

[0138] In some embodiments, the first encoding network includes an image encoding branch and a text encoding branch that are trained together. Figure 7a , extracting features from the image frame and the text in the first image-text pair through a first encoding network to obtain first image features and first text features may include the following steps:

[0139] S710 . Perform feature extraction on image frames in a first image-text pair through an image coding branch to obtain first image features.

[0140] S720: Extract features of the text in the first image-text pair through the text encoding branch to obtain a first text feature.

[0141] In some cases, since there is a correspondence between the image frame and the text in the image-text pair, in order to make the extracted image features contain the feature information of the corresponding text, and the features of the corresponding text contain the feature information of the image, to achieve information fusion and understanding in the two modalities of image and text, the features of the two modalities have useful information about each other. Therefore, in this embodiment, the image encoding branch and the text encoding branch of the first encoding network can be obtained by joint training to alleviate the semantic gap between the image frame and its corresponding text. The first encoding network is trained using image-text pairs with a corresponding relationship, so that the image encoding branch and the text encoding branch can fuse and understand the information of the two modalities when processing the image frame and its corresponding text.

[0142] Specifically, the first image-text pair includes a first image and a first text corresponding to the first image. The first encoding network includes an image encoding branch and a text encoding branch obtained by joint training. The first image-text pair is input into the first encoding network, and the image encoding branch is an image encoder. The image encoder is used to extract features of the first image to obtain first image features. The text encoding branch is a text encoder. The text encoder is used to extract features of the first text to obtain first text features. It is understandable that the first image-text pair and the second image-text pair can be input into the first encoding network together. For the second image text, the second image-text pair includes a second image and a second text corresponding to the second image. The image encoder is used to extract features of the second image to obtain second image features. The text encoder is used to extract features of the second text to obtain second text features.

[0143] For example, see Figure 7b , Figure 7bThe initial video understanding model provided by the embodiment of this specification is shown, and the initial video understanding model includes a first encoding network 602, a second encoding network 604, a sorting processing component 304, and a clustering processing component 402. Among them, the first encoding network can adopt the CLIP (Contrastive Language-Image Pre-Training Model) model. The first encoding network includes an image encoder (Vision Encoder) and a text encoder (Language Encoder) obtained by joint training. The first image-text pair and the second image-text pair are input into the initial video understanding model, and feature extraction is performed by the image encoder and the text encoder in the first encoding network to obtain the first feature pair and the second feature pair. The first feature pair and the second feature pair are input into the second encoding network to obtain the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text. Based on the associated feature classification head of the first image and the associated feature classification head of the first text, the fusion feature of the first image-text pair is obtained. Based on the associated feature classification head of the second image and the associated feature classification head of the second text, the fusion feature of the second image-text pair is obtained. The fusion features of the first image-text pair and the fusion features of the second image-text pair are input into the sorting component for sorting processing, and the fusion features of the first image-text pair and the fusion features of the second image-text pair are input into the clustering component for clustering processing to train the initial video understanding model. Exemplarily, the initial video understanding model can be called a hierarchical contrastive movie clustering model (HCMC), including a language encoder, a vision encoder, an image-text joint encoder (Video-Dialogue Transformer Encoder), a sorting processing component (Pairwise Classification Head), and a clustering processing component (Cluster Head). The determination method of the first feature distribution data can include: inputting the first fusion feature of the first image-text pair and the second fusion feature of the second image-text pair into the clustering processing component for contrast loss learning to obtain the first feature distribution data between the first image-text pair and the second image-text pair.

[0144] In the above implementation, the image encoding branch and the text encoding branch obtained through joint training can fuse and understand the information of the two modalities when processing image frames and their corresponding texts, obtain more comprehensive image text feature information, and more accurately understand the content of the video file.

[0145] In some embodiments, the unit segment is a scene. The first image-text pair and the second image-text pair belong to the same scene. The method for determining the contrast loss data may include: determining the scene contrast loss data based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair. The second image-text pair and the third image-text pair belong to different scenes.

[0146] In some cases, since image-text pairs belonging to the same scene have similarities, the first image-text pair and the second image-text pair in the same scene are used to train the model to predict whether any two image-text pairs belong to the same unit segment.

[0147] Specifically, the first image-text pair and the second image-text are obtained from the same scene, and the third image-text pair is obtained from another scene, that is, the first image-text pair and the second image-text pair belong to the same scene, and the second image-text pair and the third image-text pair belong to different scenes. The first image-text pair and the second image-text pair are used to construct positive samples, and the second image-text pair and the third image-text pair are used to construct negative samples. The first feature distribution data corresponding to the positive samples and the second feature distribution data corresponding to the negative samples are determined, and the scene contrast loss data is determined based on the first feature distribution data and the second feature distribution data, and the model parameters are updated so that the first image-text pair and the second image-text pair are closer in the feature space, and the second image-text pair and the third image-text pair are more dispersed.

[0148] In the above implementation, the model is trained by using contrastive learning to predict whether any two image-text pairs belong to the same scene.

[0149] In some embodiments, the unit segment is a storyboard. The first image-text pair and the second image-text pair belong to the same storyboard. The method for determining the contrast loss data may include: determining the storyboard contrast loss data based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair. The second image-text pair and the third image-text pair belong to different storyboards.

[0150] In some cases, since there are similarities between image-text pairs belonging to the same storyboard, the first image-text pair and the second image-text pair under the same storyboard are used to train the model to predict whether any two image-text pairs belong to the same unit segment.

[0151] Specifically, the first image-text pair and the second image-text are obtained from the same storyboard, and the third image-text pair is obtained from another storyboard, that is, the first image-text pair and the second image-text pair belong to the same storyboard, and the second image-text pair and the third image-text pair belong to different storyboards. The first image-text pair and the second image-text pair are used to construct a positive sample, and the second image-text pair and the third image-text pair are used to construct a negative sample. The first feature distribution data corresponding to the positive sample and the second feature distribution data corresponding to the negative sample are determined, and the storyboard contrast loss data is determined based on the first feature distribution data and the second feature distribution data, and the model parameters are updated so that the first image-text pair and the second image-text pair are closer in the feature space, and the second image-text pair and the third image-text pair are more dispersed.

[0152] In the above implementation, the model is trained by using contrastive learning to predict whether any two image-text pairs belong to the same storyboard.

[0153] In some embodiments, the unit segment is a scene and a storyboard. The model training process includes a first training stage and a second training stage, and the first training stage is earlier than the second training stage. Figure 8 , the method for determining the comparison loss data may include the following steps:

[0154] S810. In a first training phase, determine the frame contrast loss data based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair.

[0155] S820. In the second training phase, determine scene contrast loss data based on the third feature distribution data between the fourth image-text pair and the fifth image-text pair, and the fourth feature distribution data between the fifth image-text pair and the sixth image-text pair.

[0156] In some cases, the hierarchical structure corresponding to the video file includes scenes and storyboards. The video file includes several scenes, one scene includes several storyboards, and one storyboard includes several image-text pairs; that is, the hierarchical structure of the video file is scene, storyboard, image-text pair from top to bottom. Therefore, it is necessary to train the model to predict whether any two image-text pairs belong to the same storyboard and whether any two image-text pairs belong to the same scene.

[0157] Specifically, the model training stage is divided into two stages: the first training stage and the second training stage. In the first training stage, the model is trained to determine whether any two image-text pairs belong to the same storyboard. The first image-text pair and the second image-text pair are input into the initial video understanding model to obtain the first feature distribution data between the first image-text pair and the second image-text pair. The second image-text pair and the third image-text pair are input into the initial video understanding model to obtain the second feature distribution data between the second image-text pair and the third image-text pair. Among them, the first image-text pair and the second image-text pair belong to the same storyboard, and the second image-text pair and the third image-text pair belong to different storyboards. Contrast loss learning is performed based on the first feature distribution data and the second feature distribution data to obtain storyboard contrast loss data, and the model parameters are updated so that the first image-text pair and the second image-text pair are closer in the feature space, and the second image-text pair and the third image-text pair are more dispersed.

[0158] In the second training stage, the model is trained to determine whether any two image-text pairs belong to the same scene. The fourth image-text pair and the fifth image-text pair are input into the initial video understanding model to obtain the third feature distribution data between the fourth image-text pair and the fifth image-text pair. The fifth image-text pair and the sixth image-text pair are input into the initial video understanding model to obtain the fourth feature distribution data between the fifth image-text pair and the sixth image-text pair. Among them, the fourth image-text pair and the fifth image-text pair belong to the same scene; the fifth image-text pair and the sixth image-text pair belong to different scenes. Contrast loss learning is performed based on the fourth feature distribution data and the third feature distribution data to obtain scene contrast loss data, and the model parameters are updated so that the fourth image-text pair and the fifth image-text pair are closer in the feature space, and the fifth image-text pair and the sixth image-text pair are more dispersed.

[0159] In the above implementation, by using contrastive learning, the model is trained in the first training phase to predict whether any two image-text pairs belong to the same storyboard, and in the second training phase to predict whether any two image-text pairs belong to the same scene.

[0160] This specification provides a model training method. Fig. 9 , the model training method may include the following steps:

[0161] S910: Input the first image-text pair and the second image-text pair into the initial video understanding model, and determine the predicted probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the first feature distribution data between the first image-text pair and the second image-text pair. The first image-text pair and the second image-text pair belong to the same unit segment.

[0162] S920, determining binary classification loss data based on the true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data.

[0163] S930. Determine contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments.

[0164] S940, updating parameters of the initial video understanding model based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

[0165] For a detailed description of the model training method, please refer to the description of the video understanding method above, which will not be repeated here.

[0166] The embodiment of this specification also provides a video understanding device, see Fig.10 , the video understanding apparatus 1000 may include:

[0167] The image-text pair acquisition module 1010 is used to acquire a plurality of image-text pairs corresponding to the video file to be understood; wherein the image-text pairs include image frames and texts corresponding to the image frames;

[0168] The video file understanding module 1020 is used to input multiple image-text pairs into the target video understanding model for content understanding to obtain the video content of the video file to be understood; wherein, the loss data of the target video understanding model during the training process includes binary classification loss data and contrast loss data, the binary classification loss data is used to train the target video understanding model's ability to predict the frame order of any two image-text pairs, and the contrast loss data is used to train the target video understanding model's ability to predict whether any two image-text pairs belong to the same unit segment.

[0169] For a detailed description of the video understanding device, please refer to the description of the video understanding method above, which will not be repeated here.

[0170] This specification also provides a model training device 1100, see Fig.11 , the model training device 1100 may include:

[0171] An image-text pair input module 1110 is used to input the first image-text pair and the second image-text pair into the initial video understanding model, and determine the predicted probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment;

[0172] A classification loss determination module 1120, configured to determine binary classification loss data based on the actual probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data;

[0173] A contrast loss determination module 1130, configured to determine contrast loss data based on first feature distribution data between a first image-text pair and a second image-text pair, and second feature distribution data between the second image-text pair and a third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments;

[0174] The model parameter updating module 1140 is used to update the parameters of the initial video understanding model based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

[0175] For a detailed description of the model training device, please refer to the description of the video understanding method above, which will not be repeated here.

[0176] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer executes the method in any of the above embodiments.

[0177] The embodiments of this specification also provide a computer program product including instructions, which, when executed by a computer, enables the computer to execute the method in any of the above embodiments.

[0178] It should be understood that the specific examples herein are only intended to help those skilled in the art to better understand the embodiments of the present specification, rather than to limit the scope of the present invention.

[0179] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0180] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0181] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0182] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0183] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0184] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0186] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0187] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0188] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0189] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0190] The above is only a specific implementation of this specification, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video understanding method, characterized in that: The method comprises: Acquire a plurality of image-text pairs corresponding to the video file to be understood; wherein the image-text pairs include image frames and texts corresponding to the image frames; The multiple image-text pairs are input into a target video understanding model for content understanding to obtain the video content of the video file to be understood; wherein the loss data of the target video understanding model during the training process includes binary classification loss data and contrast loss data, the binary classification loss data is used to train the target video understanding model's ability to predict the frame order of any two image-text pairs, and the contrast loss data is used to train the target video understanding model's ability to predict whether the any two image-text pairs belong to the same unit segment.

2. The method according to claim 1, characterized in that The arbitrary two image-text pairs include a first image-text pair and a second image-text pair; the training method of the target video understanding model includes: Inputting the first image-text pair and the second image-text pair into an initial video understanding model, determining predicted probability distribution data of a frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment; Determine the binary classification loss data based on the true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data; Determine the contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments; The parameters of the initial video understanding model are updated based on the binary classification loss data and the contrast loss data to obtain the target video understanding model.

3. The method according to claim 2, characterized in that The initial video understanding model includes an encoding network and a sorting processing component; the method for determining the predicted probability distribution data includes: Performing feature extraction and feature fusion on the image frame and the text in the first image-text pair through the encoding network to obtain a first fusion feature of the first image-text pair; Performing feature extraction and feature fusion on the image frame and the text in the second image-text pair through the encoding network to obtain a second fused feature of the second image-text pair; The first fusion feature and the second fusion feature are input into the sorting processing component for binary classification processing to obtain the predicted probability distribution data.

4. The method according to claim 3, characterized in that The initial video understanding model further includes a clustering processing component; the method for determining the first feature distribution data includes: The first fused feature of the first image-text pair and the second fused feature of the second image-text pair are input into the clustering processing component for contrast loss learning to obtain first feature distribution data between the first image-text pair and the second image-text pair.

5. The method according to claim 4, characterized in that The method for determining the second characteristic distribution data includes: determining a third fusion feature of a third image-text pair; The second fused feature and the third fused feature are input into the cluster processing component to perform contrast loss learning, so as to obtain second feature distribution data between the second image-text pair and the third image-text pair.

6. The method according to claim 3, characterized in that The encoding network includes a first encoding network and a second encoding network; the method for determining the first fusion feature and the second fusion feature includes: Extracting features from the image frame and the text in the first image-text pair through the first encoding network to obtain first image features and first text features; Extracting features from the image frame and the text in the second image-text pair through the first encoding network to obtain second image features and second text features; Inputting the first image feature, the first text feature, the second image feature, and the second text feature into the second encoding network for cross-fusion and feature extraction, to obtain a correlation feature classification head of the first image, a correlation feature classification head of the first text, a correlation feature classification head of the second image, and a correlation feature classification head of the second text; Merging the associated feature classification head of the first image and the associated feature classification head of the first text to obtain the first fusion feature; The second fusion feature is obtained by merging the associated feature classification head of the second image and the associated feature classification head of the second text.

7. The method according to claim 6, characterized in that The first encoding network includes an image encoding branch and a text encoding branch obtained by common training; the first encoding network extracts features of the image frame and the text in the first image-text pair to obtain first image features and first text features, including: Extracting features from the image frames in the first image-text pair by using the image encoding branch to obtain the first image features; The text in the first image-text pair is subjected to feature extraction by the text encoding branch to obtain the first text feature.

8. The method according to any one of claims 2 to 7, characterized in that: The unit segment is a scene; the first image-text pair and the second image-text pair belong to the same scene; and the method for determining the contrast loss data includes: The scene contrast loss data is determined based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different scenes.

9. The method according to any one of claims 2 to 7, characterized in that: The unit segment is a storyboard; the first image-text pair and the second image-text pair belong to the same storyboard; and the method for determining the contrast loss data includes: The mirror contrast loss data is determined based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different mirrors.

10. The method according to any one of claims 2 to 7, characterized in that: The unit segments are scenes and storyboards; the model training process includes a first training stage and a second training stage, the first training stage is earlier than the second training stage; The method for determining the comparative loss data includes: In the first training stage, determining the frame contrast loss data based on the first feature distribution data between the first image-text pair and the second image-text pair, and the second feature distribution data between the second image-text pair and the third image-text pair; wherein the first image-text pair and the second image-text pair belong to the same frame, and the second image-text pair and the third image-text pair belong to different frames; In the second training stage, scene contrast loss data is determined based on the third feature distribution data between the fourth image-text pair and the fifth image-text pair, and the fourth feature distribution data between the fifth image-text pair and the sixth image-text pair; wherein the fourth image-text pair and the fifth image-text pair belong to the same scene; and the fifth image-text pair and the sixth image-text pair belong to different scenes.

11. A model training method, characterized in that: The method comprises: Inputting a first image-text pair and a second image-text pair into an initial video understanding model, determining predicted probability distribution data of a frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment; Determine binary classification loss data based on true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data; Determine contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments; The parameters of the initial video understanding model are updated based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

12. A video understanding device, characterized in that: The device comprises: An image-text pair acquisition module is used to acquire a plurality of image-text pairs corresponding to the video file to be understood; wherein the image-text pairs include image frames and texts corresponding to the image frames; A video file understanding module is used to input the multiple image-text pairs into a target video understanding model for content understanding to obtain the video content of the video file to be understood; wherein the loss data of the target video understanding model during the training process includes binary classification loss data and contrast loss data, the binary classification loss data is used to train the target video understanding model's ability to predict the frame order of any two image-text pairs, and the contrast loss data is used to train the target video understanding model's ability to predict whether the any two image-text pairs belong to the same unit segment.

13. A model training device, characterized in that: The device comprises: An image-text pair input module, used to input a first image-text pair and a second image-text pair into an initial video understanding model, and determine predicted probability distribution data of a frame sequence between the first image-text pair and the second image-text pair, and first feature distribution data between the first image-text pair and the second image-text pair; wherein the first image-text pair and the second image-text pair belong to the same unit segment; A classification loss determination module, configured to determine binary classification loss data based on the true probability distribution data of the frame sequence between the first image-text pair and the second image-text pair, and the predicted probability distribution data; a contrast loss determination module, configured to determine contrast loss data based on first feature distribution data between the first image-text pair and the second image-text pair, and second feature distribution data between the second image-text pair and the third image-text pair; wherein the second image-text pair and the third image-text pair belong to different unit segments; A model parameter updating module is used to update the parameters of the initial video understanding model based on the binary classification loss data and the contrast loss data to obtain a target video understanding model.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.