Video understanding, plot prediction and video question and answer method and device, equipment and medium

Through feature fusion and clustering processing of image text pairs, the problems of high computational cost of video understanding tasks and neglect of visual narrative structure in the prior art are solved, the overall understanding of video files and visual narrative reasoning are realized, and the video understanding is improved Accuracy and application potential.

CN119066224APending Publication Date: 2024-12-03BEIJING INSTITUTE FOR GENERAL ARTIFICIAL INTELLIGENCE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310629127.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

Smart Images

  • Figure CN119066224A_ABST
    Figure CN119066224A_ABST
Patent Text Reader

Abstract

The invention discloses video understanding, plot prediction and video question and answer methods, devices, equipment and a medium. The method comprises the following steps: acquiring a plurality of image text pairs corresponding to a video file, and performing feature extraction and feature fusion on image frames and texts in the image text pairs to obtain fusion features of the image text pairs; secondly, clustering processing is carried out based on the fusion features of the image text pair to obtain a video segmentation cluster, and a basic narrative structure of a video file is determined; sorting the image text pairs in the video segment clusters to obtain an in-segment arrangement sequence of the image text pairs in the video segment clusters, and obtaining a visual semantic sequence of each video segment cluster; and finally, carrying out content understanding on the video file according to the arrangement sequence in the segments. Important information such as event development and story plots in a video file is understood, and a visual narrative reasoning task is completed by taking overall video understanding and a visual narrative structure as two key factors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video understanding technology, and in particular to a video understanding, plot prediction, video question and answer method, device, equipment and medium. The present invention can be used in video search, smart home, online education and other fields. Background Art

[0002] Video understanding is a topic that has attracted much attention from both academia and industry in the current field of computer vision research. With the rapid development of the video industry, how to use artificial intelligence technology to better understand video content is becoming increasingly important.

[0003] Although some studies in related technologies have noticed the importance of video understanding, the video understanding task is still regarded as a visual recognition and classification task. Therefore, a new way to understand the video content is needed. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present invention proposes a video understanding, plot prediction, video question answering method, device, equipment and medium.

[0005] The present invention provides a video understanding method, which comprises: obtaining a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs comprise image frames and texts corresponding to the image frames; performing feature extraction and feature fusion on the image frames and texts in the image-text pairs to obtain fusion features of the image-text pairs; performing clustering processing based on the fusion features of the image-text pairs to obtain video segmentation clusters; wherein the image-text pairs corresponding to the fusion features included in the video segmentation clusters belong to the same unit segment of the video file; sorting the image-text pairs in the video segmentation clusters to obtain the intra-segment arrangement order of the image-text pairs in the video segmentation clusters; and performing content understanding on the video file according to the intra-segment arrangement order.

[0006] The present invention provides a plot prediction method, which comprises: obtaining a plurality of image-text pairs corresponding to a target video clip included in a video file; wherein the image-text pairs comprise image frames and texts corresponding to the image frames; the video file comprises a plurality of video clips; performing feature extraction and feature fusion on the image frames and texts in the image-text pairs to obtain fusion features of the image-text pairs; performing feature extraction on the plot text corresponding to the video clip to obtain plot features of the video clip; performing feature matching based on the fusion features of the image-text pairs and the plot features of the video clip to predict a target plot text corresponding to the target video clip.

[0007] The present invention provides a video question-answering method, which comprises: obtaining a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs comprise image frames and texts corresponding to the image frames; performing feature extraction and feature fusion on the image frames and texts in the image-text pairs to obtain fusion features of the image-text pairs; wherein a question text and a plurality of option texts corresponding to the question text are provided based on the video file; performing feature extraction based on the question text and each of the option texts to obtain question option features corresponding to each of the option texts; performing feature matching based on the fusion features of the image-text pairs and the question option features to obtain feature matching results; and determining the correct option text corresponding to the question text from among the plurality of option texts according to the feature matching results.

[0008] The present invention provides a video understanding device, the device comprising:

[0009] An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame;

[0010] A feature extraction and fusion module, used for performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair;

[0011] A feature clustering processing module, used for performing clustering processing based on the fusion features of the image-text pairs to obtain video segment clusters; wherein the image-text pairs corresponding to the fusion features included in the video segment clusters belong to the same unit segment;

[0012] An arrangement order determination module is used to sort the image-text pairs in the video segment cluster to obtain an intra-segment arrangement order of the image-text pairs in the video segment cluster;

[0013] The video understanding module is used to understand the content of the video file according to the arrangement order in the segments.

[0014] The present invention provides a plot prediction device, the device comprising:

[0015] An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to a target video segment included in a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; and the video file includes a plurality of video segments;

[0016] A feature extraction and fusion module, used for performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair;

[0017] A plot feature extraction module is used to extract features of the plot text corresponding to the video clip to obtain plot features of the video clip;

[0018] The plot text matching module is used to perform feature matching based on the fusion features of the image-text pair and the plot features of the video clip to obtain the target plot text corresponding to the target video clip.

[0019] The present invention provides a video question-answering device, the device comprising:

[0020] An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame;

[0021] A feature extraction and fusion module is used to extract and fuse features of the image frame and text in the image-text pair to obtain fusion features of the image-text pair; wherein a question text and a plurality of option texts corresponding to the question text are provided based on the video file;

[0022] An option feature extraction module performs feature extraction based on the question text and each of the option texts to obtain a question option feature corresponding to each of the option texts;

[0023] An option feature matching module is used to perform feature matching based on the fusion feature of the image text pair and the question option feature to obtain a feature matching result;

[0024] The correct option determination module is used to determine the correct option text corresponding to the question text from the multiple option texts according to the feature matching result.

[0025] The present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above methods when executing the computer program.

[0026] The present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods are implemented.

[0027] In the present invention, by obtaining multiple image-text pairs corresponding to the video file, feature extraction and feature fusion are performed on the image frames and texts in the image-text pairs to obtain the fusion features of the image-text pairs; the feature information of the video file is extracted in a multimodal form to provide a comprehensive basis for subsequent video understanding; secondly, clustering is performed based on the fusion features of the image-text pairs to obtain video segment clusters and determine the basic narrative structure of the video file; the image-text pairs in the video segment clusters are sorted to obtain the intra-segment arrangement order of the image-text pairs in the video segment clusters, and the visual semantic order of each video segment cluster is obtained; finally, the content of the video file is understood according to the intra-segment arrangement order. The understanding of important information such as the development of events and storylines in the video file is realized, and the visual narrative reasoning (VNR) task is completed with the overall video understanding and the visual narrative structure as two key factors. The overall understanding of the video file is realized in a new way, instead of defining the video understanding task as visual recognition and classification, which reduces the limitation of video understanding in the application scenario of long videos.

[0028] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1a A flowchart of a video understanding method provided according to an embodiment of this specification;

[0030] Figure 1b is a framework diagram of a video understanding model according to an embodiment of this specification;

[0031] Figure 2 A flowchart of a video understanding method provided according to another embodiment of the present specification;

[0032] Figure 3 A flowchart of a video understanding method provided according to another embodiment of the present specification;

[0033] Figure 4a A flowchart of a method for obtaining a first feature pair according to an embodiment of this specification;

[0034] Figure 4b is a framework diagram of a video understanding model according to another embodiment of the present specification;

[0035] Figure 5 A flowchart of a method for obtaining a first feature pair according to another embodiment of the present specification;

[0036] Figure 6a A flowchart of a video understanding method provided according to an embodiment of this specification;

[0037] Figure 6b is a schematic diagram of a hierarchical structure of a video file according to an embodiment of the present specification;

[0038] Figure 7 A flowchart of a video understanding method provided according to an embodiment of this specification;

[0039] Figure 8 A flowchart of a video understanding method provided according to an embodiment of this specification;

[0040] Figure 9a A flowchart of a video understanding method provided according to an embodiment of this specification;

[0041] Figure 9b is a schematic diagram of a hierarchical structure of a video file according to an embodiment of the present specification;

[0042] Fig.10 A flowchart of a plot prediction method provided according to an embodiment of this specification;

[0043] Fig.11 A flowchart of a video question-answering method provided according to an embodiment of this specification;

[0044] Fig.12 A schematic diagram of a framework of a video understanding device provided according to an embodiment of this specification;

[0045] Fig.13 A schematic diagram of a framework of a plot prediction device provided according to an embodiment of this specification;

[0046] Fig.14 The figure is a schematic diagram of a framework of a video question-and-answer device provided according to an embodiment of the present specification. DETAILED DESCRIPTION

[0047] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0048] Video understanding is an artificial intelligence technology that uses deep neural networks to automatically parse and understand video content. By analyzing and learning videos, video understanding can identify information such as objects, actions, scenes, and emotions in videos, thereby improving the accuracy and efficiency of applications such as video retrieval, classification, labeling, and generation. Video features are automatically learned and extracted through deep neural network models composed of multiple layers of neurons. These models can learn the characteristics and laws of videos through a large amount of training data and apply them to video understanding tasks. Video understanding has a wide range of applications, including video classification, object detection, behavior recognition, sentiment analysis, video generation, etc. For example, deep learning video understanding can be used to implement applications such as intelligent monitoring, autonomous driving, and video recommendation, improving people's production and living efficiency.

[0049] As mentioned in the background technology, although some studies in the related art have noticed the importance of video understanding, they still regard the video understanding task as a visual recognition and classification task. After analysis by the inventors, it is found that for long video files (such as movies and TV shows), the computational cost is huge and growing, which will also limit the development of this type of video understanding.

[0050] Furthermore, people and even young children can understand and reason about videos through the visual and auditory information included in video files, and form corresponding visual narratives. In the related art, video language models mainly understand videos based on dense features extracted from vision and language. However, the inventors have found through analysis that two key factors are ignored in the related art: overall understanding of the video and the visual narrative structure of the video.

[0051] Different from the visual recognition and classification that are the main focus in the related art, the inventor proposes a method for understanding videos in an overall abstract manner, and the implementation method of this specification defines the video understanding task as the task of understanding the relationship between multiple frames of images in a video file. Furthermore, since each image frame in a video file can be regarded as a functional unit for constructing the narrative ability of the video file, the order of image frames is crucial to the integrity of the video file. If unreasonable exchanges occur between image frames, the integrity of the video file may be destroyed. Therefore, the video understanding model in the implementation method of this specification has the ability to understand the hierarchical structure of video files, as well as the ability to perceive and understand the narrative structure of image frame sequences, to achieve an overall understanding of video files, and to achieve visual narrative reasoning capabilities based on video files.

[0052] Based on this, the embodiment of this specification provides a video understanding method. In order to improve the video understanding ability, a multimodal input is adopted, that is, a plurality of image-text pairs corresponding to the video file are obtained, and the image-text pairs include image frames and texts corresponding to the image frames. Then, feature extraction and feature fusion are performed on the image frames and texts in the image-text pairs to obtain the fusion features of the image-text pairs. In order to determine the narrative hierarchy of the video file, clustering processing is performed based on the fusion features of the image-text pairs to obtain video segmentation clusters, so that the fusion features of image-text pairs with high similarity are classified together, so that the image-text pairs corresponding to the fusion features included in the video segmentation cluster belong to the same unit segment. Finally, the image-text pairs in the video segmentation cluster are sorted to obtain the intra-segment arrangement order of the image-text pairs in the video segmentation cluster; and the content of the video file is understood according to the intra-segment arrangement order. At this point, video understanding is completed based on the multiple image texts corresponding to the video file. The above-mentioned video understanding method can be a processing process of a video understanding model. The model has the ability to understand the overall video file and the visual narrative reasoning ability to complete the content understanding of the video file. The video understanding model can be directly deployed on the client or server of the video understanding application system for use, or the video understanding model can be applied to downstream rich video understanding tasks after small sample learning, such as Video Question Answering (VideoQA), Video-Grounded DialogueGenerations, and Fine-Grained Action Predictions.

[0053] The video understanding method provided in the embodiments of this specification can be applied to the client or server of the video understanding application system. The client can be an electronic device with network access capability. Specifically, for example, the client can be a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a smart wearable device, a shopping guide terminal, a television, a smart speaker, a microphone, etc. Among them, smart wearable devices include but are not limited to smart bracelets, smart watches, smart glasses, smart helmets, smart necklaces, etc. Alternatively, the client can also be software that can run in an electronic device. The server can be an electronic device with certain computing and processing capabilities. It can have a network communication module, a processor, and a memory, etc. Of course, the server can also refer to software running in an electronic device. The server can also be a distributed server, which can be a system with multiple processors, memories, network communication modules, etc. operating in collaboration. Alternatively, the server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the embodiments of the specification. For example, it can be a new form of "server" based on quantum computing.

[0054] This specification provides a video understanding method. Figure 1a , the video understanding method may include the following steps:

[0055] S110. Acquire a plurality of image-text pairs corresponding to the video file; wherein the image-text pairs include image frames and texts corresponding to the image frames.

[0056] The video file may be a short video file, a long video file such as a movie or TV program, or an animation video file. The video file includes a series of image frames, and the image frames correspond to text. The text may be subtitle text, glyph text, non-subtitle text obtained from the image frame using text recognition, or text features of the image frame. An image-text pair may be formed based on the image frame and the text corresponding to the image frame.

[0057] In some cases, in order to improve the accuracy and comprehensiveness of video content understanding, and the video files involve image frames of image type and text data of text type, the two different types of data, image frames and their corresponding texts, are jointly processed in the video understanding process.

[0058] Specifically, in some embodiments, when it is necessary to understand the content of a video file, extraction can be performed based on the video file to obtain multiple image-text pairs corresponding to the video file. In other embodiments, the video file can be extracted in advance to obtain multiple image-text pairs corresponding to the video file and store them. When it is necessary to understand the content of the video file, the multiple image-text pairs corresponding to the video file can be directly obtained.

[0059] S120 , performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair.

[0060] The fusion feature is used to represent the overall features of the image frame and the corresponding text in the image-text pair. The fusion feature can be represented by a vector or other forms. The fusion feature can be directly obtained by splicing the image frame features and the text features, or by extracting the image frame features and the text features based on the cross-attention mechanism.

[0061] In some cases, as mentioned above, two types of data, image frames and their corresponding texts, can provide more comprehensive information. Therefore, feature extraction and feature fusion are performed based on the image frames and texts in the image-text pair to obtain fusion features of the image-text pair. Specifically, the image frames in the image-text pair are encoded to obtain image features of the image frames, and the text corresponding to the image frames is encoded to obtain text features of the text. The image features of the image frames and the text features of the text can be encoded to obtain fusion features of the image-text pair.

[0062] S130 , performing clustering processing based on the fusion features of the image-text pair to obtain video segment clusters.

[0063] Among them, the image text pairs corresponding to the fusion features included in the video segmentation cluster belong to the same unit segmentation of the video file. In the production process of the video file, it will be split or decomposed according to the background, location, etc. of the time, and multiple elements of the same level or different levels will be obtained. In this way, the completed video file will correspond to the video segment. Exemplarily, the video file can be a movie video file, and the video segmentation can be any of the important elements such as scenes and shots in the film production process. The scene can be the place where the event occurs in the movie, which can be indoors or outdoors, a real place, or a fictional place. The storyboard can be a continuous sequence of pictures obtained through a shot in the production of the video file. The production of the storyboard can consider factors such as the script, storyboard, scene design, and character design. Exemplarily, the video file can be an animation video file, and the video segmentation can be a storyboard or storyboard in the animation production process, which can also be called a grid. In the animation video file, the determination of the video segmentation needs to be determined by considering the content in the script and storyboard to determine the scene, character action, and shot switching of each shot.

[0064] Specifically, multiple image-text pairs corresponding to the video file have been obtained. Multiple video segments are involved in the production process of the video file. Relatively speaking, the similarity between image-text pairs in the same video segment is usually higher than the similarity between two image-text pairs in different video segments. Therefore, the fusion features of multiple image-text pairs can be divided by clustering, and multiple fusion features with high similarity are divided into the same video segment cluster, and the similarity between different video segment clusters is low, so as to find the hierarchical structure within the multiple image-text pairs corresponding to the video file.

[0065] For example, the K-Means algorithm can be used to cluster the fused features of the image-text pair. The algorithm first selects K random centroids, then assigns each fused feature to the cluster where the nearest centroid is located, then recalculates the centroid of each cluster, and again assigns each fused feature to the cluster where the nearest centroid is located. This process is repeated until the centroid no longer changes or the preset maximum number of iterations is reached.

[0066] S140 , sorting the image-text pairs in the video segment cluster to obtain the intra-segment arrangement order of the image-text pairs in the video segment cluster.

[0067] The video segment cluster includes a number of image-text pairs, and the arrangement order within the segment is the frame arrangement order of each image-text pair in the video segment cluster.

[0068] In some cases, since multiple image-text pairs in a video segment cluster are out of order after clustering, it is necessary to sort the image-text pairs in the video segment cluster.

[0069] Specifically, in some embodiments, the image-text pairs may be pre-set with frame identification information, and the image-text pairs in the video segment cluster may be sorted according to the frame identification information to obtain the intra-segment arrangement order of the image-text pairs in the video segment cluster. In some other embodiments, the frame arrangement order of each image frame in the video file may be known in advance, and based on the frame arrangement order of the video file, the image-text pairs in the video segment cluster may be sorted to obtain the intra-segment arrangement order of the image-text pairs in the video segment cluster. In other embodiments, the frame order between any two frames of image-text pairs may be known in advance, and based on the frame order between any two frames of image-text pairs, the image-text pairs in the video segment cluster may be sorted to obtain the intra-segment arrangement order of the image-text pairs in the video segment cluster.

[0070] S150: Understand the content of the video file according to the arrangement order in the segments.

[0071] Specifically, the video segment clusters corresponding to the video files have been determined through clustering processing, and the image-text pairs in the video segment clusters have been sorted to obtain the correct intra-segment arrangement order. Furthermore, on the one hand, the hierarchical structure of the video file can be reflected by the video segment clusters, thereby determining the basic narrative structure of the video file; on the other hand, within any video segment cluster, the intra-segment arrangement order can reflect the visual semantic order of any video segment cluster. Finally, based on the basic narrative structure of the video file and the visual semantic order of any video segment cluster, the video content is understood to understand the development of events in the video file, infer the storyline of the video file, and realize visual narrative reasoning (VNR) based on the video file.

[0072] For example, see Figure 1b , Figure 1b The video understanding model provided in the embodiment of the present specification is shown, and the video understanding model includes an encoding network 102, a sorting component 104, and a clustering component 106. Among them, multiple image-text pairs 108 corresponding to the video file are input into the encoding network of the video understanding model, and the image frames and texts in the image-text pairs are subjected to feature extraction and feature fusion through the encoding network to obtain the fusion features 110 of the image-text pairs. The fusion features of the image-text pairs are input into the clustering component 106 for clustering processing to obtain video segment clusters 112. The sorting component 104 is called to sort the image-text pairs in the video segment clusters to obtain the intra-segment arrangement order 114 of the image-text pairs in the video segment clusters. If there are multiple video segment clusters, the sorting component can be continuously called to sort the multiple video segment clusters, thereby obtaining multiple correctly sorted image-text pairs to achieve content understanding of the video file.

[0073] In the above implementation, by obtaining multiple image-text pairs corresponding to the video file, feature extraction and feature fusion are performed on the image frames and texts in the image-text pairs to obtain the fusion features of the image-text pairs; the feature information of the video file is extracted in a multimodal form to provide a comprehensive basis for subsequent video understanding; secondly, clustering is performed based on the fusion features of the image-text pairs to obtain video segment clusters and determine the basic narrative structure of the video file; the image-text pairs in the video segment clusters are sorted to obtain the intra-segment arrangement order of the image-text pairs in the video segment clusters, and the visual semantic order of each video segment cluster is obtained; finally, the content of the video file is understood according to the intra-segment arrangement order. The understanding of important information such as the development of events and storylines in the video file is realized, and the visual narrative reasoning (VNR) task is completed with the overall video understanding and visual narrative structure as two key factors. The overall understanding of the video file is realized in a new way, instead of defining the video understanding task as visual recognition and classification, which reduces the limitation of video understanding in the application scenario of long videos.

[0074] In some embodiments, see Figure 2 , extracting features and fusing features of the image frame and text in the image-text pair to obtain the fusion features of the image-text pair may include the following steps:

[0075] S210 , performing feature extraction on the image frame and the text in the image-text pair to obtain an image classification header for the image frame and a text classification header for the text.

[0076] S220 , performing feature fusion on the image classification head and the text classification head to obtain fusion features of the image-text pair.

[0077] Specifically, feature extraction can be performed on the image frame in the image-text pair, and the extracted features can be mapped to obtain the converged features of the image frame, that is, the image classification head of the image frame. Feature extraction can be performed on the text in the image-text pair, and the extracted features can be mapped to obtain the converged features of the text, that is, the text classification head of the text. Based on the feature combination of the image classification head and the text classification head, the fusion features of the image-text pair are obtained. For example, the image classification head and the text classification head can be spliced ​​together, or the image classification head and the text classification head can be feature fused based on the attention mechanism to obtain the fusion features of the image-text pair.

[0078] Exemplarily, feature extraction is performed on the image frame in the image-text pair to obtain the image feature of the image frame, and feature extraction is performed on the text corresponding to the image frame to obtain the text feature of the text. The image feature of the image frame and the text feature of the text are input into the image-text encoding network together, and the interdependent features between the image frame and its corresponding text are extracted to obtain an image classification head fused with text feature information and a text classification head fused with image feature information.

[0079] In the above implementation, in the process of video understanding of video files, the features of the two modes of image and text are fused to obtain the fusion features of the image-text pair, which provides a more complete data basis for subsequent video understanding and facilitates comprehensive and accurate video understanding of video files.

[0080] In some embodiments, the plurality of image-text pairs include a first image-text pair and a second image-text pair, wherein the first image-text pair includes a first image and a first text corresponding to the first image, and the second image-text pair includes a second image and a second text corresponding to the second image. Figure 3 , extracting features from the image frame and the text in the image-text pair to obtain an image classification head for the image frame and a text classification head for the text may include the following steps:

[0081] S310 , extracting features from a first image-text pair to obtain first image features of the first image and first text features of the first text to form a first feature pair.

[0082] S320 , extracting features from the second image-text pair to obtain first image features of the second image and second text features of the second text to form a second feature pair.

[0083] S330. Cross-fusing and extracting features of the first feature pair and the second feature pair through the first encoding network to obtain an associated feature classification head of the first image, an associated feature classification head of the first text, an associated feature classification head of the second image, and an associated feature classification head of the second text.

[0084] The image classification header of the image frame includes an associated feature classification header of the first image and an associated feature classification header of the second image; the text classification header includes an associated feature classification header of the first text and an associated feature classification header of the second text.

[0085] Specifically, the first image-text pair includes a first image and a first text corresponding to the first image. The first image is input into an image encoding network to obtain a first image feature of the first image. The first text is input into a text encoding network to obtain a first text feature of the first text. The first image feature of the first image and the first text feature of the first text form a first feature pair.

[0086] The second image-text pair includes a second image and a second text corresponding to the second image. The second image is input into the image encoding network to obtain a second image feature of the second image. The second text is input into the text encoding network to obtain a second text feature of the second text. The second image feature of the second image and the second text feature of the second text form a second feature pair.

[0087] The first feature pair and the second feature pair are input into the first encoding network, and the first feature pair and the second feature pair are cross-fused and feature extracted through the first encoding network to extract the temporal features between the first feature pair and the second feature pair, so that they have each other's feature information, and obtain the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text.

[0088] Exemplarily, the first encoding network may use a Transformer encoder, which may capture the complexity and temporal correlation between the first feature pair and the second feature pair, thereby achieving more accurate prediction and generation. Exemplarily, the first encoding network may use an encoder with a cross-attention mechanism.

[0089] In this embodiment, the first feature pair and the second feature pair interact and transmit information through the first encoding network to obtain the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text, so as to more effectively and accurately extract the information contained in the video file, so as to better assist the subsequent understanding of the content of the video file.

[0090] In some embodiments, the second feature pair is obtained in the same manner as the first feature pair. Figure 4a , the method for obtaining the first feature pair may include the following steps:

[0091] S410, inputting the first image-text pair into a second encoding network; wherein the second encoding network includes an image encoding branch and a text encoding branch obtained by joint training.

[0092] S420: Extract features of the first image through the image encoding branch to obtain first image features.

[0093] S430: Extract features of the first text through the text encoding branch to obtain first text features.

[0094] S440: Obtain a first feature pair based on the first image feature and the first text feature.

[0095] In some cases, since there is a correspondence between the image frame and the text in the image-text pair, in order to make the extracted image features contain the feature information of the corresponding text, and the features of the corresponding text contain the feature information of the image, to achieve information fusion and understanding in the two modalities of image and text, the features of the two modalities have useful information about each other. Therefore, in this embodiment, the image encoding branch and the text encoding branch of the second encoding network can be obtained by joint training. The second encoding network is trained using the image-text pairs with a corresponding relationship, so that the image encoding branch and the text encoding branch can fuse and understand the information of the two modalities when processing the image frame and its corresponding text.

[0096] Specifically, the first image-text pair and the second image-text pair are input into the second encoding network together. For the first image-text, the first image-text pair includes a first image and a first text corresponding to the first image. The second encoding network includes an image encoding branch and a text encoding branch obtained by joint training. The image encoding branch is an image encoder, and the image encoder is used to extract features of the first image to obtain first image features. The text encoding branch is a text encoder, and the text encoder is used to extract features of the first text to obtain first text features. The first image features and the first text features are combined to obtain a first feature pair.

[0097] For the second image text, the second image text pair includes the second image and the second text corresponding to the second image. The second image is subjected to feature extraction by an image encoder to obtain second image features. The second text is subjected to feature extraction by a text encoder to obtain second text features. The second image features and the second text features are combined to obtain a second feature pair.

[0098] For example, see Figure 4b , Figure 4bThe video understanding model provided by the embodiment of this specification is shown, and the video understanding model includes a second encoding network 202, a first encoding network 204, a sorting component 104, and a clustering component 106. Among them, the second encoding network can adopt the CLIP (Contrastive Language-Image Pre-Training Model) model. The second encoding network includes an image encoder (Vision Encoder) and a text encoder (Language Encoder) obtained by joint training. The first image-text pair and the second image-text pair are input into the video understanding model, and feature extraction is performed by the image encoder and the text encoder in the second encoding network to obtain the first feature pair and the second feature pair. The first feature pair and the second feature pair are input into the first encoding network to obtain the associated feature classification head of the first image, the associated feature classification head of the first text, the associated feature classification head of the second image, and the associated feature classification head of the second text. Based on the associated feature classification head of the first image and the associated feature classification head of the first text, the fusion feature of the first image-text pair is obtained. Based on the associated feature classification head of the second image and the associated feature classification head of the second text, the fusion feature of the second image-text pair is obtained. The fused features of the first image-text pair and the fused features of the second image-text pair are input into the sorting component for sorting processing, and the fused features of the first image-text pair and the fused features of the second image-text pair are input into the clustering component for clustering processing to achieve content understanding of the video file.

[0099] In the above implementation, the image encoding branch and the text encoding branch obtained through joint training can fuse and understand the information of the two modalities when processing image frames and their corresponding texts, obtain more comprehensive image text feature information, and more accurately understand the content of the video file.

[0100] In some embodiments, see Figure 5 , obtaining a first feature pair based on the first image feature and the first text feature may include the following steps:

[0101] S510 , performing mapping based on the first image feature to obtain an image intermediate classification head of the first image.

[0102] S520: Mapping is performed based on the first text feature to obtain a text intermediate classification head of the first text.

[0103] S530: Construct a first feature pair using the first image feature, the image intermediate classification head, the first text feature, and the text intermediate classification head.

[0104] Specifically, the first image is input into the image coding branch for feature extraction to obtain the first image feature. In order to summarize the image feature of the first image, the first image feature is mapped to obtain the image intermediate classification head of the first image. The first text is input into the image coding branch for feature extraction to obtain the first text feature. In order to summarize the text feature of the first text, the first text feature is mapped to obtain the text intermediate classification head of the first text. Based on the combination of the first image feature, the image intermediate classification head, the first text feature, and the text intermediate classification head, a first feature pair is obtained. Exemplarily, the first image feature is represented as (v 0,0 v 0,1 v 0,2 …), the image intermediate classification head is denoted as [CLS], and the first image feature is denoted as (w 0,0 w 0,1 w 0,2 …), the image intermediate classification head is denoted as [CLS], and the first feature pair is denoted as [[CLS]v 0,0 v 0, 1 v 0,2 …[CLS]w 0,0 w 0,1 w 0,2 …】.

[0105] In some implementations, understanding the content of a video file according to the arrangement order within the segment includes: determining a frame order of multiple image-text pairs according to the arrangement order within the segment; and understanding the content of the video file based on the frame order of multiple image-text pairs.

[0106] Specifically, if the number of video segment clusters is one, the intra-segment arrangement order of the video segment cluster is the frame order of multiple image-text pairs. If the number of video segment clusters is multiple, it is necessary to sort the multiple video segment clusters to obtain the order of the video segment clusters, and determine the frame order of multiple image-text pairs in combination with the order of the video segment clusters and the intra-segment arrangement order of the video segment clusters. The basic narrative structure of the video file is determined by the video segment clusters; within any video segment cluster, the visual semantic order of any video segment cluster is determined by the intra-segment arrangement order. Finally, the video content is understood based on the basic narrative structure of the video file and the visual semantic order of any video segment cluster.

[0107] In the above implementation, based on the video segment clusters and the segment arrangement order of the image-text pairs in the video segment clusters, the development of events in the video file and the storyline of the video file are understood, so as to complete the visual narrative reasoning task of the video file from the perspective of overall video understanding.

[0108] In some implementations, any two of the plurality of image-text pairs include a first image-text pair and a second image-text pair. The number of video segment clusters is multiple. Figure 6a Before determining the frame order of the plurality of image-text pairs according to the arrangement order within the segment, the method may further include:

[0109] S610: Perform binary classification processing on the fusion features of the first image-text pair and the fusion features of the second image-text pair to obtain the frame sequence of any two image-text pairs.

[0110] In some cases, related technologies use visual recognition and classification to understand videos. On the one hand, the methods used in related technologies require large computing resources. On the other hand, the methods used in related technologies ignore the overall understanding of the video and the visual narrative structure of the video file. Therefore, in this embodiment, binary classification is applied to the field of video understanding. The frame order between image-text pairs is determined by binary classification, making the results of the video understanding model easier to interpret and understand. In particular, for a large number of video files, binary classification can speed up data processing.

[0111] Specifically, there is a certain frame order between the first image-text pair and the second image-text pair, for example, the first image-text pair is located in front of the second image-text pair, or the second image-text pair is located in front of the first image-text pair. In this embodiment, the image-text pair sorting problem is converted into a binary classification processing task, and the fusion features of any two image-text pairs obtained above are subjected to binary classification processing to obtain the frame order of the arbitrary two image-text pairs.

[0112] The frame order of multiple image-text pairs is determined according to the arrangement order within the segment, including:

[0113] S620: For any video segment cluster, based on the fusion features of the image-text pairs in any video segment cluster, sort the segment elements included in any video segment cluster to obtain the intra-segment arrangement order of the segment elements in any video segment cluster.

[0114] S630: Obtain a frame order of the plurality of image-text pairs based on an arrangement order of the plurality of video segment clusters, a frame order of any two image-text pairs, and an intra-segment arrangement order of segment elements in any video segment cluster.

[0115] Among them, the elements included in the video segmentation cluster are recorded as segmentation elements. The fusion features of the image-text pairs are clustered to obtain video segmentation clusters. Since different types of video files have different hierarchical structures, different clustering results are obtained. For example, clustering the fusion features of multiple image-text pairs in a movie file can obtain several scene clusters, and for each scene cluster, the scene cluster includes several storyboard clusters; that is, the video segmentation cluster can be a scene cluster, and the segmentation element can be a storyboard cluster. For example, clustering the fusion features of multiple image-text pairs in an animation video file can obtain several grid clusters, and each grid cluster includes several image-text pairs; that is, the video segmentation cluster can be a grid cluster, and the segmentation element can be an image-text pair.

[0116] Specifically, in any video segment cluster, if the hierarchical structure corresponding to the segment element is between the video segment cluster and the image-text pair, then the segment element includes a number of image-text pairs, and the video segment cluster includes a number of segment elements. Figure 6b , the hierarchical structure corresponding to the video file is video segmentation, segmentation element, image-text pair from top to bottom. Through clustering processing, similar image-text pairs are clustered in the same segmentation element, and similar segmentation elements are clustered in the same video segmentation cluster. It can be seen that the clustering processing completes the top-down reasoning, that is, the processing process from video segmentation to segmentation element and then to image-text pair. In order to obtain the frame order of multiple image-text pairs corresponding to the video file, the sorting processing is bottom-up reasoning, that is, the processing process from image-text pair to segmentation element to video segmentation. The following is an exemplary explanation:

[0117] 1) Sorting of image-text pair levels: Any two image-text pairs have been sorted. If any segmentation element includes three or more image-text pairs, any segmentation element is seen in the search space. The search algorithm can be used to sort the image-text pairs in any segmentation element based on the frame order of any two image-text pairs.

[0118] 2) Sorting of the segmentation element hierarchy: For any segmentation element, if the fusion features of the image-text pair in any segmentation element are known, the fusion features of the image-text pair included in any segmentation element can be fused again to obtain the element fusion features corresponding to the any segmentation element. Binary classification processing is performed based on the element fusion features corresponding to any two segmentation elements to obtain the arrangement order of any two segmentation elements. Similarly, based on the arrangement order of any two segmentation elements by the search algorithm, the arrangement order of segmentation elements within any video segmentation cluster is obtained, that is, the intra-segment arrangement order of segmentation elements in any video segmentation cluster.

[0119] 3) Sorting of video segment cluster levels: For any video segment cluster, if the fusion features of the image-text pairs in any video segment cluster are known, the fusion features of the image-text pairs in any video segment cluster can be further fused to obtain the cluster fusion features corresponding to the video segment cluster. Binary classification is performed based on the cluster fusion features corresponding to any two video segment clusters to obtain the arrangement order of any two video segment clusters. Similarly, based on the arrangement order of any two video segment clusters by the search algorithm, the arrangement order of video segment clusters in the video file is obtained.

[0120] In the above implementation, the top-down and bottom-up reasoning processes have been completed, the hierarchical structure of the video file and the frame order of multiple image-text pairs corresponding to the video file have been determined, and the content of the video file is understood based on the hierarchical structure of the video file and the frame order of multiple image-text pairs. The development process of events in the video file can be understood, and the storyline of the video file can be inferred, thereby completing the visual narrative reasoning task of the video file from the perspective of overall video understanding, thereby improving the accuracy of video understanding.

[0121] In some embodiments, see Figure 7 , based on the frame order of any two image-text pairs in any video segment cluster, sorting the segment elements included in any video segment cluster to obtain the intra-segment arrangement order of the segment elements in any video segment cluster may include the following steps:

[0122] S710: Determine the element order of any two segment elements in any video segment cluster; the element order corresponds to a confidence level.

[0123] S720: Construct an N*N weighted adjacency matrix based on the confidence level.

[0124] S730 , performing a beam search based on the weighted adjacency matrix and a preset beam width to obtain an arrangement order within the segment.

[0125] Among them, the matrix elements in the weighted adjacency matrix are used to represent the weights of the frame order between the two segmentation elements corresponding to the matrix elements. N is the number of segmentation elements included in any video segmentation cluster, and N is a positive integer. Beam width refers to the number of candidate solutions retained during the beam search process. Beam width is an important hyperparameter that determines the size of the search space and the search efficiency. A smaller beam width will reduce the search space, but may miss better solutions; a larger beam width will increase the search space, but will result in higher computational costs. Therefore, the appropriate beam width needs to be adjusted according to the specific problem and algorithm. In this embodiment, the preset beam width can be equal to 3 or 5.

[0126] In some cases, beam search is a technique used to reduce the amount of computation in large-scale searches. Beam search reduces the amount of computation by limiting the search space and retaining only a portion of candidate solutions while ensuring the quality of the results. Therefore, beam search is used to sort the elements of a segment to obtain the order within the segment.

[0127] Specifically, as mentioned above, the segmentation elements correspond to element fusion features. Based on the element fusion features corresponding to any two segmentation elements, binary classification processing is performed to obtain the arrangement order of any two segmentation elements. Further, the element order of any two segmentation elements in any video segmentation cluster can be determined. The element order corresponds to a confidence. If the number of segmentation elements included in any video segmentation cluster is recorded as N, the confidence corresponding to the element order is used to construct an N*N weighted adjacency matrix. The beam width is set in advance. In the beam search, we retain a beam with a size equal to the preset beam width. Each time a node is expanded, only the preset beam width nodes with the highest confidence are retained, and their child nodes continue to be expanded. Through beam search, the segmentation elements included in any video segmentation cluster are sorted to obtain the intra-segment arrangement order of the segmentation elements in any video segmentation cluster.

[0128] In this embodiment, by setting a reasonable beam width, the search space is reduced while ensuring the quality of the solution obtained. The accuracy of the arrangement order of segment elements in any video segment cluster is improved by beam search, which is beneficial to subsequent video understanding.

[0129] In some embodiments, the unit segment is a scene. Clustering is performed based on the fusion features of the image-text pairs to obtain video segment clusters, including: clustering is performed at the scene level according to the fusion features of each image-text pair to obtain scene clusters; wherein the video segment clusters are scene clusters, and the image-text pairs corresponding to the fusion features included in the scene clusters belong to the same scene.

[0130] Among them, the video file corresponds to multiple image-text pairs. The video file corresponds to several scenes, and the video segmentation cluster is a scene cluster. Therefore, by clustering according to the scene level corresponding to the video file, similar image-text pairs belonging to the same scene can be divided into the same scene cluster. Specifically, the fusion features of each image-text pair have been obtained, and clustering is performed based on the fusion features of each image-text pair at the scene level to obtain several scene clusters. The scene cluster includes several fusion features, and the image-text pairs corresponding to the several fusion features included in the scene cluster belong to the same scene. Exemplarily, a hierarchical clustering algorithm or a density-based clustering algorithm can be used to cluster the fusion features of the image-text pairs.

[0131] In this implementation, by clustering the fusion features of each image-text pair at the scene level, the hierarchical structure of the video file can be obtained, and the video can be understood from the perspective of the entire video.

[0132] In some embodiments, the unit segmentation is a storyboard; clustering is performed based on the fusion features of the image-text pairs to obtain a video segmentation cluster, including: clustering is performed at the storyboard level according to the fusion features of each image-text pair to obtain a storyboard cluster; wherein the video segmentation cluster is a storyboard cluster, and the image-text pairs corresponding to the fusion features included in the storyboard cluster belong to the same storyboard.

[0133] Among them, the video file corresponds to multiple image-text pairs. The video file corresponds to several storyboards, and the video segmentation cluster is a storyboard cluster. Therefore, clustering is performed according to the storyboard level corresponding to the video file, and similar image-text pairs belonging to the same storyboard can be divided into the same storyboard cluster. Specifically, the fusion features of each image-text pair have been obtained, and clustering is performed based on the fusion features of each image-text pair at the storyboard level to obtain several storyboard clusters. The storyboard cluster includes several fusion features, and the image-text pairs corresponding to the several fusion features included in the storyboard cluster belong to the same storyboard. Exemplarily, a hierarchical clustering algorithm or a density-based clustering algorithm can be used to cluster the fusion features of the image-text pairs.

[0134] In this implementation, by clustering the fusion features of each image-text pair at the storyboard level, the hierarchical structure of the video file can be obtained, and the video can be understood from the perspective of the entire video.

[0135] In some embodiments, see Figure 8 , the unit segmentation is scene and storyboard; clustering is performed based on the fusion features of image and text pairs to obtain video segmentation clusters, including:

[0136] S810: Clustering the fused features of each image-text pair at the scene level to obtain a scene cluster; wherein the image-text pairs corresponding to the fused features included in the scene cluster belong to the same scene.

[0137] S820. For any scene cluster, at the storyboard level, cluster each image-text pair in any scene feature cluster to obtain a storyboard cluster; wherein the video segmentation cluster is a scene cluster and a storyboard cluster, and the image-text pairs corresponding to the fusion features included in the storyboard cluster belong to the same storyboard.

[0138] In some cases, the hierarchical structure corresponding to the video file includes scenes and storyboards. The video file includes several scenes, one scene includes several storyboards, and one storyboard includes several image-text pairs; that is, the hierarchical structure of the video file is scene, storyboard, image-text pair from top to bottom. Therefore, it is necessary to perform clustering processing according to the hierarchical structure of the scene-storyboard-image-text pair of the video file, so as to cluster the image-text pairs belonging to the same scene together, and cluster the image-text pairs belonging to the same storyboard together, so that the clustering result can reflect the hierarchical structure of the scene-storyboard-image-text pair.

[0139] Specifically, firstly, since the image-text pairs belonging to the same scene have similarities, at the scene level, the fusion features of each image-text pair are clustered to obtain a scene cluster, and the image-text pairs corresponding to the fusion features included in the final scene cluster belong to the same scene. Secondly, for any scene cluster, the image-text pairs belonging to the same storyboard have similarities, so at the storyboard level, each image-text pair in any scene feature cluster is clustered to obtain a storyboard cluster; the image-text pairs corresponding to the fusion features included in the storyboard cluster belong to the same storyboard.

[0140] The above implementation method completes top-down reasoning through clustering processing, that is, the processing process from scene to storyboard and then to image-text pair, and obtains the hierarchical structure of scene-storyboard-image-text pairs of the video file, which provides a data basis for the bottom-up reasoning process and facilitates the subsequent accurate and efficient video understanding.

[0141] In some implementations, any two of the plurality of image-text pairs include a first image-text pair and a second image-text pair. The number of scene clusters is multiple. Figure 9a Before determining the frame order of the plurality of image-text pairs according to the arrangement order within the segment, the method may further include the following steps:

[0142] S910: Perform binary classification processing on the fusion features of the first image-text pair and the fusion features of the second image-text pair to obtain a frame sequence of any two image-text pairs.

[0143] Specifically, there is a certain frame order between the first image-text pair and the second image-text pair, for example, the first image-text pair is located in front of the second image-text pair, or the second image-text pair is located in front of the first image-text pair. In this embodiment, the image-text pair sorting problem is converted into a binary classification processing task, and the fusion features of any two image-text pairs obtained above are subjected to binary classification processing to obtain the frame order of the arbitrary two image-text pairs.

[0144] The frame order of multiple image-text pairs is determined according to the arrangement order within the segment, including:

[0145] S920, for any storyboard cluster, based on the frame order of any two image-text pairs in any storyboard cluster, sort the image-text pairs included in any storyboard cluster to obtain the storyboard frame order of the image-text pairs in any storyboard cluster;

[0146] S930, for any scene cluster, based on the order of any two storyboard clusters in any scene cluster, sort the storyboard clusters included in any scene cluster to obtain the storyboard order of the storyboard cluster pairs in any scene cluster;

[0147] S940, sorting multiple scene clusters based on the order of any two scene clusters to obtain a scene order;

[0148] S950: Determine a frame order of the plurality of image-text pairs based on a frame order within a storyboard, a storyboard order within a scene, and a scene order.

[0149] The above has completed the process from scene to storyboard and then to image-text pair, and obtained the hierarchical structure of scene-storyboard-image-text pair of the video file. In this embodiment, the sorting process is performed from bottom to top to obtain the frame order of multiple image texts corresponding to the video file.

[0150] Specifically, for any storyboard cluster, based on the frame order of any two image-text pairs in any storyboard cluster, the image-text pairs included in any storyboard cluster are sorted through a beam search algorithm to obtain the frame order of the image-text pairs in any storyboard cluster, that is, the frame order within the storyboard.

[0151] After the order of frames in the storyboard clusters is arranged, the multiple storyboard clusters in any scene cluster are sorted. The order of any two storyboard clusters can be determined by binary classification, and then based on the order of any two storyboard clusters in any scene cluster, the multiple storyboard clusters included in any scene cluster are sorted by a beam search algorithm to obtain the storyboard order of the storyboard cluster pairs in any scene cluster, that is, the storyboard order within the scene.

[0152] After the order of frames in the scene cluster is arranged, the multiple scene clusters are sorted. The order of any two scene clusters can be determined by binary classification, and then based on the order of any two scene clusters, the multiple scene clusters corresponding to the video file are sorted by a beam search algorithm to obtain the arrangement order of the scene clusters, that is, the scene order.

[0153] Finally, the frame order of multiple image-text pairs is determined by the frame order within the storyboard, the storyboard order within the scene, and the scene order obtained through the bottom-up reasoning process.

[0154] Exemplarily, the video file corresponds to image-text pairs P1, P2, P3, P4, P5, P6, P7, P8, P9, P10...Pn, etc., and the fusion features T1, T2, T3, T4, T5, T6, T7, T8, T9, T10...Tn corresponding to the image-text pairs P1, P2, P3, P4, P5, P6, P7, P8, P9, P10...Pn are determined. It should be noted that the order of the above image-text pairs may be out of order. The following exemplifies the top-down and bottom-up reasoning processes.

[0155] 1) Scene clustering: cluster the image-text pairs belonging to the same scene together to obtain several scene clusters. For example, some scene clusters are represented as: [P1, P3, P5, P6], [P2, P4, P7, P8, P9, P10].

[0156] 2) Storyboard clustering: clustering the image-text pairs belonging to the same storyboard together to obtain several storyboard clusters.

[0157] For scene cluster SN1 [P1, P3, P5, P6], the clustering result is [(P1, P3, P6)(P5)]; among them, the storyboard clusters include (P1, P3, P6) and (P5).

[0158] For scene cluster SN2 [P2, P4, P7, P8, P9, P10], the clustering processing result is [(P4, P9, P10, P8)(P2, P7)]; among them, the storyboard clusters include (P4, P9, P10, P8) and (P2, P7).

[0159] 3) Frame sorting: sorting each image-text pair in each frame cluster.

[0160] For the sorting of the storyboard cluster ST1 (P1, P3, P6), first, determine the order between any two frames in the storyboard cluster ST1 (P1, P3, P6), such as the frame sorting of (P1, P3), (P1, P6), (P3, P6), and after determining the order between any two frames, determine the order of the image-text pairs in the storyboard cluster ST1 based on the beam search algorithm, such as (P1 P6P3).

[0161] For the sorting of storyboard cluster ST2 (P4, P9, P10, P8), first, determine the order between any two frames in storyboard cluster ST2 (P4, P9, P10, P8), such as the frame sorting of (P4, P9), (P4, P10), (P4, P8), (P9, P10), (P9, P8), (P10, P8). After determining the order between any two frames, determine the order of image-text pairs in storyboard cluster ST2 based on the beam search algorithm, such as (P10 P9 P4 P8).

[0162] The storyboard cluster ST3 (P2, P7) is sorted to determine the order of the image-text pairs in the storyboard cluster ST3, for example (P2P7).

[0163] 4) Storyboard sorting: sort the storyboards under each scene cluster.

[0164] For scene cluster SN1 [P1, P3, P5, P6], it includes storyboard cluster ST1 (P1, P3, P6) and storyboard cluster ST4 (P5).

[0165] Based on the fusion features T1, T3, and T6 corresponding to P1, P3, and P6, feature fusion is performed to obtain the fusion feature of the storyboard cluster ST1. Obtain the fusion feature T5 corresponding to P5. Perform binary classification based on the fusion feature of the storyboard cluster ST1 and the fusion feature T5 corresponding to P5 to obtain the order between the storyboard cluster ST1 and the storyboard cluster ST4. For example, (ST4 ST1)

[0166] For scene cluster SN2 [P2, P4, P7, P8, P9, P10], it includes storyboard cluster ST2 (P4, P9, P10, P8) and storyboard cluster ST3 (P2, P7).

[0167] Based on the fusion features T4, T9, T10, and T8 corresponding to P4, P9, P10, and P8, feature fusion is performed to obtain the fusion features of the storyboard cluster ST2. Based on the fusion features T2 and T7 corresponding to P2 and P7, feature fusion is performed to obtain the fusion features of the storyboard cluster ST3. Based on the fusion features of the storyboard cluster ST2 and the fusion features of the storyboard cluster ST3, binary classification processing is performed to obtain the order between the storyboard cluster ST2 and the storyboard cluster ST3. For example, (ST2 ST3)

[0168] 5) Scene sorting: sorting the scenes corresponding to the video files.

[0169] For scene cluster SN1 [P1, P3, P5, P6], feature fusion is performed based on the fused features T1, T3, T5, T6 corresponding to P1, P3, P5, P6 respectively to obtain the fused features of scene cluster SN1.

[0170] For scene cluster SN2 [P2, P4, P7, P8, P9, P10], feature fusion is performed based on the fused features T2, T4, T7, T8, T9, T10 corresponding to P2, P4, P7, P8, P9, P10 respectively to obtain the fused features of scene cluster SN2.

[0171] Based on the fusion features of scene cluster SN1 and the fusion features of scene cluster SN2, binary classification processing is performed to obtain the order between scene cluster SN1 and scene cluster SN2, for example, (SN2 SN1).

[0172] Therefore, the arrangement order of the storyboard clusters (ST2 ST3 ST4 ST1) can be obtained through the order between scene cluster SN1 and scene cluster SN2 (SN2 SN1), the order between storyboard cluster ST2 and storyboard cluster ST3 (ST2 ST3), and the order between storyboard cluster ST1 and storyboard cluster ST4 (ST4 ST1).

[0173] Through the order of image-text pairs in storyboard cluster ST2 (P10 P9 P4 P8), the order of image-text pairs in storyboard cluster ST1 (P1 P6 P3), storyboard cluster ST4 (P5), and the order of image-text pairs in storyboard cluster ST1 (P1 P6 P3), we can obtain the order of image-text pairs in the video file as (P10 P9 P4 P8 P1 P6 P3 P5 P1 P6 P3).

[0174] At this point, the image-text pairs corresponding to the video files have been sorted through top-down and bottom-up reasoning processes.

[0175] In the above implementation, the top-down and bottom-up reasoning processes have been completed to determine the hierarchical structure of the video file (see Figure 9b Based on the hierarchical structure of video files (scene-storyboard-frame) and the frame order of multiple image-text pairs corresponding to the video files, the content of the video files can be understood, the development process of events in the video files can be understood, the storyline of the video files can be inferred, and the visual narrative reasoning task of the video files can be completed from the perspective of overall video understanding, thereby improving the accuracy of video understanding.

[0176] This specification provides a plot prediction method. Fig.10 , the plot prediction method comprises the following steps:

[0177] S1010. Acquire multiple image-text pairs corresponding to target video segments included in a video file; wherein the image-text pairs include image frames and texts corresponding to the image frames; and the video file includes multiple video segments.

[0178] S1020: Perform feature extraction and feature fusion on the image frame and text in the image-text pair to obtain fusion features of the image-text pair.

[0179] S1030: Extract features of the plot text corresponding to the video clip to obtain plot features of the video clip.

[0180] S1040: Perform feature matching based on the fusion features of the image-text pair and the plot features of the video clip to predict the target plot text corresponding to the target video clip.

[0181] Among them, plot prediction refers to the use of machine learning or artificial intelligence technology. Specifically, multiple image-text pairs corresponding to the target video clip are input into the plot prediction model, and the image frames and texts in the image-text pairs are subjected to feature extraction and feature fusion to obtain the fusion features of the image-text pairs. By extracting and fusion features of the image frames and texts in the image-text pairs, the fusion features of the image-text pairs are obtained; the feature information of the target video clip is extracted in a multimodal form to provide a comprehensive basis for subsequent plot prediction. The video file corresponds to a script, and the video file is divided into multiple video clips, and the video clips correspond to plot texts. For the plot text corresponding to each video clip, feature extraction is performed to obtain the video clip plot features corresponding to each video clip. For each video clip, the video clip plot features and the fusion features of the image-text pairs are used to perform feature matching, and the target plot text corresponding to the target video clip is determined in the script corresponding to the video file.

[0182] The above-mentioned plot prediction method predicts the target plot text corresponding to the target video clip by matching the fusion features of the image-text pair corresponding to the video file with the plot features of the video clip, thereby realizing plot prediction based on the video file and the script. It should be noted that the process of extracting the fusion features of the image-text pair by the plot prediction model in this embodiment is the same as the process of extracting the fusion features of the image-text pair by the video understanding model in the above text. The video understanding model in the above text can be a pre-trained model, and the plot prediction model in this embodiment can be obtained by training using a small sample based on the video understanding model.

[0183] In some embodiments, the video file includes multiple scenes, and the target video clip is the target scene. Feature matching is performed based on the fusion features of the image-text pair and the plot features of the video clip to obtain the target plot text corresponding to the target video clip, including: feature matching is performed based on the fusion features of the multiple image-text pairs and the plot features of the scene to predict the target plot text corresponding to the target scene.

[0184] Specifically, a video file includes multiple scenes, and multiple image-text pairs in any scene can be input into the plot prediction model. Feature extraction and feature fusion are performed on the image frames and texts in the image-text pairs to obtain the fusion features of the image-text pairs. There are plot texts corresponding to the scenes. For the plot text corresponding to each scene, feature extraction is performed on it to obtain the plot features of each scene. For each scene, feature matching is performed using its plot features and the fusion features of the image-text pairs to determine the target plot text corresponding to the target scene in the script corresponding to the video file.

[0185] This specification provides a video question-and-answer method. Fig.11 , the video question answering method comprises the following steps:

[0186] S1110. Acquire multiple image-text pairs corresponding to the video file; wherein the image-text pairs include image frames and texts corresponding to the image frames.

[0187] S1120, performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair; wherein a question text and a plurality of option texts corresponding to the question text are provided based on the video file.

[0188] S1130. Perform feature extraction based on the question text and each option text to obtain question option features corresponding to each option text.

[0189] S1140, performing feature matching based on the fusion features of the image-text pair and the question option features to obtain a feature matching result.

[0190] S1150. Determine the correct option text corresponding to the question text from multiple option texts based on the feature matching result.

[0191] The abbreviation of video question answering is VideoQA, which refers to the use of machine learning and natural language processing technology to enable computers to understand video content and answer questions raised by users. In the field of video question answering, question text and multiple option texts corresponding to the question text are provided for video files.

[0192] Specifically, multiple image-text pairs can be extracted from a video file, wherein the image-text pairs include image frames and texts corresponding to the image frames. Multiple image-text pairs are input into a video question-answering model, and feature extraction and feature fusion are performed on the image frames and texts in the image-text pairs to obtain fusion features of the image-text pairs. The question text and each option text are combined, and feature extraction is performed on the combined text to obtain question option features corresponding to each option text. Feature matching is performed on the fusion features of the image-text pairs and the question option features to obtain feature matching results. According to the feature matching results, the correct option text corresponding to the question text is determined from the multiple option texts.

[0193] Exemplarily, a question Q is provided for a video file V, and question Q corresponds to two options A and B. Question Q is combined with option A for text and feature extraction to obtain question option feature F1 corresponding to option A. Question Q is combined with option B for text and feature extraction to obtain question option feature F2 corresponding to option B. Feature matching is performed on question option feature F1 and the fusion feature of the image-text pair to obtain feature matching result X1. Feature matching is performed on question option feature F2 and the fusion feature of the image-text pair to obtain feature matching result X2. If feature matching result X2 is better than feature matching result X1, option B corresponding to feature matching result X2 is taken as the correct option and returned to the user.

[0194] It should be noted that the process of extracting the fusion features of image-text pairs by the video question-answering model in this embodiment is the same as the process of extracting the fusion features of image-text pairs by the video understanding model in the above text. The video understanding model in the above text can be a pre-trained model, and the video question-answering model in this embodiment can be trained using a small sample based on the video understanding model. The video question-answering model in this embodiment can be applied to video search, smart home, online education and other fields, which helps to improve user experience and work efficiency.

[0195] The embodiment of this specification also provides a video understanding device, see Fig.12 , the video understanding device 1200 may include:

[0196] The image-text pair acquisition module 1210 is used to acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame;

[0197] A feature extraction and fusion module 1220 is used to extract and fuse features of the image frame and text in the image-text pair to obtain fusion features of the image-text pair;

[0198] A feature clustering processing module 1230 is used to perform clustering processing based on the fusion features of the image-text pairs to obtain video segment clusters; wherein the image-text pairs corresponding to the fusion features included in the video segment clusters belong to the same unit segment;

[0199] An arrangement order determination module 1240 is used to sort the image-text pairs in the video segment cluster to obtain an intra-segment arrangement order of the image-text pairs in the video segment cluster;

[0200] The video understanding module 1250 is used to understand the content of the video file according to the arrangement order within the segments.

[0201] The present specification also provides a plot prediction device, see Fig.13 , the plot prediction device 1300 comprises:

[0202] The image-text pair acquisition module 1310 is used to acquire a plurality of image-text pairs corresponding to a target video segment included in a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; and the video file includes a plurality of video segments;

[0203] A feature extraction and fusion module 1320 is used to extract and fuse features of the image frame and text in the image-text pair to obtain fusion features of the image-text pair;

[0204] A plot feature extraction module 1330 is used to extract features of the plot text corresponding to the video clip to obtain plot features of the video clip;

[0205] The plot text matching module 1340 is used to perform feature matching based on the fusion features of the image-text pair and the plot features of the video clip to obtain the target plot text corresponding to the target video clip.

[0206] This specification also provides a video question-and-answer device. Fig.14 , the video question-answering device 1400 includes:

[0207] The image-text pair acquisition module 1410 is used to acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame;

[0208] The feature extraction and fusion module 1420 is used to extract and fuse features of the image frame and the text in the image-text pair to obtain fusion features of the image-text pair; wherein a question text and a plurality of option texts corresponding to the question text are provided based on the video file;

[0209] Option feature extraction module 1430 performs feature extraction based on the question text and each of the option texts to obtain question option features corresponding to each of the option texts;

[0210] Option feature matching module 1440, used to perform feature matching based on the fusion feature of the image-text pair and the question option feature to obtain a feature matching result;

[0211] The correct option determination module 1450 is used to determine the correct option text corresponding to the question text from the multiple option texts according to the feature matching result.

[0212] The embodiments of the present specification also provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer executes the method in any of the above embodiments.

[0213] The embodiments of this specification also provide a computer program product including instructions, which, when executed by a computer, enables the computer to execute the method in any of the above embodiments.

[0214] It should be understood that the specific examples herein are only intended to help those skilled in the art to better understand the embodiments of the present specification, rather than to limit the scope of the present invention.

[0215] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0216] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0217] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0218] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0219] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0220] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0221] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0222] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0223] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0224] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0225] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0226] The above is only a specific implementation of this specification, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in this specification, which should be included in the protection scope of this specification. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A video understanding method, It is characterized in that The method comprises: Acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; Performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair; Clustering is performed based on the fusion features of the image-text pairs to obtain video segment clusters; wherein the image-text pairs corresponding to the fusion features included in the video segment clusters belong to the same unit segment of the video file; Sorting the image-text pairs in the video segment cluster to obtain an intra-segment arrangement order of the image-text pairs in the video segment cluster; The content of the video file is understood according to the arrangement order within the segments.

2. The method according to claim 1, It is characterized in that The step of extracting and fusing features of the image frame and the text in the image-text pair to obtain fused features of the image-text pair includes: Performing feature extraction on the image frame and the text in the image-text pair to obtain an image classification header of the image frame and a text classification header of the text; The image classification head and the text classification head are subjected to feature fusion to obtain fusion features of the image-text pair.

3. The method according to claim 2, It is characterized in that The multiple image-text pairs include a first image-text pair and a second image-text pair, the first image-text pair includes a first image and a first text corresponding to the first image, and the second image-text pair includes a second image and a second text corresponding to the second image; the feature extraction of the image frame and the text in the image-text pair to obtain the image classification header of the image frame and the text classification header of the text includes: Performing feature extraction on the first image-text pair to obtain a first image feature of the first image and a first text feature of the first text to form a first feature pair; Performing feature extraction on the second image-text pair to obtain a first image feature of the second image and a second text feature of the second text to form a second feature pair; The first feature pair and the second feature pair are cross-fused and feature extracted through a first encoding network to obtain an associated feature classification head of the first image, an associated feature classification head of the first text, an associated feature classification head of the second image, and an associated feature classification head of the second text; wherein the image classification head of the image frame includes an associated feature classification head of the first image and an associated feature classification head of the second image; and the text classification head includes an associated feature classification head of the first text and an associated feature classification head of the second text.

4. The method according to claim 3, It is characterized in that The second feature pair is obtained in the same manner as the first feature pair; the first feature pair is obtained in the same manner as the first feature pair, including: Inputting the first image-text pair into a second encoding network; wherein the second encoding network includes an image encoding branch and a text encoding branch obtained by joint training; Extracting features of the first image by using the image encoding branch to obtain features of the first image; Extracting features of the first text by using the text encoding branch to obtain the first text features; The first feature pair is obtained based on the first image feature and the first text feature.

5. The method according to claim 4, It is characterized in that The obtaining the first feature pair based on the first image feature and the first text feature includes: Mapping is performed based on the first image feature to obtain an image intermediate classification head for the first image; Mapping is performed based on the first text feature to obtain a text intermediate classification head of the first text; The first feature pair is constructed by combining the first image feature, the image intermediate classification head, the first text feature, and the text intermediate classification head.

6. The method according to claim 1, It is characterized in that The understanding of the content of the video file according to the arrangement order in the segments includes: Determining the frame order of the plurality of image-text pairs according to the arrangement order within the segment; The content of the video file is understood based on the frame sequence of the multiple image-text pairs.

7. The method according to claim 6, It is characterized in that Any two of the plurality of image-text pairs include a first image-text pair and a second image-text pair; the number of the video segment clusters is multiple; before determining the frame order of the plurality of image-text pairs according to the arrangement order within the segment, the method further includes: Performing binary classification processing on the fusion features of the first image-text pair and the fusion features of the second image-text pair to obtain a frame sequence of the arbitrary two image-text pairs; The step of determining the frame order of the plurality of image-text pairs according to the arrangement order within the segments comprises: For any video segment cluster, based on the fusion features of the image-text pairs in the any video segment cluster, the segment elements included in the any video segment cluster are sorted to obtain the intra-segment arrangement order of the segment elements in the any video segment cluster; The frame order of the plurality of image-text pairs is obtained based on the arrangement order of the plurality of video segment clusters, the frame order of the arbitrary two image-text pairs, and the intra-segment arrangement order of the segment elements in any of the video segment clusters.

8. The method according to claim 7, It is characterized in that The step of sorting the segment elements included in any video segment cluster based on the frame order of any two image-text pairs in any video segment cluster to obtain the intra-segment arrangement order of the segment elements in any video segment cluster includes: Determine the element order of any two segment elements in any video segment cluster; the element order corresponds to a confidence level; Constructing an N*N weighted adjacency matrix based on the confidence; wherein the matrix elements in the weighted adjacency matrix are used to represent the weight of the frame order between two segmentation elements corresponding to the matrix elements; N is the number of segmentation elements included in any video segmentation cluster, and N is a positive integer; A beam search is performed based on the weighted adjacency matrix and a preset beam width to obtain the intra-segment arrangement order.

9. The method according to claim 1, It is characterized in that The unit segment is a scene; the clustering process based on the fusion features of the image-text pair is performed to obtain a video segment cluster, including: Clustering is performed on the scene level according to the fusion features of each of the image-text pairs to obtain a scene cluster; wherein the video segment cluster is the scene cluster, and the image-text pairs corresponding to the fusion features included in the scene cluster belong to the same scene.

10. The method according to claim 1, It is characterized in that The unit segment is a storyboard; the clustering process based on the fusion features of the image-text pair is performed to obtain a video segment cluster, including: Clustering is performed on the storyboard level according to the fusion features of each of the image-text pairs to obtain a storyboard cluster; wherein the video segment cluster is the storyboard cluster, and the image-text pairs corresponding to the fusion features included in the storyboard cluster belong to the same storyboard.

11. The method according to claim 6, It is characterized in that The unit segmentation is a scene and a storyboard; the clustering process is performed based on the fusion features of the image-text pair to obtain a video segment cluster, including: At the scene level, clustering is performed on the fusion features of each of the image-text pairs to obtain a scene cluster; wherein the image-text pairs corresponding to the fusion features included in the scene cluster belong to the same scene; For any scene cluster, at the storyboard level, clustering processing is performed on each image-text pair in any scene feature cluster to obtain a storyboard cluster; wherein the video segmentation cluster is the scene cluster and the storyboard cluster, and the image-text pairs corresponding to the fusion features included in the storyboard cluster belong to the same storyboard.

12. The method according to claim 11, It is characterized in that Any two of the plurality of image-text pairs include a first image-text pair and a second image-text pair; the number of the scene clusters is multiple; before determining the frame order of the plurality of image-text pairs according to the arrangement order within the segment, the method further includes: Performing binary classification processing on the fusion features of the first image-text pair and the fusion features of the second image-text pair to obtain a frame sequence of the arbitrary two image-text pairs; The step of determining the frame order of the plurality of image-text pairs according to the arrangement order within the segments comprises: For any storyboard cluster, based on the frame order of any two image-text pairs in the any storyboard cluster, the image-text pairs included in the any storyboard cluster are sorted to obtain the storyboard frame order of the image-text pairs in the any storyboard cluster; For any scene cluster, based on the order of any two storyboard clusters in the any scene cluster, the storyboard clusters included in the any scene cluster are sorted to obtain the storyboard order of the storyboard cluster pairs in the any scene cluster; Based on the order of any two scene clusters, sorting the multiple scene clusters to obtain a scene order; The frame order of the plurality of image-text pairs is determined based on the frame order within the storyboard, the storyboard order within the scene, and the scene order.

13. A plot prediction method, It is characterized in that The method comprises: Acquire a plurality of image-text pairs corresponding to a target video segment included in a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; and the video file includes a plurality of video segments; Performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair; Extracting features of the plot text corresponding to the video clip to obtain plot features of the video clip; Feature matching is performed based on the fusion features of the image-text pair and the plot features of the video clip to predict the target plot text corresponding to the target video clip.

14. The method according to claim 13, It is characterized in that The video file includes a plurality of scenes, and the target video clip is a target scene; the feature matching based on the fusion feature of the image-text pair and the plot feature of the video clip to obtain the target plot text corresponding to the target video clip includes: Feature matching is performed based on the fusion features of the multiple image-text pairs and the plot features of the scene to predict the target plot text corresponding to the target scene.

15. A video question-answering method, It is characterized in that The method comprises: Acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; Performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair; wherein a question text and a plurality of option texts corresponding to the question text are provided based on the video file; Perform feature extraction based on the question text and each of the option texts to obtain question option features corresponding to each of the option texts; Perform feature matching based on the fusion features of the image-text pair and the question option features to obtain a feature matching result; The correct option text corresponding to the question text is determined from the multiple option texts according to the feature matching result.

16. A video understanding device, It is characterized in that The device comprises: An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; A feature extraction and fusion module, used to extract and fuse features of the image frame and text in the image-text pair to obtain fusion features of the image-text pair; A feature clustering processing module, used for performing clustering processing based on the fusion features of the image-text pairs to obtain video segment clusters; wherein the image-text pairs corresponding to the fusion features included in the video segment clusters belong to the same unit segment; An arrangement order determination module is used to sort the image-text pairs in the video segment cluster to obtain an intra-segment arrangement order of the image-text pairs in the video segment cluster; The video understanding module is used to understand the content of the video file according to the arrangement order in the segments.

17. A plot prediction device, It is characterized in that The device comprises: An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to a target video segment included in a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; and the video file includes a plurality of video segments; A feature extraction and fusion module, used for performing feature extraction and feature fusion on the image frame and the text in the image-text pair to obtain fusion features of the image-text pair; A plot feature extraction module is used to extract features of the plot text corresponding to the video clip to obtain plot features of the video clip; The plot text matching module is used to perform feature matching based on the fusion features of the image-text pair and the plot features of the video clip to obtain the target plot text corresponding to the target video clip.

18. A video question-answering device, It is characterized in that The device comprises: An image-text pair acquisition module, used to acquire a plurality of image-text pairs corresponding to a video file; wherein the image-text pairs include an image frame and text corresponding to the image frame; A feature extraction and fusion module is used to extract and fuse features of the image frame and text in the image-text pair to obtain fusion features of the image-text pair; wherein a question text and a plurality of option texts corresponding to the question text are provided based on the video file; An option feature extraction module performs feature extraction based on the question text and each of the option texts to obtain a question option feature corresponding to each of the option texts; An option feature matching module is used to perform feature matching based on the fusion feature of the image text pair and the question option feature to obtain a feature matching result; The correct option determination module is used to determine the correct option text corresponding to the question text from the multiple option texts according to the feature matching result.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 14 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 14 are implemented.