Video question answering processing method and related apparatus

Abstract

The application discloses a video question and answer processing method and related devices, wherein the method comprises: extracting M segment features and a plurality of frame features of a target video; performing joint feature extraction and semantic feature extraction on a target frame in the M video segments to obtain video role features and target semantic features; performing question text feature extraction on a question text input of the target video to obtain question text features; fusing the M segment features and the plurality of frame features with the target semantic features to obtain fused video features, and then fusing the fused video features and the video role features based on the question text features to obtain a video feature vector; inputting the video feature vector and the question text features into a video feature encoding layer to obtain a preset classification to which the video feature belongs, and determining a question answer of the question text according to the preset classification. The application improves the accuracy and generalization of the video question and answer result by realizing the fusion of multi-modal information in the same feature space.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a video question-answering processing method and related devices. Background Art

[0002] In recent years, video question answering (VQA) has garnered increasing attention as a key application of multimodal machine learning. Video question answering tasks require understanding and answering text questions in conjunction with the video content. Unlike image question answering, the information provided by videos is temporally sequential, and video images cannot be modeled as independent frames. Furthermore, this task lacks large-scale pre-trained data, making it impossible to use multimodal pre-trained models to unify information from different modalities into the same feature space. Therefore, effectively aligning the information provided by the video and the question text becomes a key challenge in solving video question answering. Existing technologies typically focus on extracting video features, modeling frames and segments, or using attention mechanisms to capture contextual information. However, they do not adequately address the technical challenges of integrating information from different modalities into the same feature space. Consequently, the resulting question answering results from video question answering are inaccurate or lack generalization capabilities. Summary of the Invention

[0003] The embodiments of the present application provide a video question-answering processing method and related devices, which achieve the fusion of multimodal information in the same feature space by fusing the video semantic features, video frame features, and question text features of the video, thereby improving the accuracy and generalization of the video question-answering results.

[0004] In a first aspect, a video question-answering processing method is provided, the method comprising:

[0005] Obtain a target video and segment the target video to obtain M video segments;

[0006] Perform feature extraction on each of the M video clips to obtain M clip features;

[0007] Extract features from the target video in frames to obtain multiple frame features;

[0008] Perform joint feature extraction on the target frame in the M video clips to obtain the video character features, where the target frame is any frame in the video clip;

[0009] Extract semantic features of target frames in M ​​video clips to obtain target semantic features;

[0010] Fusing target segment features, target frame features, video character features, and target semantic features to obtain fused video features;

[0011] Receive input text of questions about the target video and obtain question text features;

[0012] The fused video features and question text features are input into the video feature encoding layer to obtain the preset classification to which the video features belong, and the answer to the question text is determined based on the preset classification.

[0013] In combination with the first aspect, in a possible design, semantic features are extracted from target frames in the M video clips to obtain target semantic features, including:

[0014] The target frames of M video clips are input into the image description model for processing to obtain the image description of the target frame of each video clip; the image description of the target frame of each video clip is segmented, and the classification identifier CLS is added before the segmentation result, and the segmentation result and CLS are input into the pre-trained language model for encoding to obtain the features of each segmentation and the features of CLS in the image description of the target frame of each video clip; the target semantic features include the features of the CLS corresponding to the target frames in the M video clips.

[0015] In combination with the first aspect, in a possible design, joint feature extraction is performed on target frames in M ​​video clips to obtain video character features, including:

[0016] The target frames of M video clips are input into the image description model for processing to obtain the image description of the target frame of each video clip; the image description of the target frame of each video clip is input into the text parser for role extraction, and the extracted results are used to generate a role structure graph corresponding to the target frame of each video clip according to the semantic structure. The role structure graph includes nodes identifying different roles and edge information between nodes; multiple role structure graphs are merged according to the same nodes to obtain the target role structure graph, and the target role structure graph is input into the graph convolutional network for encoding to obtain video role features.

[0017] In combination with the first aspect, in one possible design, M segment features and multiple frame features are fused with target semantic features to obtain fused video features, including:

[0018] Map M segment features, multiple frame features, and target semantic features to multiple vector spaces respectively; input the vector space mapped to the M segment features or multiple frame features and the vector space mapped to the target semantic features into the attention mechanism network, wherein the vector space inputted by the M segment features or multiple frame features and the vector space inputted by the target semantic features form a three-dimensional vector space K, Q, and V; obtain the output vector processed by the attention mechanism network, and determine the output vector related to the segment features or frame features in the output vector as the fused video feature.

[0019] In combination with the first aspect, in a possible design, the video features and the video character features fused based on the question text features are fused to obtain a video feature vector, including:

[0020] Perform linear mapping on the question text features to obtain multiple weight vectors. The number of weight vectors is the same as the number of fused feature vectors and the number of vector types of video role features, and the sum of the weight vectors is 1. The fused feature vector and the video role features are weighted and summed with their corresponding weight vectors to obtain the video feature vector.

[0021] In conjunction with the first aspect, in one possible design, the video feature vector and the question text feature are input into the video feature encoding layer to obtain the preset category to which the video feature belongs, and the answer to the question text is determined based on the preset category, including:

[0022] The video feature vector is spliced ​​with the question text feature to obtain a spliced ​​feature; the spliced ​​feature is input into the video feature encoding layer to classify the spliced ​​feature to obtain the probability that the spliced ​​feature belongs to each preset category; based on the target preset category with the highest probability of the spliced ​​feature belonging to, the target question answer corresponding to the target preset category is determined, and the target question answer is determined as the question answer of the question text.

[0023] In a second aspect, an electronic device is provided, the device comprising:

[0024] An acquisition unit, configured to acquire a target video and segment the target video to obtain M video segments;

[0025] The feature extraction unit is configured to perform feature extraction on each of the M video clips to obtain M clip features; perform feature extraction on the target video frame by frame to obtain multiple frame features; perform joint feature extraction on the target frame in the M video clips to obtain video character features, where the target frame is any frame in the video clips; perform semantic feature extraction on the target frame in the M video clips to obtain target semantic features; obtain a question text input for the target video, and perform feature extraction on the question text to obtain question text features;

[0026] A feature fusion unit is used to fuse M segment features and multiple frame features with target semantic features to obtain fused video features, and then fuse the fused video features with video character features based on question text features to obtain a video feature vector;

[0027] The classification unit is used to input the video feature vector and the question text feature into the video feature encoding layer, obtain the preset classification to which the video feature belongs, and determine the answer to the question text according to the preset classification.

[0028] In combination with the second aspect, in a possible design, the feature extraction unit is specifically used to: input the target frames of M video clips into the image description model for processing to obtain the image description of the target frame of each video clip; segment the image description of the target frame of each video clip, and add a classification identifier CLS before the segmentation result, and input the segmentation result and CLS into the pre-trained language model for encoding to obtain the features of each segmentation and the features of the CLS in the image description of the target frame of each video clip; the target semantic features include the features of the CLS corresponding to the target frames in the M video clips.

[0029] In combination with the second aspect, in a possible design, the feature extraction unit is specifically used to: input the target frames of M video clips into the image description model for processing to obtain an image description of the target frame of each video clip; input the image description of the target frame of each video clip into the text parser for role extraction, and generate a role structure graph corresponding to the target frame of each video clip according to the semantic structure of the extracted results, wherein the role structure graph includes nodes identifying different roles and edge information between nodes; merge multiple role structure graphs according to the same nodes to obtain a target role structure graph, and input the target role structure graph into the graph convolutional network for encoding to obtain video role features.

[0030] In combination with the second aspect, in a possible design, the feature fusion unit is specifically used to: map M segment features, multiple frame features and target semantic features to multiple vector spaces respectively; input the vector space mapped to the M segment features or multiple frame features and the vector space mapped to the target semantic features into the attention mechanism network, wherein the vector space inputted by the M segment features or multiple frame features and the vector space inputted by the target semantic features constitute a three-dimensional vector space K, Q, V; obtain the output vector processed by the attention mechanism network, and determine the output vector in the output vector that is related to the segment features or frame features as the fused video feature.

[0031] In combination with the second aspect, in a possible design, the feature fusion unit is specifically used to: perform linear mapping on the question text features to obtain multiple weight vectors, the number of weight vectors is the same as the number of fused feature vectors and the number of vector types of video role features, and the sum of the weight vectors is 1; the fused feature vector and the video role feature are respectively weighted and summed with their corresponding weight vectors to obtain a video feature vector.

[0032] In combination with the second aspect, in a possible design, the classification unit is specifically used to: splice the video feature vector with the question text feature to obtain a spliced ​​feature; input the spliced ​​feature into the video feature encoding layer to classify the spliced ​​feature to obtain the probability that the spliced ​​feature belongs to each preset category; determine the target question answer corresponding to the target preset category based on the target preset category with the highest probability that the spliced ​​feature belongs to, and determine the target question answer as the question answer to the question text.

[0033] In a third aspect, an electronic device is provided, the structure of which includes a processor, an input device, and may also include a memory; the processor is coupled to the input device and the memory, and can be used to combine the input information of the input device to execute the computer program instructions stored in the memory, so that the device executes the method of the first aspect or any aspect of the first aspect.

[0034] In a fourth aspect, a chip system is provided, comprising: a processor, the processor being coupled to an interface circuit and a memory, the interface circuit being used to exchange code instructions with the processor, the memory being used to store programs or instructions, and when the program or instructions are executed by the processor, the chip system implements the method described in the first aspect or any aspect of the first aspect in combination with the output information.

[0035] Optionally, there may be one or more processors in the chip system, and the processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0036] Optionally, the memory in the chip system may be one or more memories. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in this application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. This application does not specifically limit the type of memory or the configuration of the memory and the processor.

[0037] In a fifth aspect, a readable storage medium is provided for storing instructions, which, when executed, enables the method of the first aspect or any aspect of the first aspect to be implemented.

[0038] In a sixth aspect, a computer program product is provided. When a computer reads and executes the computer program product, the computer executes the method of the first aspect or any aspect of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments.

[0040] Figure 1A A schematic diagram of a video question-and-answer scenario provided in an embodiment of the present application;

[0041] Figure 1B This is a flow chart of a method for improving the accuracy of video question answering based on a multimodal fusion model;

[0042] Figure 1C A schematic diagram of a VIOLET model framework provided in an embodiment of the present application;

[0043] Figure 2A A flow chart of a video question-answering processing method is proposed for the embodiment of this application;

[0044] Figure 2B A schematic diagram of a process for obtaining fragment features provided in an embodiment of the present application;

[0045] Figure 2C A schematic diagram of a process for obtaining frame features provided in an embodiment of the present application;

[0046] Figure 2D A schematic diagram of a process for obtaining frame features through pooling provided in an embodiment of the present application;

[0047] Figure 2E A schematic diagram of a video character feature extraction process provided in an embodiment of the present application;

[0048] Figure 2F A schematic diagram of a process for obtaining a target role structure diagram provided in an embodiment of the present application;

[0049] Figure 2G A schematic diagram of a process for extracting semantic features provided in an embodiment of the present application;

[0050] Figure 2H A schematic diagram of a process for obtaining fused video features provided in an embodiment of the present application;

[0051] Figure 2I A schematic diagram of another process for obtaining fused video features provided in an embodiment of the present application;

[0052] Figure 2J A block diagram of a model for video question-answering processing provided in an embodiment of the present application;

[0053] Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0054] Figure 4This is a schematic diagram of the hardware structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The terms "first," "second," "third," and "fourth," etc., in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, rather than to describe a specific order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0056] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0057] "Multiple" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0058] First, the application scenario and system architecture of the embodiment of the present application are introduced with reference to the diagrams.

[0059] See also Figure 1A , Figure 1A A video question-answering scenario diagram provided in an embodiment of the present application is shown in FIG. Figure 1A As shown, the embodiments of the present application can be applied to electronic devices such as mobile phones, car computers, and smart homes that include cameras and support voice or text input functions, and can understand and answer user questions input by voice or text based on the visual information provided by the camera. The core devices involved in the present invention are video input devices, text input devices, and processing devices, wherein the text input device can be replaced by a voice input device for voice-text recognition. For example, a home video is taken with a camera, and the question "Help me find out where the cat is?" is input by a text or voice input device. The video input device and the text input device are respectively connected to the processing device, and the processing device processes the taken home video and the received input question to obtain an output result, such as: on the bed. The video input device, text input device, and processing device can be independent devices, or as Figure 1AAs shown in , they are all integrated into the same electronic device. This embodiment of the application does not make specific limitations.

[0060] The following introduces the existing related technologies related to the embodiments of the present application.

[0061] Existing video question-answering technologies fall into two categories: pre-trained models that leverage large-scale video-text alignment data, and non-pre-trained models that differ primarily in how they extract video features and fuse them with text features. The typical network architecture first segments the video and text into tokens, then uses a transformer encoder to encode them. Finally, the features are classified to generate the answer to the video question. Existing technologies differ primarily in model details and feature processing methods.

[0062] Prior Art 1 introduces a method for improving the accuracy of video question answering tasks by using a multimodal fusion model of video subtitles. This method first extracts visual features and subtitle features from video data using a region convolutional neural network (RCNN). This feature is then embedded into the question feature and each candidate answer feature and fed into a bidirectional encoder representation from transformers (BERT) language model to calculate the confidence of each candidate answer. The specific process of this method can be found in Figure 1B , Figure 1B This is a flow chart of a method for improving the accuracy of video question answering based on a multimodal fusion model.

[0063] This method has the following defects: (1) It requires the video to have subtitles, which greatly limits the scope of applicable videos and problems; (2) It assumes that video features and text features are in the same space by default, while the model is pre-trained on text data and cannot handle video features well.

[0064] Prior art 2 introduces a video question-answering method based on the object region attention mechanism, which proposes a method that uses the object words appearing in the question as a guide, extracts objects in the video, establishes a correspondence between the two, generates an attention map of the corresponding words in the video, and extracts object region features based on a dynamic selection guidance model.

[0065] This method has the following defects: it is only applicable to scenarios where questions are asked about the objects mentioned in the question, and its scope of application is very limited. For example, it is difficult to answer the question "What is next to the telephone pole" in the video.

[0066] Prior art 3 introduces the end-to-end video-language transformers with masked visual-token modeling (VIOLET) model. This method collects a large amount of video-text pre-training data and uses a visual unit modeling method based on mask completion for multimodal pre-training. In this scheme, text data is provided by video audio recognition or subtitles. The video and text are segmented into word sequences and input into the transformer model for encoding. The implementation method of the video question answering task is the same as the aforementioned prior art 1. For details, please refer to Figure 1C , Figure 1C A schematic diagram of a VIOLET model framework provided in an embodiment of the present application. Figure 1C As shown in the figure, the video encoder (videoswin transformer, VT) is on the left and the text encoder (language embedder, LE) is on the right. The video-text multimodal model is trained through three pre-training tasks: masked visual-token modeling (MVM), visual-text matching (VTM), and masked language modeling (MLM).

[0067] This method has the following defects: it requires a large amount of pre-training data for training, which is difficult to obtain and the quality of unsupervised data is difficult to guarantee. At the same time, it requires the video formats in upstream and downstream tasks to be similar. For example, using grayscale videos in downstream tasks may result in insufficient model generalization ability.

[0068] Based on the above description, this embodiment of the application proposes a video question answering processing method, which is applied to the above Figure 1A On electronic devices in Figure 2A , is the flow chart of this method, as Figure 2A As shown, the method includes the following steps:

[0069] 201. Obtain a target video and segment the target video to obtain M video segments.

[0070] First, the device for executing the method of the embodiment of the present application may be Figure 1A The processing device in the embodiment of the present application is to obtain the target video. Figure 1AAs shown in , a video is captured using a camera. Alternatively, a video of a thread can be obtained from another device. Segmenting the target video means dividing the target video into M video segments according to a preset duration. For example, if the target video is a 10-second video, it is divided into M = 10 video segments according to a preset duration of 1 second, and each video segment is 1 second long.

[0071] 202. Perform feature extraction on each of the M video clips to obtain M clip features.

[0072] Feature extraction is performed on each of the M video clips, that is, the data of each video clip is processed into a feature vector. Specifically, for example, a three-dimensional separable 3D convolution neural network (S3D) is used to output the data of each video clip to obtain a feature vector.

[0073] See also Figure 2B , Figure 2B A schematic diagram of a process for obtaining fragment features provided in an embodiment of the present application is shown as follows: Figure 2B As shown, the target video is divided into M = 10 video segments, and features are extracted for each video segment to obtain the segment features Hc corresponding to the target video. Hc is composed of feature vectors corresponding to each video segment, that is, Hc = {Hc1, Hc2, Hc3, ... Hc10}.

[0074] 203. Perform feature extraction on the target video in frames to obtain multiple frame features.

[0075] Feature extraction is performed on the target video frame by frame. That is, a single frame of data in the target video is processed to obtain a feature vector corresponding to the single frame image. Specific frame feature extractors that can be used include vision transformers (ViT) or residual neural networks (ResNet). Alternatively, in some cases, two feature extractors can be used to extract frame features from the same frame image to obtain richer frame features, thereby improving the generalization ability of the processing method.

[0076] See also Figure 2C , Figure 2C A schematic diagram of a process for obtaining frame features provided in an embodiment of the present application is shown as follows: Figure 2C As shown, the target video is divided into n frame images, and features are extracted for each frame image to obtain frame features Hf1, Hf2, ... Hfn of each frame image. Then the frame features Hf of the target video are {Hf1, Hf2, ... Hfn}.

[0077] In some cases, in order to reduce the amount of frame-level feature processing, frame-level features can be pooled. Assuming the target video is a video with 16 frames per second (fps), it is divided into a sequence of 160 frame images, and average pooling is performed in units of 16 frames. The process can be found in Figure 2D ,like Figure 2D As shown in FIG, a schematic diagram of a process of obtaining frame features by pooling processing provided in an embodiment of the present application is shown in FIG. Figure 2D As shown in the figure, for frames 1 to 16, the features of each frame are Hf11, Hf12, ..., Hf116, and after average pooling, the frame image corresponding to these 16 frames is Hf1. Similarly, the other frames are also average pooled in units of 16 frames, and the final frame features of the target video are Hf = {Hf1, Hf2, ..., Hfm}, where m = n / 16.

[0078] Assuming that each frame image uses two feature extractors for feature extraction, the target video can include two frame features, denoted as Hfv and Hfr. The extraction process of Hfv and Hfr can be the same as the above-mentioned single frame feature extraction method, which will not be repeated here.

[0079] 204. Perform joint feature extraction on target frames in the M video clips to obtain video character features, where the target frame is any frame in the video clips.

[0080] After dividing the target video into video segments in the aforementioned manner, in addition to performing overall feature extraction on each video segment, the target frame in each video segment can be selected to find the correlation between the target frames of different video segments, and then joint feature extraction can be performed based on this correlation to obtain video character features.

[0081] For the specific process, please refer to Figure 2E , Figure 2E A schematic diagram of a video character feature extraction process provided in an embodiment of the present application is shown in FIG. Figure 2E As shown in the figure, it is assumed that the target video is divided into video segments with a sequence length of 10 and frame images with a sequence length of 160. Each video segment includes 16 frame images. One of the frame images contained in each video segment is selected as the target frame. The target frame in video segment 1 is frame image 8, the target frame in video segment 2 is frame image 24, and so on, until the target frame in video segment 10 is frame image 152. The target frame in each video segment can be randomly selected or selected according to a certain rule. For example Figure 2E , select the middle frame of each video clip as the target frame.

[0082] After obtaining the target frame in each video clip, multiple target frames are associated to obtain associated data, and then feature extraction is performed on the associated data to obtain video character features.

[0083] Optionally, joint feature extraction is performed on the target frames in the M video clips to obtain video role features, including: inputting the target frames of the M video clips into an image description model for processing to obtain an image description of the target frame of each video clip; inputting the image description of the target frame of each video clip into a text parser for role extraction, and generating a role structure graph corresponding to the target frame of each video clip according to the semantic structure of the extracted results, wherein the role structure graph includes nodes identifying different roles and edge information between nodes; merging multiple role structure graphs according to the same nodes to obtain a target role structure graph, and inputting the target role structure graph into a graph convolutional network for encoding to obtain video role features.

[0084] Specifically, the target frame of each video clip is input into an image captioning model to obtain an image caption for that target frame. The image captioning model can be a model such as multi-grained vision language pretraining: aligning texts with visual concepts (X-VLM), where the image caption is textual information. For example, after inputting target frame image 8 into the image captioning model, the resulting image caption is "a metal pot filled with vegetables placed on top of a stove." The image captions of target frames from different video clips are input into a text parser, such as an abstract semantic representation (AMR) parser, to extract roles. For example, after extracting the role of "a metal pot filled with vegetables placed on top of a stove," the roles obtained may include "metal," "pot," "vegetables," "filled," "stove," and "on top." Based on these roles, a role structure graph corresponding to each target frame can be generated. The role structure graphs are then merged according to the same roles to obtain a target role structure graph.

[0085] See Figure 2F , Figure 2F A schematic diagram of a process for obtaining a target role structure diagram provided in an embodiment of the present application is shown as follows: Figure 2F As shown in Figure 1, assume that the image description of the target frame of video clip 1 is S1: a metal pot filled with vegetables is placed on the stove, and the image description of the target frame of video clip 2 is S2: a person is chopping celery into pieces on the stove. The two image descriptions are input into the text parser for role extraction, and the corresponding roles and the role structure diagram composed of these roles are obtained, as shown in Figure 1. Figure 2FAs shown in , the role structure graph includes nodes that identify different roles and edge information between nodes. The edge information between nodes is used to describe the association relationship between node roles. When the role structure graphs corresponding to different target frames include the same nodes (roles), they can be merged. Figure 2F In the example above, the character structure graphs of the two target frames are merged using the node "stove" as the connection point. The target frames of other time-frequency segments can be processed in the same way to obtain the merged target character structure graph.

[0086] After obtaining the target character structure diagram, it is input into the graph convolution network (GCN) for encoding to obtain the video character features.

[0087] 205. Extract semantic features of target frames in the M video clips to obtain target semantic features.

[0088] After obtaining the target frames in M ​​video clips and the image descriptions of the target frames according to the aforementioned method, since the image descriptions are text information, semantic features can be extracted from the image descriptions of each target frame to obtain the semantic features of each target frame, and then form the target semantic features of the target video.

[0089] Optionally, semantic features are extracted from the target frames in the M video clips to obtain target semantic features, including: inputting the target frames of the M video clips into an image description model for processing to obtain an image description of the target frame of each video clip; segmenting the image description of the target frame of each video clip, adding a classification CLS before the segmentation result, and inputting the segmentation result and CLS into a pre-trained language model for encoding to obtain the features of each segmentation and the features of the CLS in the image description of the target frame of each video clip; the target semantic features include the features of the CLS corresponding to the target frames in the M video clips.

[0090] Specifically, as described above, the target frame can be input into the image description model to obtain the image description of each target frame. The image description of each target frame is then segmented, and a classification identifier (CLS) is added before the segmentation result. The segmentation result and CLS are input into the pre-trained language model for encoding. The pre-trained language model can be a BERT language model. For details, please refer to Figure 2G , Figure 2G A schematic diagram of a process for extracting semantic features provided in an embodiment of the present application is shown in FIG. Figure 2G As shown, the image description of the target frame is "cutting celery into pieces". Word segmentation can refer to dividing the entire sentence into words or phrases according to their meanings, or dividing the entire sentence into single words. Figure 2GThe latter segmentation method is used. After prepending the segmentation result with a CLS, the result is fed into the BERT language model to obtain the features of each segmentation and the CLS features. The CLS features are used as the semantic features of the target frame. The CLS features corresponding to the target frames of all video clips in the target video constitute the target semantic features of the target video.

[0091] 206. Obtain question text for the target video input, and perform feature extraction on the question text to obtain question text features.

[0092] In the embodiment of the present application, a corresponding question is raised for a target video. Accordingly, a processing device obtains input question text and performs feature extraction on the question text to obtain question text features.

[0093] The process for extracting features from question text can be similar to the semantic feature extraction process for image descriptions described above. The question text is segmented, and the CLS is added before the segmentation result. Both the segmentation result and the CLS are then fed into the BERT language model for encoding. The CLS features are obtained as the question text features Hq.

[0094] 207. Fuse the M segment features and multiple frame features with the target semantic features to obtain fused video features; then fuse the fused video features with the video character features based on the question text features to obtain a video feature vector.

[0095] After obtaining M segment features, multiple frame features, and target semantic features of the target video according to the above process, these features can be fused to obtain fused video features, so that semantic features are fused into the video features.

[0096] Optionally, M segment features and multiple frame features are fused with the target semantic feature to obtain a fused video feature, including: mapping the M segment features, multiple frame features and target semantic features to multiple vector spaces respectively; inputting the vector space mapped to the M segment features or multiple frame features and the vector space mapped to the target semantic feature into the attention mechanism network, wherein the vector space inputted by the M segment features or multiple frame features and the vector space inputted by the target semantic feature form a three-dimensional vector space K, Q, V; obtaining an output vector processed by the attention mechanism network, and determining the output vector in the output vector that is related to the segment features or frame features as the fused video feature.

[0097] Specifically, in the embodiment of the present application, an attention mechanism can be used to fuse the obtained multiple types of features. The general formula of the attention mechanism is:

[0098]

[0099] This means that three input vectors, Q (query), K (key), and V (value), are specified. The dot product of the transpose of Q and K is performed, divided by a scaling factor, and the result is processed using softmax. Finally, the result is tensor multiplied with V to obtain the output vector. To extract more interaction information, a multi-head attention model can be used to capture different interaction information in multiple different projection spaces.

[0100] In the embodiment of the present application, M segment features Hc and multiple frame features Hf can be collectively referred to as Hv, and the target semantic feature is Hs. Hv and Hs are fused by inputting the Q and K vectors of Hv and the V vector of Hs into the same attention mechanism network for processing, or inputting the V vector of Hv and the Q and K vectors of Hs into the same attention mechanism network for processing. For details, please refer to Figure 2H , Figure 2H A schematic diagram of a process for obtaining fused video features provided in an embodiment of the present application is shown in FIG. Figure 2H As shown, represents the target semantic features of the l-th layer input, Represents the segment feature Hc or frame feature Hf of the target video input at the lth layer (segment features and frame features are fused with the target semantic features separately). After mapping Hs and Hv to the three vector spaces Q, K, and V, the V vector V of Hv is V , and the Q vector and K vector of Hs, namely Q C and K C Input the multi-head attention mechanism network on the left. V vector V of Hs C , and the Q vector and K vector of Hv, namely Q V and K V Input the multi-head attention network on the left. The outputs of the multi-head attention network are summed and normalized, then fed into the feedforward network. The inputs to the feedforward network are then summed and normalized to obtain the output of layer l+1. The fused video features are Hv, fused with Hs, and can be expressed as Hv', which includes Hc' and Hf'.

[0101] Optionally, the embodiment of the present application may also use other encoding methods to obtain the fused video features. Figure 2I , Figure 2I Another schematic diagram of the process of obtaining fused video features provided in an embodiment of the present application is as follows: Figure 2I As shown in (a), H S and H V The three vectors Q, K, and V are simultaneously input into the multi-head attention mechanism network, and then a series of encodings are performed to obtain the output vector. Or as Figure 2IAs shown in (b2), the vectors Q, K of Hv and the vector V of Hs are input into the multi-head attention mechanism network for encoding to obtain the output vector. If Hv is only encoded in one layer, then according to the output vector The fused video feature Hv' can be determined. If Hv needs to be multi-layer encoded, it is also necessary to Figure 2I The processing in (b1) obtains In order to determine the corresponding input in the next layer of processing The vector V.

[0102] After obtaining the question text features, the fused video features, and the video character features, the multi-granularity features can be fused based on the question text features, that is, the fused video features and the video character features are fused based on the question text features to obtain the video feature vector.

[0103] Optionally, the video features and video role features after fusion are fused based on the question text features to obtain a video feature vector, including: linearly mapping the question text features to obtain multiple weight vectors, the number of weight vectors is the same as the number of fused feature vectors and vector types of video role features, and the sum of the weight vectors is 1; the fused feature vector and the video role features are weighted and summed with their corresponding weight vectors to obtain a video feature vector.

[0104] Specifically, after obtaining the question text feature through the above process, it itself is a multi-dimensional feature vector, which can be linearly mapped to achieve dimensionality reduction and obtain multiple weight vectors. For example, the question text feature itself is a 1*178 feature vector. In the embodiment of the present application, the fused video vector includes Hc' and Hf', and the video character feature is Hg. Then the question text feature can be mapped to a 1*3 weight vector. The sum of these weight vectors is 1, and each dimension of the weight vector corresponds to another feature. For example, the weight vector of Hc' is w1, the weight vector of Hf' is w2, and the weight vector of Hg is w3. Finally, these features and the weight vector corresponding to each feature are weighted and summed to obtain the video feature vector. That is, the video feature vector R = Hc'*w1+Hf'*w2+Hg*w3.

[0105] 208. Input the video feature vector and the question text feature into the video feature encoding layer to obtain the preset category to which the video feature belongs, and determine the answer to the question text according to the preset category.

[0106] The video feature vector R and the question text feature Hq are fed into the video feature encoding layer (vision-vision fusion) to obtain the output features of the common feature. Specifically, R and Hq are concatenated to obtain a concatenated sequence. Then, CLS is added before the concatenated sequence of the video feature vector + question text feature. Both the concatenated sequence and CLS are fed into the video feature encoding layer to obtain the CLS features, labeled F.

[0107] Finally, for each question text in the target video, there will be optional answers as preset categories. For example, assuming the question answer type is "yes / no", the preset categories include "yes" and "no" (or 0 / 1) categories. Assuming the question answer type is "event", it means that the question text is related to the event, such as "time of the event", "place of the event", "how the event happened", "what event happened", "people related to the event", etc. For example, the question text is "Where is the cat?", and the optional answers include "on the bed", "on the sofa", and "on the ground", then the preset categories also include preset categories representing these three optional answers. Input F into the classification layer classification to determine the category to which F belongs, and determine the question answer corresponding to the question text based on the correspondence between the preset classification and the optional answers. For example, the category described by F is category B, and the answer corresponding to category B is "on the bed", then it can be determined that the answer to the question "Where is the cat?" is "on the bed".

[0108] The overall implementation process of the above video question-answering processing method can be found in Figure 2J , Figure 2J A model block diagram for video question answering processing provided in an embodiment of the present application is shown as follows: Figure 2J As shown in Figure 1, the model includes an input and preprocessing module, a feature extraction module, and a feature fusion module. The input and preprocessing module includes inputting the target video, segmenting the target video, extracting frames, and inputting question text. The feature extraction layer includes extracting video character features from target frame i selected from each video segment, extracting target semantic features from target frame i, extracting frame features from video frames, and extracting segment features from video segment j. The video character feature extraction process is feature extraction at the character level, the target semantic feature extraction process is feature extraction at the semantic level, the frame feature extraction process is feature extraction at the frame level, and the segment feature extraction process is feature extraction at the segment level.

[0109] After feature extraction is completed, various features are fused. First, the video features are fused with the semantic features, that is, the frame features are fused with the semantic features, and the segment features are fused with the semantic features, to obtain video features that have been fused with semantic features, which are called fused video features (including fused frame features Hf' and fused segment features Hc'). Then, guided by the question text features, the features at different levels are fused, that is, the fusion of video character features and fused video features is completed to obtain the video feature vector R. Finally, the video feature vector R is fused with the question text features Hq (marked as video-video feature fusion in the figure) to obtain the final output feature F. The output feature F can be used for classification related to the answer to the question to obtain the final answer to the question.

[0110] It can be seen that in the embodiment of the present application, multi-granularity multimodal feature extraction is first performed to obtain features at the role level, semantic level, frame level and segment level, and then the features of the video modality (including frame features and segment features) are fused with the features of the semantic modality (target semantic features) to obtain fused video features that have been fused with semantic features. The multi-granularity features (including fused video features and video role features) are then guided by the question text, that is, based on the question text features, to obtain a video feature vector. This process uses the image description model to align the video semantic space with the text semantic space, solves the common feature space mismatch problem in the multimodal model, and realizes the fusion processing of information of different modalities in the same feature space by fusing multi-granularity features. Finally, the fused video features are fused and encoded with the question text to obtain the classification result of the output feature, and the answer to the question is determined based on the classification result. The whole process effectively improves the accuracy of the video question and answer results in the absence of large-scale video-text alignment pre-training data and model, while improving the generalization ability of the processing model.

[0111] in addition, Figure 2JThe model in the embodiment adopts some specific feature extractors, text encoders, etc. It should be understood that these tools are optional in the actual implementation process, and other tools can also be used. For example, in addition to ViT or ResNet, the frame feature extractor can also adopt CNN, accelerated version (faster) RCNN, contrastive language-image pretraining model (CLIP) and other image encoders. In addition to S3D network, the segment feature extractor can use video vision transformer (VIVIT), hierarchical vision transformer using shifted windows (SWIN transformer), is space-time attention all you need for video understanding (Timesformer) and other video encoders. The image description model can be replaced by models such as clip and blip. In addition to AMR parser, the image description text parser can also be replaced by Stanford parser and the like. The method of using these tools to implement the embodiment of the present application is as described above and will not be repeated here.

[0112] like Figure 3 As shown, an electronic device 300 provided in an embodiment of the present application includes a device for implementing the above Figure 2A The module or unit of the method embodiment shown and any possible design method thereof. The module or unit can be a hardware circuit, or software, or a combination of a hardware circuit and software. The electronic device can be a terminal device, or a device in a terminal device or a terminal device (for example, a chip, or a chip system, or a circuit), or a device that can be used in conjunction with a terminal device. The device may include a receiving and acquiring unit 301, a feature extraction unit 302, a feature fusion unit 303, and a classification unit 304, wherein:

[0113] An acquisition unit 301 is configured to acquire a target video and segment the target video to obtain M video segments;

[0114] The feature extraction unit 302 is configured to perform feature extraction on each of the M video segments to obtain M segment features; perform feature extraction on the target video frame by frame to obtain multiple frame features; perform joint feature extraction on the target frame in the M video segments to obtain video character features, where the target frame is any frame in the video segments; perform semantic feature extraction on the target frame in the M video segments to obtain target semantic features; obtain a question text input for the target video, and perform feature extraction on the question text to obtain question text features;

[0115] A feature fusion unit 303 is configured to fuse the M segment features and multiple frame features with the target semantic feature to obtain a fused video feature, and then fuse the fused video feature with the video character feature based on the question text feature to obtain a video feature vector;

[0116] The classification unit 304 is used to input the video feature vector and the question text feature into the video feature encoding layer, obtain the preset category to which the video feature belongs, and determine the answer to the question text according to the preset category.

[0117] For a more detailed description of the acquisition unit 301 , feature extraction unit 302 , feature fusion unit 303 , and classification unit 304 , please refer to the relevant description in the above method embodiment, which will not be described again here.

[0118] like Figure 4 As shown, Figure 4 A schematic diagram of the hardware structure of an electronic device in an embodiment of the present application is shown. Figure 2A The structure of the electronic device corresponding to the method of the embodiment can refer to Figure 4 The electronic device 800 includes a processor 111, a transceiver 112, and an input device 114, wherein the processor 111 is electrically coupled to the transceiver 112 and the input device 114;

[0119] The processor 111 is configured to execute part or all of the computer program instructions in the memory. When the part or all of the computer program instructions are executed, the device executes the method described in any one of the above embodiments.

[0120] The transceiver 112 is used to communicate with other devices; for example, to obtain a target video.

[0121] The input device 114 is used to receive user input information, such as obtaining a question text input for the target video. The input device can specifically be a voice input device or a text input device.

[0122] Optionally, a memory 113 is also included for storing computer program instructions. Optionally, the memory 113 (memory #1) is located within the device, the memory 113 (memory #2) is integrated with the processor 111, or the memory 113 (memory #3) is located outside the device.

[0123] It should be understood that Figure 4 The electronic device 800 shown may be a chip or circuit. For example, the chip or circuit may be provided within a terminal device or electronic device. The transceiver 112 may also be a communication interface. A transceiver includes a receiver and a transmitter. Furthermore, the electronic device 800 may also include a bus system.

[0124] Among them, the processor 111, the memory 113, and the transceiver 112 are connected via a bus system, and the processor 111 is used to execute instructions stored in the memory 113 to control the transceiver to receive and send signals, thereby completing the steps of the first device or the second device in the implementation method involved in this application. The memory 113 can be integrated into the processor 111 or set separately from the processor 111.

[0125] As an implementation method, the functions of the transceiver 112 can be implemented by a transceiver circuit or a dedicated transceiver chip. The processor 111 can be implemented by a dedicated processing chip, a processing circuit, a processor, or a general-purpose chip. The processor can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. The processor can further include a hardware chip or other general-purpose processor. The above-mentioned hardware chip can be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), and other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., or any combination thereof. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0126] It should also be understood that the memory mentioned in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DR RAM). It should be noted that the memory described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0127] An embodiment of the present application provides a computer storage medium storing a computer program, which includes a computer program for executing the corresponding method in the above embodiment.

[0128] An embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the corresponding method in the above embodiment.

[0129] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0130] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0131] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0136] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A video question-answering processing method, applied to electronic equipment, characterized in that: The method comprises: Obtain a target video, and segment the target video to obtain M video segments; Performing feature extraction on each of the M video clips to obtain M clip features; Extracting features from the target video in frames to obtain multiple frame features; Performing joint feature extraction on a target frame in the M video clips to obtain video character features, where the target frame is any frame in the video clips; Extracting semantic features of target frames in the M video clips to obtain target semantic features; Obtaining a question text inputted for the target video, and performing feature extraction on the question text to obtain question text features; fusing the M segment features and the multiple frame features with the target semantic feature to obtain a fused video feature, and then fusing the fused video feature with the video character feature based on the question text feature to obtain a video feature vector; The video feature vector and the question text feature are input into a video feature coding layer to obtain a preset category to which the video feature belongs, and the answer to the question text is determined according to the preset category.

2. The method according to claim 1, characterized in that The extracting semantic features of the target frames in the M video clips to obtain target semantic features includes: Inputting the target frames of the M video clips into an image description model for processing to obtain an image description of the target frame of each video clip; Segmenting the image description of the target frame of each video clip, adding a classification identifier CLS before the segmentation result, and inputting the segmentation result and the CLS into a pre-trained language model for encoding to obtain features of each segmentation and features of the CLS in the image description of the target frame of each video clip; The target semantic features include features of the CLS corresponding to the target frames in the M video clips.

3. The method according to claim 1 or 2, characterized in that The step of performing joint feature extraction on the target frames in the M video clips to obtain video character features includes: Inputting the target frames of the M video clips into an image description model for processing to obtain an image description of the target frame of each video clip; Inputting the image description of the target frame of each video clip into a text parser for role extraction, and generating a role structure graph corresponding to the target frame of each video clip according to the semantic structure of the extracted result, wherein the role structure graph includes nodes identifying different roles and edge information between the nodes; Multiple role structure graphs are merged according to the same nodes to obtain a target role structure graph, and the target role structure graph is input into a graph convolutional network for encoding to obtain the video role features.

4. The method according to claim 1 or 2, characterized in that The fusing the M segment features and the multiple frame features with the target semantic feature to obtain a fused video feature includes: Mapping the M segment features, the multiple frame features, and the target semantic features to multiple vector spaces respectively; Input the vector space mapped to the M segment features and the multiple frame features and the vector space mapped to the target semantic feature into the attention mechanism network, wherein the vector space input by the M segment features and the multiple frame features and the vector space input by the target semantic feature form a three-dimensional vector space K, Q, V; An output vector processed by the attention mechanism network is obtained, and an output vector in the output vector that is related to the segment feature or the frame feature is determined as the fused video feature.

5. The method according to claim 1 or 2, characterized in that The step of fusing the fused video features and the video character features based on the question text features to obtain a video feature vector includes: Performing linear mapping on the question text features to obtain a plurality of weight vectors, wherein the number of the weight vectors is the same as the number of the fused feature vectors and the number of the vector types of the video character features, and the sum of the weight vectors is 1; The fused feature vector and the video character feature are weighted and summed with the weight vector corresponding to each of the features to obtain the video feature vector.

6. The method according to claim 1 or 2, characterized in that The step of inputting the video feature vector and the question text feature into a video feature encoding layer, obtaining a preset category to which the video feature belongs, and determining an answer to the question text according to the preset category includes: Splicing the video feature vector with the question text feature to obtain a splicing feature; Inputting the splicing features into a video feature encoding layer to classify the splicing features, and obtaining the probability that the splicing features belong to each preset classification; According to the target preset category to which the splicing feature belongs with the highest probability, a target question answer corresponding to the target preset category is determined, and the target question answer is determined as the question answer of the question text.

7. An electronic device, characterized in that: The structure of the device includes a processor, an input device, and a memory; the processor is coupled to the input device and the memory, and is used to combine input information from the input device and execute computer program instructions stored in the memory so that the device performs the method according to any one of claims 1 to 6.

8. A chip system comprising: A processor coupled to an interface circuit and a memory, wherein the interface circuit is used to exchange code instructions with the processor, and the memory is used to store programs or instructions. When the program or instruction is executed by the processor, the chip system executes the method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that Used to store instructions, when the instructions are executed, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, which, when read and executed by a computer, enables the computer to execute the method according to any one of claims 1 to 6.

Images