Video question answering method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a video question answering method and device, equipment and a storage medium. The method comprises the following steps: encoding a user question and a to-be-analyzed video to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the to-be-analyzed video; determining target key information from the question code according to a pre-trained key information query vector; generating a plurality of video frame codes based on user question guidance according to the target key information and the plurality of continuous video frame codes, and performing fusion processing on the plurality of video frame codes based on user question guidance based on the target key information to obtain target coding information; and inputting the target code information and the question code into a pre-training language model to obtain a response answer output by the pre-training language model, thereby saving computing resources and improving processing efficiency.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a video question-answering method, apparatus, device, and storage medium. Background Art

[0002] With the booming development of online video platforms and the continuous growth of video content, the demand for tools to effectively manage, analyze and process such rich video content is growing, which has led to the emergence of video understanding methods and analysis technologies. Large Language Model (LLM) technology has shown excellent performance in language and multimodal tasks and is widely used in the field of video understanding. Based on LLM technology, a series of video question-answering algorithms have been proposed, which can answer various questions raised by users based on the video content.

[0003] The LLM-based video question answering algorithm needs to convert video frames into visual tags and use the visual tags and text tags as the input of LLM.

[0004] Current technologies generally use visual encoders to independently encode the extracted video frames, which has significant redundancy in both time and space. In addition, current technologies usually splice the encoding of each video frame together as the input of LLM, which cannot take into account both efficiency and resource consumption. Summary of the invention

[0005] The embodiments of the present application provide a video question-answering method, apparatus, device, and storage medium to solve the problems of temporal and spatial redundancy existing in current visual coding technology.

[0006] In a first aspect, the present application provides a video question-answering method, comprising:

[0007] Encoding the user question and the video to be analyzed to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the video to be analyzed;

[0008] determining target key information from the question encoding according to a pre-trained key information query vector;

[0009] generating a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes, and fusing the plurality of video frame codes guided by user questions based on the target key information to obtain target code information;

[0010] The target encoding information and the question encoding are input into a pre-trained language model to obtain a response answer output by the pre-trained language model.

[0011] Optionally, determining the target key information from the question encoding according to the pre-trained key information query vector includes:

[0012] Initializing the key information query vector to obtain a first key information query vector, wherein the first key information query vector has the same spatial structure as the video frame code, and the first key information query vector has the same dimension as the question code and the video frame code;

[0013] Training the first key information query vector to obtain an updated second key information query vector;

[0014] The second key information query vector and the question code are analyzed and processed to obtain target key information in the question code.

[0015] Optionally, the analyzing and processing the second key information query vector and the question code to obtain the target key information in the question code includes:

[0016] determining an attention weight between the question encoding and the second key information query vector;

[0017] Determine target key information related to the second key information query vector from the question encoding according to the attention weight, wherein the number and dimension of the target key information are the same as the number and dimension of the second key information query vector.

[0018] Optionally, generating a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes comprises:

[0019] For any one video frame code among the plurality of continuous video frame codes, determining a correlation degree between the video frame code and a position corresponding to the target key information;

[0020] Determining first weight vectors at different positions of the video frame encoding according to the association degree;

[0021] The video frame code is weighted according to the first weight vector to obtain the video frame code guided by the user question.

[0022] Optionally, determining, for any one video frame code among the plurality of continuous video frame codes, a degree of association between the video frame code and a position corresponding to the target key information includes:

[0023] Determining a cosine similarity matrix of the video frame encoding and the target key information;

[0024] The degree of association between the video frame encoding and the corresponding position of the target key information is determined according to the cosine similarity matrix.

[0025] Optionally, the fusing of the plurality of video frame codes guided by the user question based on the target key information to obtain target coding information includes:

[0026] Determine, according to the cosine similarity matrix, a second weight vector for the same position of the video frame encoding guided by the user question at different times;

[0027] The target coding information is obtained by weightedly merging the plurality of video frame codes guided by the user question according to the second weight vector.

[0028] In a second aspect, the present application provides a video question-and-answer device, comprising:

[0029] A processing module, used to encode the user question and the video to be analyzed, to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the video to be analyzed;

[0030] A determination module, configured to determine target key information from the question encoding according to a pre-trained key information query vector;

[0031] The processing module is further configured to generate a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes, and perform fusion processing on the plurality of video frame codes guided by user questions based on the target key information to obtain target code information;

[0032] The processing module is further used to input the target encoding information and the question encoding into a pre-trained language model to obtain a response answer output by the pre-trained language model.

[0033] Optionally, the processing module is further used to initialize the key information query vector to obtain a first key information query vector, wherein the first key information query vector has the same spatial structure as the video frame code, and the first key information query vector has the same dimension as the question code and the video frame code;

[0034] Training the first key information query vector to obtain an updated second key information query vector;

[0035] The second key information query vector and the question code are analyzed and processed to obtain target key information in the question code.

[0036] Optionally, the determination module is further used to determine an attention weight between the question encoding and the second key information query vector;

[0037] Determine target key information related to the second key information query vector from the question encoding according to the attention weight, wherein the number and dimension of the target key information are the same as the number and dimension of the second key information query vector.

[0038] Optionally, the determination module is further used to determine, for any one of the plurality of continuous video frame codes, a degree of association between the video frame code and a position corresponding to the target key information;

[0039] Determining first weight vectors at different positions of the video frame encoding according to the association degree;

[0040] The processing module is further used to perform weighted processing on the video frame encoding according to the first weight vector to obtain the video frame encoding guided by the user question.

[0041] Optionally, the determination module is further used to determine a cosine similarity matrix between the video frame encoding and the target key information;

[0042] The degree of association between the video frame encoding and the corresponding position of the target key information is determined according to the cosine similarity matrix.

[0043] Optionally, the determination module is further used to determine, according to the cosine similarity matrix, a second weight vector for the same position of the video frame encoding guided by the user question at different moments;

[0044] The processing module is further used to perform weighted fusion on the plurality of video frame codes guided by the user question according to the second weight vector to obtain the target coding information.

[0045] In a third aspect, the present application provides a video question-and-answer device, comprising:

[0046] Memory;

[0047] processor;

[0048] Wherein, the memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory to implement the video question-answering method as described in the first aspect and various possible implementations of the first aspect.

[0050] In a fourth aspect, the present application provides a computer storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the video question-and-answer method as described in the first aspect and various possible implementations of the first aspect.

[0051] The present application provides a video question-answering method, device, equipment and storage medium. The method encodes the user question and the video to be analyzed to obtain the question code corresponding to the user question and multiple continuous video frame codes corresponding to the video to be analyzed; determines the target key information from the question code according to the pre-trained key information query vector; generates multiple video frame codes guided by the user question according to the target key information and multiple continuous video frame codes, and fuses the multiple video frame codes guided by the user question based on the target key information to obtain the target code information; inputs the target code information and the question code into the pre-trained language model to obtain the response answer output by the pre-trained language model, thereby saving computing resources and improving processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0053] Figure 1 A video question-and-answer method process provided for this application Figure 1 ;

[0054] Figure 2 A video question-and-answer method process provided for this application Figure 2 ;

[0055] Figure 3 A schematic diagram of the structure of a video question-and-answer device provided in this application;

[0056] Figure 4 A schematic diagram of the structure of a video question-and-answer device provided in this application.

[0057] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0060] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0061] With the development of online video platforms and the rapid growth of video content, users have an increasing demand for the ability to efficiently manage, deeply analyze, and quickly process these massive and diverse video contents.

[0062] The LLM-based video question-answering algorithm can accurately and quickly answer various questions raised by users according to the video content, providing users with a more intelligent and convenient video interaction experience.

[0063] The current technology generally adopts the method of performing independent visual encoding for each video frame, which has a large redundancy in both time and space. In addition, the existing technology usually simply splices the encoding results of each video frame in sequence as the input of LLM, ignoring the rationality of resource consumption.

[0064] In view of the above problems existing in the prior art, the present application provides a video question-answering method. The method designs a set of key information query vectors to query key information from the questions raised by the user, and uses the key information to guide the formation of video frame encoding. Based on the key information, the features of the video frames related to the question are enhanced, the features of the irrelevant video frames are suppressed, and they are integrated into a set of visual tokens, taking into account both encoding efficiency and computing resources.

[0065] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0066] Figure 1 This is a video question-and-answer method process provided by the embodiment of the present application Figure 1 .like Figure 1As shown, the video question-answering method provided in this embodiment includes:

[0067] S101: Encode the user question and the video to be analyzed to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the video to be analyzed.

[0068] The user question refers to a query, request or question about the video content input by the user, and the input form may be text or audio, for example.

[0069] It is understandable that encoding user questions involves natural language processing technology, such as word segmentation, part-of-speech tagging, syntactic analysis, semantic understanding, etc. After encoding, the user question is converted into one or more encoding vectors, and the encoding vector can reflect the semantic content and intention of the question.

[0070] Furthermore, continuous video frames are extracted from the video to be analyzed, the video file is converted into independent image frames, and image preprocessing and feature extraction are performed on each extracted video frame to obtain visual coding information of each image frame so that the computer can understand and analyze it.

[0071] In an optional embodiment, after encoding the user question and the video to be analyzed to obtain the question code corresponding to the user question and multiple continuous video frame codes corresponding to the video to be analyzed, the visual code and the question code are aligned. Specifically, the alignment of the visual code and the question code is achieved by constructing an alignment layer and training it.

[0072] The encoding process of user questions and videos to be analyzed aims to convert unstructured data into a structured form that is easy for computers to understand and process, so as to support subsequent video analysis tasks.

[0073] S102: Determine target key information from the question encoding according to the pre-trained key information query vector.

[0074] The pre-trained key information query vector refers to a vector representation pre-trained on a large amount of data. The target key information refers to the information in the user's question about a specific object, action, scene or event in the video to be analyzed.

[0075] It can be understood that the key information query vector is specifically used to query key information. It can be a vector based on word embedding or a context-aware vector based on a deep learning model, which is used to capture and represent the semantic features of the key information in the query.

[0076] Furthermore, a cross-attention mechanism is used to extract key information from the question encoding using the key information query vector. Specifically, the key information query vector is used as a query to interact with the key-value pairs generated by the encoder. In the cross-attention mechanism, mechanisms such as dot product attention and multi-head attention are usually used to calculate the similarity or correlation between the query and the key-value pair.

[0077] According to the calculated attention weights, the part most relevant to the key information query vector is extracted from the vector representation generated by the encoder and used as the key information in the question.

[0078] Determining target key information from question encoding based on pre-trained key information query vectors aims to extract key information from user questions in order to improve the accuracy of responses in subsequent video analysis processing.

[0079] S103: Generate multiple video frame codes guided by user questions according to the target key information and multiple continuous video frame codes, and perform fusion processing on the multiple video frame codes guided by user questions based on the target key information to obtain target coding information.

[0080] The video frame coding guided by user questions refers to the coding obtained by performing specific processing or screening on the video frames according to the key information in the user questions. The target coding information refers to the video coding representation obtained by fusion processing and capable of fully reflecting the key information in the user questions.

[0081] It can be understood that, according to the extracted target key information, multiple continuous video frame codes are screened or weighted to obtain video frame codes guided by user questions. Further, multiple video frame codes guided by user questions are fused to obtain target code information. The fusion processing method includes but is not limited to weighted average, maximum pooling, and attention weighting.

[0082] Multiple video frame codes guided by user questions are generated according to target key information and multiple continuous video frame codes, and the multiple video frame codes guided by user questions are fused based on the target key information to obtain target coding information, which is conducive to enhancing features related to user questions, suppressing features irrelevant to user questions, and improving the accuracy of outputting question response answers.

[0083] S104: Input the target encoding information and the question encoding into the pre-trained language model to obtain a response answer output by the pre-trained language model.

[0084] Among them, the pre-trained language model is a deep learning model that has been trained with a large amount of text data and can understand and generate natural language text. The pre-trained language model can generate answer text based on the input target encoding information and question encoding.

[0085] It can be understood that the target encoding information and question encoding are used as inputs of the pre-trained language model. The pre-trained language model analyzes and processes the input target encoding information and question encoding based on its internal knowledge structure and training experience, and outputs one or more candidate answers.

[0086] In an optional embodiment, the video question answering method further includes updating the parameters of the language model by autoregression. The process includes defining a loss function and optimizing the model parameters by a back propagation algorithm to further improve the accuracy of the answer output by the language model.

[0087] Inputting the target encoding information and question encoding into the pre-trained language model and obtaining the response answer outputted by the model can provide users with more accurate, intelligent and personalized video question-and-answer services.

[0088] A video question-answering method provided by an embodiment of the present application encodes user questions and videos to be analyzed to obtain question codes corresponding to the user questions and multiple continuous video frame codes corresponding to the videos to be analyzed; determines target key information from the question code according to a pre-trained key information query vector; generates multiple video frame codes guided by user questions according to the target key information and the multiple continuous video frame codes, and fuses the multiple video frame codes guided by user questions based on the target key information to obtain target code information; inputs the target code information and the question code into a pre-trained language model to obtain a response answer output by the pre-trained language model, thereby saving computing resources and time and improving the accuracy of the response answer output based on the user question.

[0089] Figure 2 A video question-answering method process provided in the embodiment of the present application Figure 2 This embodiment is in Figure 1 Based on the embodiment, a possible implementation of the video question-answering method is described in detail. Figure 2 As shown, the method includes:

[0090] S201: Encode the user question and the video to be analyzed to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the video to be analyzed.

[0091] Among them, step S201 is similar to the above step S101 and will not be repeated here.

[0092] S202: Initialize the key information query vector to obtain a first key information query vector.

[0093] The first key information query vector has the same spatial structure as the video frame code, and the first key information query vector has the same dimension as the question code and the video frame code.

[0094] It can be understood that in machine learning and deep learning models, initialization refers to giving initial values ​​to the parameters of the model before training begins.

[0095] According to the requirements of the task and the characteristics of the model, a suitable initialization method is selected, and based on the selected initialization method, an initial vector matching the dimension of the key information query vector is generated, and the generated initial vector is assigned to the key information query vector to obtain the first key information query vector.

[0096] Exemplarily, the query vector is randomly initialized, that is, a value is randomly extracted from a uniform distribution or a normal distribution to initialize each element in the query vector, thereby obtaining the initialized first key information query vector.

[0097] S203: Training the first key information query vector to obtain an updated second key information query vector.

[0098] Specifically, by collecting and preparing labeled or unlabeled training data sets, the key information query vector is trained. According to the task requirements, a suitable loss function is selected to measure the difference between the model output and the true label. According to the loss function and the model structure, a suitable optimization algorithm is selected to update the weight of the key information query vector. During the training process, the weight of the key information query vector is continuously adjusted through multiple iterations.

[0099] By training the first key information query vector, a second key information query vector is obtained, which can more accurately capture and represent the query intent and semantic information of keywords.

[0100] S204: Determine an attention weight between the question encoding and the second key information query vector.

[0101] It can be understood that the second key information query vector is used as a query sequence, the question code is used as a key sequence, and the two input sequences are split and used as a query set and a key value set in the calculation process respectively.

[0102] Further, the similarity between each element in the query set and each element in the key value set is determined. For example, the similarity between each element in the query set and each element in the key value set is calculated by dot product, cosine similarity, etc. Then, the similarity between each element quantity or sentence in the question encoding and the second key information query vector is obtained. According to the similarity result, a weight is assigned to each element in the key value set, and the higher the weight, the more relevant the element is to the element in the query set.

[0103] Specifically, the softmax function is applied to convert the similarity score into a probability distribution, namely the attention weight. The softmax function can map the original score to the interval (0, 1), and the sum of all scores is 1.

[0104] S205: Determine target key information related to the second key information query vector from the question encoding according to the attention weight.

[0105] The number and dimension of the target key information are the same as the number and dimension of the second key information query vector.

[0106] It can be understood that the attention weights are used to perform weighted summation on the elements in the question encoding to obtain a weighted representation, which reflects the importance of different parts of the question to the query vector.

[0107] Furthermore, according to the importance of different parts of the question to the query vector, the elements in the question encoding are sorted or filtered, and the information in the question encoding that is more relevant to the query intent is used as the target key information.

[0108] S206: Determine the cosine similarity matrix of the video frame encoding and the target key information.

[0109] It can be understood that for each video frame code and each target key information, the cosine similarity between them is calculated. Cosine similarity can reflect the similarity of vectors in direction and is not affected by the length of the vector.

[0110] The cosine similarity matrix is ​​used to represent the cosine similarity between multiple video frame codes and multiple target key information. The rows of the matrix represent the video frame codes, the columns represent the target key information, and the elements in the matrix represent the cosine similarity between the corresponding video frame codes and the target key information.

[0111] S207: Determine the degree of association between the video frame encoding and the corresponding position of the target key information according to the cosine similarity matrix.

[0112] It can be understood that the correlation between the video frame encoding and the target key information at the corresponding spatial position is determined by analyzing the element values ​​in the cosine similarity matrix.

[0113] Furthermore, the higher the similarity value, the stronger the correlation between the video frame code at the position and the target key information; the lower the similarity value, the weaker the correlation between the video frame code at the position and the target key information.

[0114] S208: Determine first weight vectors for different positions of video frame encoding according to the association degree.

[0115] It can be understood that the first weight vector is a vector corresponding to the video frame encoding position, in which each element represents the weight value assigned to the position according to the degree of association between the video frame encoding and the target key information. The higher the degree of association between a certain position of the video frame encoding and the target key information, the higher the corresponding weight value.

[0116] S209: Perform weighted processing on the video frame codes according to the first weight vector to obtain the video frame codes guided by the user question.

[0117] It can be understood that by performing weighted processing on the video frame encoding according to the first weight vector, it is possible to enhance the features of the area related to the user question and suppress the features of the area irrelevant to the user question, thereby obtaining the video frame encoding guided by the user question.

[0118] Furthermore, for each position in the video frame code, weighted processing is performed according to the weight value in the first weight vector corresponding to the position. The specific method of weighted processing can be multiplication by the weight value, or other operations that can reflect the weight difference. After weighted processing, the video frame code guided by the user question is obtained.

[0119] The video frame encoding is weighted by the first weight vector to obtain the video frame encoding guided by the user question, which provides strong support for subsequent video analysis and processing tasks.

[0120] S210: Determine, according to the cosine similarity matrix, a second weight vector for encoding the same position of the video frame guided by the user question at different moments.

[0121] Each element in the second weight vector corresponds to the weight of the video frame encoding at a specific position at a different time.

[0122] It can be understood that the rows related to the target position are extracted from the cosine similarity matrix, corresponding to the similarity between the feature vectors encoded at the target position of the video frames at different times and the feature vectors of the target key information.

[0123] Furthermore, the weights of the coding units at the same position in different video frames are calculated by applying the softmax function to the cosine similarity matrix in the time dimension, and the weight values ​​at different times are combined into a vector, namely the second weight vector.

[0124] S211: performing weighted fusion on multiple video frame codes guided by user questions according to a second weight vector to obtain target coding information.

[0125] It can be understood that each video frame encoding guided by the user question is multiplied by the corresponding weight, and then the weighted visual encodings are added to obtain the video frame fusion encoding, which fuses the information from different video frames.

[0126] According to the second weight vector, multiple video frame codes guided by user questions are weightedly fused to obtain target coding information, thereby eliminating time redundancy in coding and providing strong support for video analysis and processing in subsequent steps.

[0127] S212: Input the target encoding information and the question encoding into the pre-trained language model to obtain a response answer output by the pre-trained language model.

[0128] Among them, step S212 is similar to the above-mentioned step S104 and will not be repeated here.

[0129] The embodiment of the present application provides a video question-answering method, which initializes a key information query vector to obtain a first key information query vector; trains the first key information query vector to obtain an updated second key information query vector; determines an attention weight between a question code and a second key information query vector; determines target key information related to the second key information query vector from the question code according to the attention weight; determines a cosine similarity matrix of a video frame code and target key information; determines a correlation degree between the video frame code and a corresponding position of the target key information according to the cosine similarity matrix; determines a first weight vector at different positions of the video frame code according to the correlation degree; performs weighted processing on the video frame code according to the first weight vector to obtain a video frame code guided by a user question; and determines a second weight vector at the same position of the video frame code guided by the user question at different times according to the cosine similarity matrix; performs weighted fusion on multiple video frame codes guided by the user question according to the second weight vector to obtain target code information; inputs the target code information and the question code into a pre-trained language model to obtain a response answer output by the pre-trained language model, thereby eliminating time and space redundancy in the code, saving computing resources, and improving processing efficiency.

[0130] Figure 3 This is a schematic diagram of the structure of a video question-answering device provided in this application. Figure 3 As shown, the video question-answering device 300 provided in this embodiment includes:

[0131] The processing module 301 is used to encode the user question and the video to be analyzed to obtain the question code corresponding to the user question and multiple continuous video frame codes corresponding to the video to be analyzed;

[0132] A determination module 302, configured to determine target key information from the question encoding according to a pre-trained key information query vector;

[0133] The processing module 301 is further configured to generate a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes, and perform fusion processing on the plurality of video frame codes guided by user questions based on the target key information to obtain target code information;

[0134] The processing module 301 is further used to input the target encoding information and the question encoding into a pre-trained language model to obtain a response answer output by the pre-trained language model.

[0135] Optionally, the processing module 301 is further used to initialize the key information query vector to obtain a first key information query vector, wherein the first key information query vector has the same spatial structure as the video frame encoding, and the first key information query vector has the same dimension as the question encoding and the video frame encoding;

[0136] Training the first key information query vector to obtain an updated second key information query vector;

[0137] The second key information query vector and the question code are analyzed and processed to obtain target key information in the question code.

[0138] Optionally, the determination module 302 is further used to determine an attention weight between the question encoding and the second key information query vector;

[0139] Determine target key information related to the second key information query vector from the question encoding according to the attention weight, wherein the number and dimension of the target key information are the same as the number and dimension of the second key information query vector.

[0140] Optionally, the determination module 302 is further configured to determine, for any one of the plurality of continuous video frame codes, a degree of association between the video frame code and a position corresponding to the target key information;

[0141] Determining first weight vectors at different positions of the video frame encoding according to the association degree;

[0142] The processing module is further used to perform weighted processing on the video frame encoding according to the first weight vector to obtain the video frame encoding guided by the user question.

[0143] Optionally, the determination module 302 is further used to determine a cosine similarity matrix between the video frame encoding and the target key information;

[0144] The degree of association between the video frame encoding and the corresponding position of the target key information is determined according to the cosine similarity matrix.

[0145] Optionally, the determination module 302 is further configured to determine, according to the cosine similarity matrix, a second weight vector for the same position of the video frame encoding guided by the user question at different moments;

[0146] The processing module 301 is further configured to perform weighted fusion on the plurality of video frame codes guided by the user question according to the second weight vector to obtain the target coding information.

[0147] The video question-and-answer device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0148] Figure 4 This is a schematic diagram of the structure of a video question-answering device provided in this application. Figure 4 As shown, the video question-answering device provided by the present application includes: a receiver 401 , a transmitter 402 , a processor 403 and a memory 404 .

[0149] Receiver 401, for receiving instructions and data;

[0150] A transmitter 402, used for sending instructions and data;

[0151] Memory 404, for storing computer-executable instructions;

[0152] The processor 403 is used to execute the computer-executable instructions stored in the memory 404 to implement the various steps performed by the video question-answering method in the above embodiment. For details, please refer to the relevant description in the above video question-answering method embodiment.

[0153] Optionally, the memory 404 may be independent or integrated with the processor 403 .

[0154] When the memory 404 is independently provided, the electronic device further includes a bus for connecting the memory 404 and the processor 403 .

[0155] The present application also provides a computer storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, a video question and answer method as performed by the above-mentioned video question and answer device is implemented.

[0156] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0157] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0158] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video question-answering method, characterized in that: The method comprises: Encoding the user question and the video to be analyzed to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the video to be analyzed; determining target key information from the question encoding according to a pre-trained key information query vector; generating a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes, and fusing the plurality of video frame codes guided by user questions based on the target key information to obtain target code information; The target encoding information and the question encoding are input into a pre-trained language model to obtain a response answer output by the pre-trained language model.

2. The method according to claim 1, characterized in that The determining target key information from the question encoding according to the pre-trained key information query vector includes: Initializing the key information query vector to obtain a first key information query vector, wherein the first key information query vector has the same spatial structure as the video frame code, and the first key information query vector has the same dimension as the question code and the video frame code; Training the first key information query vector to obtain an updated second key information query vector; The second key information query vector and the question code are analyzed and processed to obtain target key information in the question code.

3. The method according to claim 2, characterized in that The analyzing and processing the second key information query vector and the question code to obtain the target key information in the question code includes: determining an attention weight between the question encoding and the second key information query vector; Determine target key information related to the second key information query vector from the question encoding according to the attention weight, wherein the number and dimension of the target key information are the same as the number and dimension of the second key information query vector.

4. The method according to claim 1, characterized in that The step of generating a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes comprises: For any one video frame code among the plurality of continuous video frame codes, determining a correlation degree between the video frame code and a position corresponding to the target key information; Determining first weight vectors at different positions of the video frame encoding according to the association degree; The video frame code is weighted according to the first weight vector to obtain the video frame code guided by the user question.

5. The method according to claim 4, characterized in that The determining, for any one of the plurality of continuous video frame codes, a degree of association between the video frame code and a position corresponding to the target key information comprises: Determining a cosine similarity matrix of the video frame encoding and the target key information; The degree of association between the video frame encoding and the corresponding position of the target key information is determined according to the cosine similarity matrix.

6. The method according to claim 5, characterized in that The step of fusing the plurality of video frame codes guided by the user question based on the target key information to obtain target coding information includes: Determine, according to the cosine similarity matrix, a second weight vector for the same position of the video frame encoding guided by the user question at different times; The target coding information is obtained by weightedly merging the plurality of video frame codes guided by the user question according to the second weight vector.

7. A video question-answering device, characterized in that: The device comprises: A processing module, used to encode the user question and the video to be analyzed, to obtain a question code corresponding to the user question and a plurality of continuous video frame codes corresponding to the video to be analyzed; A determination module, configured to determine target key information from the question encoding according to a pre-trained key information query vector; The processing module is further configured to generate a plurality of video frame codes guided by user questions according to the target key information and the plurality of continuous video frame codes, and perform fusion processing on the plurality of video frame codes guided by user questions based on the target key information to obtain target code information; The processing module is further used to input the target encoding information and the question encoding into a pre-trained language model to obtain a response answer output by the pre-trained language model.

8. The device according to claim 7, characterized in that The processing module is further used to initialize the key information query vector to obtain a first key information query vector, wherein the first key information query vector has the same spatial structure as the video frame code, and the first key information query vector has the same dimension as the question code and the video frame code; Training the first key information query vector to obtain an updated second key information query vector; The second key information query vector and the question code are analyzed and processed to obtain target key information in the question code.

9. A video question-answering device, characterized in that: include: Memory; processor; Wherein, the memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the video question-answering method according to any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which, when executed by a processor, are used to implement the video question-answering method according to any one of claims 1 to 6.

Images