Method, apparatus, computer device, computer-readable storage medium, and computer program product for generating answers to video understanding questions

Abstract

The present application relates to the field of video understanding technology, and in particular to a method, device, computer equipment, computer-readable storage medium, and computer program product for generating answers to video understanding questions. The method comprises: obtaining long video data and video understanding questions; dividing the long video data into multiple short video clips, and using a short-term visual subtitle generator to generate corresponding description texts based on the image information of each short video clip; splicing the description texts in sequence according to the time sequence corresponding to each short video clip to obtain a target description text; inputting the target description text and the video understanding question into a large language model to obtain an answer corresponding to the video understanding question generated by the large language model based on the target description text. The use of this method can solve the problem of high cost and long cycle of inference training of existing long video understanding models, simplify the complexity of long video understanding, and improve the efficiency and accuracy of question and answer generation in long video understanding.

Description

Technical Field

[0001] The present application relates to the technical field of video understanding, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for generating answers to video understanding questions. Background Art

[0002] Video understanding is a research direction in computer vision and artificial intelligence that aims to understand video content by analyzing and interpreting video data. In recent years, video understanding technology for short videos with a duration of five to fifteen seconds has made significant progress.

[0003] However, it is not easy to extend the current video understanding models for short videos to long videos of several minutes or hours. Existing long video understanding models rely on expensive and complex long-term temporal modeling schemes, which are prone to long inference training cycles. Summary of the invention

[0004] Based on this, it is necessary to provide an efficient method, device, computer equipment, computer-readable storage medium and computer program product for generating answers to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for generating answers to video understanding questions, comprising:

[0006] Obtaining long video data and a video understanding problem corresponding to the long video data;

[0007] The long video data is divided into a plurality of short video segments, and a short-term visual subtitle generator is used to generate a description text corresponding to each of the short video segments based on the image information of each of the short video segments;

[0008] According to the time sequence corresponding to each of the short video clips, sequentially splice the description texts corresponding to each of the short video clips to obtain the target description text;

[0009] The target description text and the video understanding question are input into a large language model to obtain an answer corresponding to the video understanding question generated by the large language model based on the target description text.

[0010] In one embodiment, the step of inputting the target description text and the video understanding question into a large language model to obtain an answer corresponding to the video understanding question generated by the large language model based on the target description text includes:

[0011] Determine the attention distribution of the target description text by using the similarity between each of the description texts through the large language model;

[0012] Extracting answer candidate content from the target description text according to the attention distribution using the large language model;

[0013] The large language model is used to generate an answer corresponding to the video understanding question according to the video understanding question and the answer candidate content.

[0014] In one embodiment, extracting answer candidate content from the target description text using the large language model according to the attention distribution includes:

[0015] The large language model is used to filter out text content whose attention distribution is lower than a preset threshold from the target description text to obtain the answer candidate content.

[0016] In one embodiment, the training method of the short-term visual caption generator includes:

[0017] Acquire a short video image sample and a description text sample corresponding to the short video image sample, and input the short video image sample and the description text sample into an initial subtitle generator;

[0018] Decoding the short video image sample by the subtitle generator to obtain corresponding image information, and using the image information to generate a predicted description text corresponding to the short video image sample;

[0019] Decoding the description text sample by the subtitle generator to obtain corresponding text information, determining a generation loss value between the image information, the text information and the predicted description text, a matching loss value between the image information and the text information, and a model loss value between the image information and the text information;

[0020] The subtitle generator adjusts model parameters based on the generation loss value, the matching loss value and the model loss value, and uses the subtitle generator after the model parameters are adjusted as the short-term visual subtitle generator.

[0021] In one embodiment, dividing the long video data into a plurality of short video segments, and using a short-term visual subtitle generator to generate a description text corresponding to each of the short video segments, comprises:

[0022] Segmenting the long video data according to a unit sampling threshold to obtain a plurality of short video segments whose segment lengths match the unit sampling threshold;

[0023] Inputting the video frame image of each of the short video clips into the short-term visual caption generator;

[0024] Decoding the video frame image by the short-term visual caption generator to obtain target image information corresponding to the video frame image;

[0025] The target image information is processed by the short-term visual caption generator using target model parameters to obtain description text corresponding to the target image information.

[0026] In one embodiment, the training method of the large language model includes:

[0027] Obtain a video description sample corresponding to the long video sample, a question sample corresponding to the long video sample, and an answer sample corresponding to the long video sample;

[0028] Constructing a video question-answering sentence using the video description sample, the question sample, and the answer sample, and inputting the video question-answering sentence into an initial large model;

[0029] The video description samples and the answer samples are used to train the attention mechanism network of the initial large model, and the answer samples and the question samples are used to train the long video understanding network of the initial large model, and the trained initial large model is used as the large language model.

[0030] In a second aspect, the present application also provides a device for generating answers to video understanding questions, comprising:

[0031] A data acquisition module, used to acquire long video data and video understanding problems corresponding to the long video data;

[0032] A text generation module, used to divide the long video data into a plurality of short video segments, and use a short-term visual subtitle generator to generate a description text corresponding to each of the short video segments based on the image information of each of the short video segments;

[0033] A text splicing module, used to sequentially splice the description texts corresponding to each of the short video clips according to the time sequence corresponding to each of the short video clips, to obtain a target description text;

[0034] The answer generation module is used to input the target description text and the video understanding question into the large language model to obtain an answer corresponding to the video understanding question generated by the large language model based on the target description text.

[0035] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for generating an answer to a video understanding question as described in any one of the embodiments of the first aspect is implemented.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating an answer to a video understanding question as described in any one of the embodiments of the first aspect.

[0037] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method for generating an answer to a video understanding question described in any one of the embodiments of the first aspect above is implemented.

[0038] The above-mentioned method, device, computer equipment, computer-readable storage medium and computer program product for generating answers to video understanding problems can simplify the complexity of long video understanding and improve the efficiency and accuracy of question-answer generation in long video understanding by dividing the long video understanding task into two stages of processing tasks. In the first stage, the long video data is divided into multiple short video clips, and a short-term visual subtitle generator is used to generate a description text corresponding to each short video clip based on the image information of each short video clip. In the second stage, the description texts corresponding to each short video clip are sequentially spliced ​​in accordance with the time sequence corresponding to each short video clip to obtain the target description text, and the target description text and the video understanding problem are input into the large language model to obtain the answer corresponding to the video understanding problem generated by the large language model based on the target description text. In addition, in the above-mentioned method for generating answers to video understanding problems, since the description texts of multiple short video clips generated in the first stage are used to generate the target description text input into the large language model in the second stage, parameter sharing and decoupling in long video understanding can be achieved, the pre-training process of the large language model is optimized, and the training efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A diagram of an application environment of a method for generating answers to video understanding questions in one embodiment;

[0041] Figure 2 A schematic diagram of a flow chart of a method for generating answers to video understanding questions in one embodiment;

[0042] Figure 3 A flowchart of an answer generation step in one embodiment;

[0043] Figure 4 A schematic diagram of a process flow of a short-term visual caption generator training step in one embodiment;

[0044] Figure 5 is a structural block diagram of a subtitle generator 500 in one embodiment;

[0045] Figure 6 A flowchart illustrating a text generation step in one embodiment;

[0046] Figure 7 A flowchart of a large language model training step in one embodiment;

[0047] Figure 8 A flowchart of a method for generating answers to video understanding questions in another embodiment;

[0048] Fig. 9 A schematic diagram of a method for generating answers to video understanding questions in one embodiment;

[0049] Fig.10 is a structural block diagram of a device 1000 for generating answers to video comprehension questions in one embodiment;

[0050] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0053] The method for generating answers to video understanding questions provided in the embodiments of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Specifically, the server 104 can obtain long video data and video understanding problems corresponding to the long video data from the terminal 102. The server 104 can divide the long video data into multiple short video clips, and use a short-term visual subtitle generator based on the image information of each short video clip to generate a description text corresponding to each short video clip. The server 104 can sequentially splice the description text corresponding to each short video clip in the time sequence corresponding to each short video clip to obtain the target description text. The server 104 can input the target description text and the video understanding problem into the large language model to obtain the answer corresponding to the video understanding problem generated by the large language model based on the target description text. Optionally, in some embodiments, the server 104 can also return the answer to the generated video understanding problem to the terminal 102 for display.

[0054] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart TVs, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0055] In an exemplary embodiment, Figure 2 As shown, a method for generating answers to video understanding questions is provided. This embodiment applies this method to Figure 1 Taking the server 104 in the example as an example, the method includes the following steps S202 to S208. Among them:

[0056] Step S202, obtaining long video data and video understanding questions corresponding to the long video data.

[0057] Among them, long video data can be used to represent video data with a duration of several minutes, several hours or longer.

[0058] Video comprehension questions can be used to express questions about the content of the video, the events occurring in the video, or the actions of the operator recorded in the video.

[0059] Exemplarily, the server may obtain the long video data and the video comprehension questions corresponding to the long video data from a local database, a data communication network or a terminal. Optionally, in some embodiments, the video comprehension questions may include but are not limited to "what operations the operator performed in the video", "whether the actions of the operator in the video meet the specifications", "whether the equipment in the video is working properly", etc.

[0060] Step S204: segment the long video data into multiple short video segments, and use a short-term visual subtitle generator to generate a description text corresponding to each short video segment based on the image information of each short video segment.

[0061] Among them, short video clips can be used to represent video clips with a duration of five to fifteen seconds.

[0062] The short-term visual caption generator can be used to represent a text generation model.

[0063] Description text can be used to describe the screen content of a short video clip.

[0064] Exemplarily, the server may divide the long video data into short video segments of a corresponding number of segments according to a preset number of segments. Alternatively, the server may also divide the long video data into short video segments of each segment length according to a preset segment duration. Each short video segment is input into a short-term visual subtitle generator, and the short-term visual subtitle generator selects a description sample corresponding to the image sample as the description text corresponding to the short video segment currently being processed according to the matching relationship between the image information of each short video segment and the image sample learned in the pre-training stage.

[0065] Step S206, sequentially splicing the description texts corresponding to each short video segment according to the time sequence corresponding to each short video segment to obtain the target description text.

[0066] For example, the server may splice the description texts corresponding to each short video segment in the time sequence in which each short video segment is arranged in the long video data, to form a target description text in a sequence form. For example, when the description texts of the short video segments include C1, C2, and C3, the target description text C=[C1, C2, C3] may be spliced.

[0067] Step S208: input the target description text and the video understanding question into the large language model to obtain an answer corresponding to the video understanding question generated by the large language model based on the target description text.

[0068] For example, the server can form a prompt sentence with the target description text and the video understanding question, and input the prompt sentence into the large language model. The target description text is processed by the large language model based on the pre-trained model parameters, and the keywords in the target description text are extracted, and the answer corresponding to the video understanding question is obtained by combining the keywords. Among them, the keywords can be used to represent the words obtained by summarizing and inducing the target description text.

[0069] In the above-mentioned method for generating answers to video understanding problems, by dividing the long video understanding task into two stages of processing tasks, in the first stage: the long video data is divided into multiple short video clips, and a short-term visual subtitle generator is used to generate a description text corresponding to each short video clip based on the image information of each short video clip. In the second stage: according to the time sequence corresponding to each short video clip, the description text corresponding to each short video clip is sequentially spliced ​​to obtain the target description text, and the target description text and the video understanding problem are input into the large language model to obtain the answer corresponding to the video understanding problem generated by the large language model based on the target description text. This can simplify the complexity of long video understanding and improve the efficiency and accuracy of question-answering generation in long video understanding. In addition, in the above-mentioned method for generating answers to video understanding problems, since the description texts of multiple short video clips generated in the first stage are used to generate the target description text input into the large language model in the second stage, parameter sharing and decoupling in long video understanding can be achieved, the pre-training process of the large language model is optimized, and the training efficiency is improved.

[0070] In an exemplary embodiment, Figure 3 As shown, the above step S208 may also include the following steps S302 to S306. Among them:

[0071] Step S302, using a large language model and utilizing the similarity between each description text to determine the attention distribution of the target description text.

[0072] Among them, attention distribution can be used to enable the large language model to focus on important information parts in the target description text when processing the sequence of target description text.

[0073] Exemplarily, an attention network may be deployed in the large language model. The server may use the attention network of the large language model to process the description text corresponding to each short video clip in the target description text to obtain the similarity between each description text. The probability distribution is calculated using the similarity between each description text to obtain the attention distribution of the target description text.

[0074] Step S304: extract answer candidates from the target description text according to the attention distribution using a large language model.

[0075] Step S306: Generate an answer corresponding to the video understanding question based on the video understanding question and the candidate answer contents using the large language model.

[0076] For example, the server can extract text content with a higher attention distribution parameter from the target description text as the answer candidate content according to the attention distribution. By matching the video understanding network and the answer candidate content using a large language model, the answer candidate content with a higher matching degree can be selected as the answer corresponding to the video understanding question, or multiple answer candidate contents with a higher matching degree can be spliced ​​to form the answer corresponding to the video understanding question.

[0077] In this embodiment, a large language model is used based on an attention mechanism to determine the attention distribution of the target description text using the similarity between each description text, and candidate answer content is extracted from the target description text based on the attention distribution. The answer corresponding to the video comprehension question is generated based on the video comprehension question and the candidate answer content. This can avoid redundant operation text content that is irrelevant to the content of the long video data from affecting the accuracy and efficiency of answer generation.

[0078] In an exemplary embodiment, step S304 may further include: using a large language model to filter out text content with attention distribution below a preset threshold from the target description text to obtain answer candidate content.

[0079] Optionally, in some embodiments, the server may use short video clips as a filtering dimension, and use a large language model to filter out the description text of short video clips whose attention distribution is lower than a preset threshold from the target description text. The text content remaining after filtering the target description text constitutes the answer candidate content. Alternatively, in other embodiments, the server may also use text segmentation as a filtering dimension, perform segmentation processing on the target description text, and obtain corresponding multiple text segmentations. The text segmentations whose attention distribution is lower than the preset threshold are filtered out, and the text segmentations remaining after filtering are combined to form the answer candidate content.

[0080] In this embodiment, by filtering out text content with attention distribution below a preset threshold, redundant noisy text irrelevant to the content of long video data can be reduced, which helps to improve the reasoning ability of large language models when processing long, noisy and redundant text inputs.

[0081] In an exemplary embodiment, Figure 4 As shown, a training method for a short-term visual caption generator is also provided, including the following steps S402 to S408. Among them:

[0082] Step S402: Obtain a short video image sample and a description text sample corresponding to the short video image sample, and input the short video image sample and the description text sample into an initial subtitle generator.

[0083] Exemplarily, the server can obtain short video image samples and description text samples corresponding to the short video image samples from a local database or a data communication network, use the short video image samples as training data, and use the description text samples corresponding to the short video image samples as training labels corresponding to the training data, and input them into the initial subtitle generator.

[0084] Step S404: decode the short video image sample through a subtitle generator to obtain corresponding image information, and use the image information to generate a predicted description text corresponding to the short video image sample.

[0085] Exemplarily, the subtitle generator may include an image decoder. The server may input the short video image sample into the image decoder of the subtitle generator, and use the image decoder to decode the short video image sample to obtain image information of the short video image sample. The image information of the short video image sample is processed using the initial model parameters of the subtitle generator, and a predicted description text corresponding to the video image sample is generated based on an autoregressive method.

[0086] Step S406, decoding the description text sample through the subtitle generator to obtain the corresponding text information, determining the generation loss value between the image information, text information and the predicted description text, the matching loss value between the image information and the text information, and the model loss value between the image information and the text information.

[0087] Exemplarily, the subtitle generator may include multiple text decoders. The server may input the description text sample into one of the text decoders, and use the text decoder to decode the description text sample to obtain the corresponding text information. The image information, text information and predicted description text are processed using the initial model parameters of the subtitle generator to obtain the corresponding generation loss value. The description text sample is input into another text decoder, and the image information is input into the cross-attention mechanism network in the text decoder to obtain the matching loss value between the image information and the text information generated by the text decoder. The description text sample is input into another text decoder, and the image information is input into the cross-attention mechanism network of the text decoder to obtain the model loss value between the image information and the text information generated by the text decoder.

[0088] Step S408, the subtitle generator adjusts the model parameters based on the generation loss value, the matching loss value and the model loss value, and uses the subtitle generator after the model parameters are adjusted as a short-term visual subtitle generator.

[0089] Exemplarily, the server can iteratively adjust the model parameters of the subtitle generator based on the generation loss value, matching loss value and model loss value until the number of iterative adjustments reaches a preset threshold or the adjusted generation loss value, matching loss value and model loss value reach a preset loss threshold, and use the subtitle generator with adjusted model parameters as a short-term visual subtitle generator used in the long video understanding application stage.

[0090] In this embodiment, an initial subtitle generator composed of an image decoder and a text decoder is used to decode short video image samples and description text samples, and predicted description text is generated based on the decoded image information. The loss of the subtitle generator is determined based on the image information, text information and predicted description text, and the model parameters are adjusted based on the loss to obtain a short-term visual subtitle generator through training, which can improve the training efficiency and the inference accuracy of the subtitle generator.

[0091] In an exemplary embodiment, Figure 5 As shown, a schematic diagram of the structure of a subtitle generator 500 is provided, including an image decoder 502, a first text decoder 504, an image-based second text decoder 506 and an image-based third text decoder 508.

[0092] Exemplarily, the server may input the short video image sample 10 into the image decoder 502. The short video image sample 10 is decoded by the first feedforward network 5022 and the self-attention mechanism 5024 in the image decoder 502 to obtain the image information 12. The server may generate the predicted description text by autoregression through the subtitle generator 500 with reference to the following formula:

[0093] .

[0094] in, Represents the predicted description text. V represents a short video image sample 10.

[0095] The server may input the description text sample 20 into the first text decoder 504, and decode the description text sample 20 using the second feedforward network 5042, the first cross-attention mechanism 5044, and the first bidirectional self-attention mechanism 5046 of the first text decoder 504 to obtain text information 22. The subtitle generator 500 determines the generation loss value 30 by referring to the following formula:

[0096] .

[0097] in, Indicates that a loss value of 30 is generated. Indicates the similarity between the image information 12 and the text information 22. I represents the image information 12. T represents the text information 22. represents a temperature parameter. N represents a batch size of the image information 12. M represents a batch size of the text information 22.

[0098] The server can input the description text sample 20 into the second image-based text decoder 506, and input the image information 12 into the second cross-attention mechanism 5064. The image information 12 and the text information 22 are processed by the third feedforward network 5062, the second cross-attention mechanism 5064, and the second bidirectional self-attention mechanism 5066 in the second image-based text decoder 506, and the matching loss value 32 is obtained by referring to the following formula:

[0099] .

[0100] in, Indicates a matching loss value of 32. Indicates a match tag. represents the probability of matching the image information 12 with the text information 22. The definitions of the remaining parameters are the same as the above formula.

[0101] The server may input the description text sample 20 into the third image-based text decoder 508, and input the image information 12 into the third cross-attention mechanism 5084. The image information 12 and the text information 22 are processed by the fourth feedforward network 5082, the third cross-attention mechanism 5084, and the random self-attention mechanism 5086 in the third image-based text decoder 508, and the model loss value 34 is obtained by referring to the following formula:

[0102] .

[0103] in, It can represent the model loss value 34. Represents the tth word in the text information 22. Represents the model parameters. The definitions of the remaining parameters are the same as the above formula.

[0104] In this embodiment, the image description is generated by the image-guided text decoder autoregression, so that the subsequently generated description text is strongly correlated with the image information of the short video clip, thereby improving the generation accuracy of the description text.

[0105] In an exemplary embodiment, Figure 6 As shown, step S204 may also include the following steps S602 to S608. Among them:

[0106] Step S602 : segment the long video data according to the unit sampling threshold to obtain a plurality of short video segments whose segment durations match the unit sampling threshold.

[0107] Exemplarily, the unit sampling threshold may be stored in the server, and segmentation is performed once every unit sampling threshold in the long video data to divide the long video data into a plurality of short video segments whose segment durations match the unit sampling threshold.

[0108] Step S604: input the video frame image of each short video clip into a short-term visual caption generator.

[0109] Step S606: decode the video frame image through the short-term visual caption generator to obtain target image information corresponding to the video frame image.

[0110] Exemplarily, the server may input the video frame images of each short video clip into the short-term visual subtitle generator, respectively, and use the image decoder of the short-term visual subtitle generator to decode the video frame images to obtain the target image information corresponding to the video frame images.

[0111] Step S608: The target image information is processed by the short-term visual caption generator using the target model parameters to obtain a description text corresponding to the target image information.

[0112] The target model parameters may be used to represent the model parameters when the subtitle generator training is completed.

[0113] Exemplarily, the server may refer to the generation formula of the description text provided in the above embodiment through the short-term visual caption generator, use the target model parameters to process the target image information, and obtain the description text corresponding to the target image information.

[0114] In this embodiment, by segmenting the long video data according to the unit sampling threshold, the long video understanding task can be converted into a short video understanding task, which is convenient for accelerating the efficiency of subsequent description text generation. The short video clips are decoded by the short-term visual subtitle generator and the description text corresponding to the short video clips is generated by autoregression, which can improve the accuracy of description text generation.

[0115] In an exemplary embodiment, Figure 7 As shown, a method for training a large language model is also provided, including the following steps S702 to S706. Among them:

[0116] Step S702, obtaining a video description sample corresponding to the long video sample, a question sample corresponding to the long video sample, and an answer sample corresponding to the long video sample.

[0117] Step S704, construct a video question and answer sentence using the video description sample, the question sample and the answer sample, and input the video question and answer sentence into the initial large model.

[0118] Exemplarily, the server can obtain video description samples corresponding to the long video samples, question samples corresponding to the long video samples, and answer samples corresponding to the long video samples from a local database or a data communication network. Multiple rounds of question-answering dialogues are formed using the combination of question samples and answer samples, question prompt sentences are formed using the video description text and question samples, and video question-answering sentences are formed using the splicing of multiple rounds of question-answering dialogues and question prompt sentences. The video question-answering sentences are input into the initial large model.

[0119] Step S706: Use the video description samples and answer samples to train the attention mechanism network of the initial large model, use the answer samples and question samples to train the long video understanding network of the initial large model, and use the trained initial large model as the large language model.

[0120] Exemplarily, the server may use the attention mechanism network of the initial large model to determine the similarity between the video description text and the answer sample, and adjust the network parameters of the attention mechanism network based on the similarity. The long video understanding network of the initial large model is used to determine the matching degree between the question sample and the answer sample, and the network parameters of the long video understanding network are adjusted based on the matching degree. When the inference loss of the trained initial large model reaches a preset loss threshold or the number of iterative trainings of the initial large model reaches a preset number threshold, the trained initial large model is used as the large language model used in the application stage.

[0121] In this embodiment, by training the initial large model in a prompt manner, the training cost of the large language model can be reduced and the training cycle of the large language model can be shortened.

[0122] In an exemplary embodiment, Figure 8 As shown, a method for generating answers to video understanding questions is also provided, comprising the following steps S802 to S814. Among them:

[0123] Step S802: Obtain long video data and video comprehension questions corresponding to the long video data.

[0124] Step S804 , segmenting the long video data according to the unit sampling threshold to obtain a plurality of short video segments whose segment durations match the unit sampling threshold.

[0125] For example, Fig. 9 As shown in FIG. 1 , the server can obtain long video data with a duration of 300 seconds and a video understanding question corresponding to the long video data: "What operation steps did the operator perform in the video?" The long video data is segmented according to the unit sampling threshold of 100 seconds to obtain three short video segments whose segment duration matches the unit sampling threshold of 100 seconds.

[0126] Step S806: input the video frame image of each short video clip into a short-term visual subtitle generator to obtain a description text corresponding to each short video clip.

[0127] For example, Fig. 9 As shown, the server can generate a description text corresponding to the first short video clip through a short-term visual subtitle generator: "The operator connects the oil unloading pipe to the tanker and aligns it with the measuring cylinder", the description text corresponding to the second short video clip is "The operator is checking the quality of the sampled oil in the measuring cylinder", and the description text corresponding to the third short video clip is "The operator takes the sampled oil away from the operating area".

[0128] Step S808, sequentially splicing the description texts corresponding to each short video segment according to the time sequence corresponding to each short video segment to obtain the target description text.

[0129] Step S810, determining the attention distribution of the target description text by using the similarity between each description text through a large language model.

[0130] Step S812, using the large language model to filter out text content with attention distribution below a preset threshold from the target description text to obtain answer candidate content.

[0131] Step S814: Generate an answer corresponding to the video understanding question based on the video understanding question and the candidate answer contents using the large language model.

[0132] Optionally, in some embodiments, the large language model may include but is not limited to pre-trained large language models such as GPT-3.5, Qwen2, and LLaMA3.

[0133] For example, Fig. 9 As shown, the server can splice the description text corresponding to each short video clip in chronological order to form the target description text. The large language model is used to extract the answer candidate content from the target description text based on the attention mechanism, and the answer corresponding to the video understanding question is "After the operator uses the oil in the tanker and checks the oil quality, he takes the sample away from the operation area."

[0134] Optionally, in some embodiments, the server may also use the initial large model to perform overall long-term video understanding. For example, the server may input each short video clip into the initial large model and generate a description text corresponding to each short video clip in the following manner:

[0135] .

[0136] in, Represents the description text of the mth short video clip. Represents the mth short video clip.

[0137] The server can obtain the target description text by referring to the following formula:

[0138] .

[0139] Where C represents the target description text. Represents the description text of N short video clips.

[0140] The server can generate the answer to the video understanding question through the following formula:

[0141] .

[0142] in, It can represent the answer candidate content, C represents the target description text, and Q represents the video understanding question. Represents the answer to the video understanding question.

[0143] In this embodiment, a two-stage long video understanding framework is used to process the long video question-answering task. In the first stage, a short-term visual subtitle generator is used to divide the long video into multiple short segments, and a corresponding text description is generated for each short segment without the need for a special long video framework (such as a memory queue, a state space layer, etc.), which can improve the efficiency of generating the description text. In the second stage, a large language model is used to aggregate, summarize and summarize the description text, and long-term reasoning is performed to generate the corresponding answer, which can improve the reasoning efficiency and accuracy of the answer. The long video understanding task is effectively decomposed into short-term and long-term modeling sub-problems, thereby improving the overall processing efficiency and accuracy. In addition, since the large language model in the second stage can summarize the description text of short video segments with more noise and possible redundancy, and then answer the video understanding question based on the summarized video summary information, it can help to screen out potential interference or irrelevant information and eliminate redundant distances, thereby significantly improving the reasoning ability of the large language model in long video understanding.

[0144] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0145] Based on the same inventive concept, the embodiment of the present application also provides a device for generating answers to video understanding questions for implementing the method for generating answers to video understanding questions involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of the embodiments of the device for generating answers to one or more video understanding questions provided below can refer to the limitations of the method for generating answers to video understanding questions above, and will not be repeated here.

[0146] In an exemplary embodiment, Fig.10 As shown, a device 1000 for generating answers to video comprehension questions is provided, comprising: a data acquisition module 1002, a text generation module 1004, a text splicing module 1006 and an answer generation module 1008, wherein:

[0147] The data acquisition module 1002 is used to acquire long video data and video understanding problems corresponding to the long video data.

[0148] The text generation module 1004 is used to divide the long video data into multiple short video segments, and use a short-term visual subtitle generator to generate a description text corresponding to each short video segment based on the image information of each short video segment.

[0149] The text splicing module 1006 is used to sequentially splice the description text corresponding to each short video segment according to the time sequence corresponding to each short video segment to obtain the target description text.

[0150] The answer generation module 1008 is used to input the target description text and the video understanding question into the large language model to obtain the answer corresponding to the video understanding question generated by the large language model based on the target description text.

[0151] In an exemplary embodiment, the answer generation module 1008 includes: an attention determination unit, which is used to determine the attention distribution of the target description text using the similarity between each description text through a large language model; an answer screening unit, which is used to extract answer candidate content from the target description text according to the attention distribution using the large language model; and an answer generation unit, which is used to generate an answer corresponding to the video understanding question based on the video understanding question and the answer candidate content using the large language model.

[0152] In an exemplary embodiment, the answer screening unit is further used to use a large language model to filter out text content with attention distribution below a preset threshold from the target description text to obtain candidate answer content.

[0153] In an exemplary embodiment, the answer generation device 1000 for video understanding questions also includes: a generator training module, which is used to obtain short video image samples and description text samples corresponding to the short video image samples, and input the short video image samples and the description text samples into an initial subtitle generator; the subtitle generator decodes the short video image samples to obtain corresponding image information, and uses the image information to generate predicted description text corresponding to the short video image samples; the subtitle generator decodes the description text samples to obtain corresponding text information, determines the generation loss value between the image information, the text information and the predicted description text, the matching loss value between the image information and the text information, and the model loss value between the image information and the text information; the subtitle generator adjusts the model parameters based on the generation loss value, the matching loss value and the model loss value, and uses the subtitle generator after the model parameters are adjusted as a short-term visual subtitle generator.

[0154] In an exemplary embodiment, the text generation module 1004 includes: a segmentation unit, used to segment long video data according to a unit sampling threshold to obtain multiple short video segments whose segment duration matches the unit sampling threshold; an image decoding unit, used to input the video frame image of each short video segment into a short-term visual subtitle generator; the short-term visual subtitle generator is used to decode the video frame image to obtain target image information corresponding to the video frame image; and a text generation unit, used to process the target image information using the target model parameters through the short-term visual subtitle generator to obtain a description text corresponding to the target image information.

[0155] In an exemplary embodiment, the answer generation device 1000 for video understanding questions also includes: a large model training module, which is used to obtain video description samples corresponding to long video samples, question samples corresponding to long video samples, and answer samples corresponding to long video samples; construct video question and answer sentences using video description samples, question samples, and answer samples, and input the video question and answer sentences into the initial large model; train the attention mechanism network of the initial large model using video description samples and answer samples, train the long video understanding network of the initial large model using answer samples and question samples, and use the trained initial large model as the large language model.

[0156] Each module in the apparatus 1000 for generating answers to the above-mentioned video comprehension questions can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0157] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig.11 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as long video data, video understanding questions, short video clips, description texts, target description texts, answers to video understanding questions, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating answers to video understanding questions is implemented.

[0158] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0160] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0161] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0162] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0163] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0164] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for generating answers to video understanding questions, characterized in that: The method comprises: Obtaining long video data and a video understanding problem corresponding to the long video data; The long video data is divided into a plurality of short video segments, and a short-term visual subtitle generator is used to generate a description text corresponding to each of the short video segments based on the image information of each of the short video segments; According to the time sequence corresponding to each of the short video clips, sequentially splice the description texts corresponding to each of the short video clips to obtain the target description text; Inputting the target description text and the video understanding question into a large language model, and obtaining an answer corresponding to the video understanding question generated by the large language model based on the target description text; The training method of the short-term visual caption generator includes: Acquire a short video image sample and a description text sample corresponding to the short video image sample, and input the short video image sample and the description text sample into an initial subtitle generator, wherein the subtitle generator includes an image decoder, a first text decoder, an image-based second text decoder, and an image-based third text decoder; Decoding the short video image sample by the image decoder to obtain corresponding image information, and generating a predicted description text corresponding to the short video image sample by the subtitle generator using the image information; Decoding the description text sample by the first text decoder to obtain corresponding text information, and determining a generation loss value between the image information, the text information and the predicted description text based on the similarity between the image information and the text information; Input the description text sample into the second image-based text decoder, input the image information into the second cross-attention mechanism of the second image-based text decoder, and process the image information and the text information using the third feedforward network, the second cross-attention mechanism, and the second bidirectional self-attention mechanism of the second image-based text decoder to obtain a matching loss value between the image information and the text information; Inputting the description text sample into the third image-based text decoder, inputting the image information into the third cross-attention mechanism of the third image-based text decoder, and processing the image information and the text information using the fourth feedforward network, the third cross-attention mechanism, and the random self-attention mechanism of the third image-based text decoder to obtain a model loss value between the image information and the text information; The subtitle generator adjusts model parameters based on the generation loss value, the matching loss value and the model loss value, and uses the subtitle generator after the model parameters are adjusted as the short-term visual subtitle generator.

2. The method according to claim 1, characterized in that: The step of inputting the target description text and the video understanding question into a large language model to obtain an answer corresponding to the video understanding question generated by the large language model based on the target description text includes: Determine the attention distribution of the target description text by using the similarity between each of the description texts through the large language model; Extracting answer candidate content from the target description text according to the attention distribution using the large language model; The large language model is used to generate an answer corresponding to the video understanding question according to the video understanding question and the answer candidate content.

3. The method according to claim 2, characterized in that The extracting answer candidate content from the target description text according to the attention distribution using the large language model includes: The large language model is used to filter out text content whose attention distribution is lower than a preset threshold from the target description text to obtain the answer candidate content.

4. The method according to claim 1, characterized in that The step of dividing the long video data into a plurality of short video segments and using a short-term visual subtitle generator to generate a description text corresponding to each of the short video segments includes: Segmenting the long video data according to a unit sampling threshold to obtain a plurality of short video segments whose segment lengths match the unit sampling threshold; Inputting the video frame image of each of the short video clips into the short-term visual caption generator; Decoding the video frame image by the short-term visual caption generator to obtain target image information corresponding to the video frame image; The target image information is processed by the short-term visual caption generator using target model parameters to obtain description text corresponding to the target image information.

5. The method according to claim 1, characterized in that The training method of the large language model includes: Obtain a video description sample corresponding to the long video sample, a question sample corresponding to the long video sample, and an answer sample corresponding to the long video sample; Constructing a video question-answering sentence using the video description sample, the question sample, and the answer sample, and inputting the video question-answering sentence into an initial large model; The video description samples and the answer samples are used to train the attention mechanism network of the initial large model, and the answer samples and the question samples are used to train the long video understanding network of the initial large model, and the trained initial large model is used as the large language model.

6. A device for generating answers to video comprehension questions, characterized in that: The device comprises: A data acquisition module, used to acquire long video data and video understanding problems corresponding to the long video data; A text generation module, used to divide the long video data into a plurality of short video segments, and use a short-term visual subtitle generator to generate a description text corresponding to each of the short video segments based on the image information of each of the short video segments; A text splicing module, used to sequentially splice the description texts corresponding to each of the short video clips according to the time sequence corresponding to each of the short video clips, to obtain a target description text; An answer generation module, used for inputting the target description text and the video understanding question into a large language model, and obtaining an answer corresponding to the video understanding question generated by the large language model based on the target description text; A generator training module is used to obtain a short video image sample and a description text sample corresponding to the short video image sample, and input the short video image sample and the description text sample into an initial subtitle generator, wherein the subtitle generator includes an image decoder, a first text decoder, an image-based second text decoder, and an image-based third text decoder; the short video image sample is decoded by the image decoder to obtain the corresponding image information, and the subtitle generator uses the image information to generate a predicted description text corresponding to the short video image sample; the description text sample is decoded by the first text decoder to obtain the corresponding text information, and a generation loss value between the image information, the text information and the predicted description text is determined based on the similarity between the image information and the text information; the description text sample is input into the image-based second text decoder, and the image information is input into the The second cross-attention mechanism of the image-based second text decoder is used to process the image information and the text information using the third feedforward network, the second cross-attention mechanism and the second bidirectional self-attention mechanism of the image-based second text decoder to obtain a matching loss value between the image information and the text information; the description text sample is input into the image-based third text decoder, the image information is input into the third cross-attention mechanism of the image-based third text decoder, the image information and the text information are processed using the fourth feedforward network, the third cross-attention mechanism and the random self-attention mechanism of the image-based third text decoder to obtain a model loss value between the image information and the text information; the model parameters are adjusted by the subtitle generator based on the generation loss value, the matching loss value and the model loss value, and the subtitle generator after the model parameters are adjusted is used as the short-term visual subtitle generator.

7. The device according to claim 6, characterized in that The answer generation module comprises: An attention determination unit, configured to determine the attention distribution of the target description text by using the large language model and the similarity between each of the description texts; An answer screening unit, configured to extract answer candidate content from the target description text using the large language model according to the attention distribution; An answer generation unit is used to generate an answer corresponding to the video understanding question based on the video understanding question and the answer candidate content by using the large language model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images