A method and device for enhancing long video understanding aligned with audio information

Abstract

The present invention provides a method and device for enhancing the understanding of long videos aligned with audio information, which solves the problem that the audio information in the video is not considered when processing video data in the prior art, and the training and reasoning costs of large language models are high. 1) Through the cross-attention module, the sequence length of the video picture features is reduced while enhancing the fusion of video spatiotemporal features, further reducing the training cost; 2) By encoding the audio information in the video and downsampling it to the sequence length of the video picture features, the audio modality in the video is aligned; 3) The audio modality and picture modality aligned in the time dimension are added to fuse the information. The audio modality is downsampled and aligned with the length of the video picture modality selected at intervals, retaining all the information. Then it is added to the picture modality to make up for the information missing problem in the picture modality caused by the interval frame extraction.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method and device for enhancing long video understanding aligned with audio information. Background Art

[0002] Existing video understanding solutions are all based on large visual-language models (LVLMs), which encode the image information of video frames and feed them into a large language model together with the encoded features of the text question, and use the corresponding text answer as a label for model training, so as to align the text space with the video feature space, thereby achieving the video understanding task. Furthermore, for long video understanding, existing solutions often extract as many frames as possible and feed them into the model, so as to preserve the spatial and temporal information of the video as much as possible.

[0003] Although the existing technical solutions are relatively simple to implement, they have the following disadvantages: 1) The existing solutions do not utilize the audio information in the video: that is, the existing solutions only use the picture information of the video as the input of the model, but ignore the audio information in the video (such as background music, words spoken by people in the video). Therefore, the obtained video understanding model cannot correctly understand the audio information in the video, and thus cannot accurately understand the video; 2) The existing solutions for understanding long videos usually extract as many frames as possible to retain the spatial and temporal information of the long video: however, this will greatly increase the input sequence length (the number of video tokens) of the large language model, thereby greatly increasing the training and reasoning cost of the large language model. Summary of the invention

[0004] The present invention provides a method and device for enhancing long video understanding aligned with audio information, so as to solve the problems existing in the prior art that when processing video data, the audio information in the video is not considered and the cost of training and reasoning of a large language model is high.

[0005] In a first aspect, the present invention provides a method for enhancing long video understanding aligned with audio information, which specifically comprises the following steps:

[0006] Step S1, obtaining video data DATA1, extracting picture information and audio information of the data DATA1, and forming A-frame video frame data DATA2 and audio data DATA3;

[0007] Step S2, extracting K video frames at equal intervals from the data DATA2, and encoding the K video frames by a video encoder to generate a size of Video coding features M (dense video picture features);

[0008] Step S3, extracting N video frames at equal intervals from the data DATA2, and encoding the N video frames by a video encoder to generate a size of Video coding features X (sparse video picture features); or

[0009] From the video coding feature M, the video coding features corresponding to N frames of video frames are extracted at equal intervals to form a Video coding features X;

[0010] Step S4: Input X as Query and M as Key and Value into the Resampler module for cross attention processing to form a Features O;

[0011] Step S5, encoding the audio data DATA3 to form an audio coding feature Y;

[0012] Step S6: Input the audio coding feature Y into the downsampling module to form a size of Audio coding feature P;

[0013] Step S7, superimposing the feature O and the feature P to form a feature R; or

[0014] The feature O and the feature P are concatenated to form a feature R.

[0015] Wherein, in step S2, the K video frames include a start frame and an end frame.

[0016] Wherein, in step S3, the N video frames include a start frame and an end frame.

[0017] Among them, N, K, K1, N1 are all positive integers, K>N; ; .

[0018] Among them, B and D are both positive integers; B represents the batch size (batch_size); H represents height; W represents width; and D represents dimension.

[0019] Preferably, a method for enhancing long video understanding aligned with audio information further includes:

[0020] Step S8: input the feature R, the text Prompt-P and the answer L to the text Prompt-P into the large language model LLM, train the large language model through the loss function, and form a trained large language model LLM-New.

[0021] Preferably, in step S1, the extraction of the picture information and audio information of the data DATA1 is implemented by using TorchVideo.

[0022] Wherein, in step S3, the “correspondence” refers to the video coding features formed after the i-th frame (i is a positive integer) of the video frame is encoded by the video encoder.

[0023] Preferably, in step S4, X is used as Query, M as Key and Value, and input into the Resampler module for cross attention processing to form a The feature O specifically includes the following steps:

[0024] Step S401: pass through three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V;

[0025] Step S402: Perform Softmax operation on feature Q, feature K and feature V to form feature O.

[0026] Preferably, in step S401, three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V, which are specifically expressed as follows:

[0027] Q=W Q X

[0028] K=W K M

[0029] V=W V M

[0030] Preferably, in step S402, a Softmax operation is performed on the feature Q, the feature K and the feature V to form a feature O, which is specifically represented as follows:

[0031] O = Softmax(QK T )V

[0032] Where T stands for transpose.

[0033] Preferably, in step S6, the downsampling module includes a convolution layer and a pooling layer.

[0034] In this application, the feature superposition means matrices of the same size, where the elements at the same position in each matrix are added together. For example, two 2×2×1 matrices a and b, matrix a is , the matrix b is , then the superposition of matrix a and matrix b is a 2×2×1 matrix c, and matrix c is ; The feature concatenation means concatenating matrix a and matrix b to form a 2×2×2 matrix d; feature sum O and feature P are superimposed to form feature R of size B×N1×D; feature O and feature P are concatenated to form a feature R of size Features R.

[0035] In a second aspect, the present invention further provides a long video understanding enhancement device aligned with audio information, which specifically includes the following modules:

[0036] The video data extraction module is used to obtain the video data DATA1, extract the picture information and audio information of the data DATA1, and form the A-frame video frame data DATA2 and audio data DATA3;

[0037] The first video encoding module is used to extract K frames of video frames at equal intervals from the data DATA2, and perform feature encoding on the K frames of video frames through a video encoder to form a size of Video coding features M (dense video picture features);

[0038] The second video encoding module is used to extract N frames of video frames at equal intervals from the data DATA2, and perform feature encoding on the N frames of video frames through a video encoder to form a size of Video coding features X (sparse video picture features); or

[0039] It is used to obtain the video coding features corresponding to N frames of video frames extracted at equal intervals from the video coding feature M, forming a size of Video coding features X;

[0040] The cross attention module is used to input X as Query, M as Key and Value into the Resampler module for cross attention processing to form a size of Features O;

[0041] An audio encoding module, used for encoding the audio data DATA3 to form an audio encoding feature Y;

[0042] The audio feature sampling module is used to input the audio coding feature Y into the downsampling module to form a size of Audio coding feature P;

[0043] A feature fusion module, used for superimposing the feature O and the feature P to form a feature R; or

[0044] It is used to perform feature splicing on the feature O and the feature P to form feature R.

[0045] Wherein, in the first video encoding module, the K video frames include a start frame and an end frame.

[0046] Wherein, in the second video encoding module, the N video frames include a start frame and an end frame.

[0047] Among them, N, K, K1, N1 are all positive integers, K>N; ; .

[0048] Among them, B and D are both positive integers; B represents the batch size (batch_size); H represents height; W represents width; and D represents dimension.

[0049] Preferably, a long video understanding enhancement device aligned with audio information further includes:

[0050] The large language model training module is used to input the feature R, text Prompt-P and the answer L of the text Prompt-P into the large language model LLM, train the large language model through the loss function, and form a trained large language model LLM-New.

[0051] Preferably, in the video data extraction module, the extraction of the picture information and audio information of the data DATA1 is implemented by using TorchVideo.

[0052] Preferably, in the cross attention module, X is used as the query, M is used as the key and value, and is input into the Resampler module for cross attention processing to form a size of The feature O specifically includes the following sub-modules:

[0053] The first processing submodule is used to pass three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V;

[0054] The second processing submodule is used to perform Softmax operation on feature Q, feature K and feature V to form a Features of O.

[0055] Preferably, the first processing submodule is used to pass three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V, which are specifically expressed as follows:

[0056] Q=W Q X

[0057] K=W K M

[0058] V=W V M

[0059] Preferably, the second processing submodule is used to perform Softmax operation on the feature Q, feature K and feature V to form a The characteristic O is specifically expressed as follows:

[0060] O = Softmax(QK T )V

[0061] Preferably, in the audio feature sampling module, the downsampling module includes a convolution layer and a pooling layer.

[0062] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a long video understanding enhancement method aligned with audio information as described in any one of the first aspects of the present application.

[0063] In a fourth aspect, the present invention further provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing a long video understanding enhancement method aligned with audio information as described in any one of the first aspects of the present application when the computer program is called.

[0064] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0065] The present invention provides a long video understanding enhancement method and device aligned with audio information, which solves the problem that the audio information in the video is not considered when processing video data in the prior art, and the training and reasoning cost of the large language model is high. 1) Through the cross-attention module, the sparse video screen features are used as the query, and the dense video screen features are used as the key and value, so that the rich spatial feature information contained in the key and value are integrated with the time information in the query, and the feature with the length equal to the audio information is obtained, while enhancing the fusion of the video spatiotemporal features, the sequence length of the video screen features is also reduced, and the training cost is further reduced; 2) By encoding the audio information in the video and downsampling it to the sequence length of the video screen features, the audio mode in the video is aligned; 3) The audio mode and the screen mode aligned in the time dimension are added to perform information fusion. At the same time, although the video screen retains the starting frame and the end frame, they are still taken at intervals, which results in some information loss even if cross-attention is performed. At this time, the audio mode is not sampled at intervals, but is downsampled to align the length of the video screen mode, retaining all information. Then it is added to the picture mode to make up for the information missing problem caused by interval frame extraction in the picture mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which constitute part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention. In the accompanying drawings:

[0067] Figure 1 It is a flow chart of a method for enhancing long video understanding aligned with audio information according to a preferred embodiment of the present invention.

[0068] Figure 2 It is a schematic diagram of forming long-term and short-term fusion features of video and audio coding downsampling features in a long video understanding enhancement method aligned with audio information in a preferred embodiment of the present invention.

[0069] Figure 3 It is a flowchart of sparse and dense video frame feature fusion in a long video understanding enhancement method aligned with audio information in a preferred embodiment of the present invention.

[0070] Figure 4 It is a flowchart of a method for enhancing long video understanding for a large language model according to a preferred embodiment of the present invention.

[0071] Figure 5 It is a schematic diagram of the structure of a long video understanding enhancement device aligned with audio information according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0072] The present invention provides a method and device for enhancing the understanding of a long video aligned with audio information. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0073] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0074] Embodiment 1:

[0075] like Figure 1-Figure 4 As shown, the long video understanding enhancement method aligned with audio information described in this embodiment specifically includes the following steps:

[0076] Step S1, obtain video data DATA1, extract the picture information and audio information of the data DATA1, and form A frame video frame data DATA2 and audio data DATA3; wherein the A frame is the total number of frames of the original video, and the extraction of the picture information and audio information of the data DATA1 can be achieved using a video processing tool (such as TorchVideo).

[0077] Step S2, extracting K video frames at equal intervals from the data DATA2, and encoding the K video frames by a video encoder to generate a size of The video coding feature M (dense video picture feature) is obtained; wherein the K video frames include a start frame and an end frame.

[0078] Step S3, extracting N video frames at equal intervals from the data DATA2, and encoding the N video frames by a video encoder to generate a size of Video coding features X (sparse video picture features); or

[0079] From the video coding feature M, the video coding features corresponding to N frames of video frames are extracted at equal intervals to form a The video encoding feature X is as follows; wherein the N video frames include a start frame and an end frame.

[0080] Step S4: Input X as Query and M as Key and Value into the Resampler module for cross attention processing to form a The feature O specifically includes the following steps:

[0081] Step S401: pass through three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V, which are specifically expressed as follows:

[0082] Q=W Q X

[0083] K=W K M

[0084] V=W V M

[0085] Step S402: Softmax operation is performed on the features Q, K and V to form feature O, which is specifically represented as follows:

[0086] O = Softmax(QK T )V

[0087] Where T stands for transpose.

[0088] Step S5: Encode the audio data DATA3 to form an audio coding feature Y.

[0089] Step S6: Input the audio coding feature Y into the downsampling module to form a size of The audio coding feature P of the image is obtained; wherein the downsampling module includes a convolution layer and a pooling layer.

[0090] Step S7, superimposing the feature O and the feature P to form a feature R; or

[0091] The feature O and the feature P are concatenated to form a feature R.

[0092] Among them, N, K, K1, N1 are all positive integers, K>N; ; .

[0093] Among them, B and D are both positive integers; B represents the batch size (batch_size); H represents height; W represents width; and D represents dimension.

[0094] Optionally, a long video understanding enhancement method aligned with audio information further includes:

[0095] Step S8: input the feature R, the text Prompt-P and the answer L to the text Prompt-P into the large language model LLM, train the large language model through the loss function, and form a trained large language model LLM-New.

[0096] like Figure 2 As shown in (left), the original video Video is input into a video processing tool (such as torchvideo) for processing to generate video frame information and audio information A. The video frame information is extracted including the first and last K video frames (i.e., dense frames, also called short-term video frames). Figure 2 As shown in (middle), the dense frames are input into the video encoder, processed by the video encoder to generate video dense features, and at the same time, the video dense features are processed by sparse information extraction to generate video sparse features. Finally, the video dense features and video sparse features are input into the Resampler for processing to generate video long-term and short-term (here "long-term and short-term" refers to the length of the time interval between the extracted equally spaced video frames) fusion features. Figure 2 As shown in (right), the audio information A passes through the audio encoder and downsampling module to generate audio coding downsampling features.

[0097] like Figure 3 As shown in the figure, the sparse video frame features are input into the linear layer 1 as the Query (indicated by Q in the figure), and the dense video frame features are input into the linear layer 2 and the linear layer 3 as the Key (indicated by K in the figure) and the Value (indicated by V in the figure), respectively. Q, K and V are input into the cross attention module, and the long-term and short-term video frame fusion features are generated through the cross attention module.

[0098] like Figure 4 As shown, a video is processed by a long video understanding enhancement method aligned with audio information described in this embodiment to generate a hybrid coding feature, and the hybrid coding feature is input into a large language model to understand the content of the video. Finally, the user inputs a text prompt based on the content of the video to ask the large language model a question, and the large language model outputs the result and feeds it back to the user.

[0099] For example, in a long surveillance video analysis scenario, the method described in this embodiment can be implemented through the following steps.

[0100] Step 1: Extract K frames (including the first and last frames) from the video, and extract N frames with equal intervals from the K frames as long time frames. At the same time, extract the audio information of the video.

[0101] Step 2: Send the video frame information, audio information and text prompt into the model to get the model output (e.g., for accident detection tasks, you can give a prompt like "Is there an accident in the detected video?").

[0102] Step 3. Video content summary and analysis: Based on the model’s answer, the downstream post-processing module can be connected to realize a fully automatic monitoring-processing process. For example, if the current task is “detect whether there is a fire?”, if the model outputs “there is a fire”, then the fire emergency plan can be automatically initiated.

[0103] Through this practical application process, the invention can effectively combine the image modal information and audio modal information in the video to perform long-term monitoring and analysis tasks, and realize automated abnormal situation processing through downstream modules.

[0104] Embodiment 2:

[0105] like Figure 5 As shown, the long video understanding enhancement device aligned with audio information described in this embodiment specifically includes the following modules:

[0106] The video data extraction module is used to obtain the video data DATA1, extract the picture information and audio information of the data DATA1, and form A-frame video frame data DATA2 and audio data DATA3; wherein, the extraction of the picture information and audio information of the data DATA1 can be implemented using a video processing tool (such as TorchVideo).

[0107] The first video encoding module is used to extract K video frames at equal intervals from the data DATA2, and perform feature encoding on the K video frames through a video encoder to form a size of The video coding feature M (dense video picture feature) is obtained; wherein the K video frames include a start frame and an end frame.

[0108] The second video encoding module is used to extract N frames of video frames at equal intervals from the data DATA2, and perform feature encoding on the N frames of video frames through a video encoder to form a size of video coding feature X; or

[0109] It is used to obtain the video coding features corresponding to N frames of video frames extracted at equal intervals from the video coding feature M, forming a size of The video encoding feature X is as follows; wherein the N video frames include a start frame and an end frame.

[0110] The cross attention module is used to input X as Query, M as Key and Value into the Resampler module for cross attention processing to form a size of Features of O.

[0111] Optionally, the cross-attention module specifically includes the following sub-modules:

[0112] The first processing submodule is used to pass three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V, which are specifically expressed as follows:

[0113] Q=W Q X

[0114] K=W K M

[0115] V=W V M

[0116] The second processing submodule is used to perform Softmax operation on feature Q, feature K and feature V to form a The characteristic O is specifically expressed as follows:

[0117] O = Softmax(QK T )V

[0118] Where T stands for transpose.

[0119] The audio encoding module is used to encode the audio data DATA3 to form an audio encoding feature Y.

[0120] The audio feature sampling module is used to input the audio coding feature Y into the downsampling module to form a size of The audio coding feature P wherein the downsampling module comprises a convolutional layer and a pooling layer.

[0121] A feature fusion module, used for superimposing the feature O and the feature P to form a feature R; or

[0122] It is used to perform feature splicing on the feature O and the feature P to form feature R.

[0123] Among them, N, K, K1, N1 are all positive integers, K>N; ; .

[0124] Among them, B and D are both positive integers; B represents the batch size (batch_size); H represents height; W represents width; and D represents dimension.

[0125] Optionally, a long video understanding enhancement device aligned with audio information further includes:

[0126] The large language model training module is used to input the feature R, text Prompt-P and the answer L of the text Prompt-P into the large language model LLM, train the large language model through the loss function, and form a trained large language model LLM-New.

[0127] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.

Claims

1. A method for enhancing long video understanding aligned with audio information, characterized in that: The specific steps include: Step S1, obtaining video data DATA1, extracting picture information and audio information of the data DATA1, and forming A-frame video frame data DATA2 and audio data DATA3; Step S2, extracting K video frames at equal intervals from the data DATA2, and encoding the K video frames by a video encoder to generate a size of Video coding features M; Step S3, extracting N video frames at equal intervals from the data DATA2, and encoding the N video frames by a video encoder to generate a size of Video coding features X; or From the video coding feature M, the video coding features corresponding to N frames of video frames are extracted at equal intervals to form a Video coding features X; Step S4: Input X as Query and M as Key and Value into the Resampler module for cross attention processing to form a Features O; Step S5, encoding the audio data DATA3 to form an audio coding feature Y; Step S6: Input the audio coding feature Y into the downsampling module to form a size of Audio coding feature P; Step S7, superimposing the feature O and the feature P to form a feature R; or The feature O and the feature P are feature spliced ​​to form a feature R; Wherein, in step S2, the K video frames include a start frame and an end frame; Wherein, in step S3, the N video frames include a start frame and an end frame; Among them, N, K, K1, N1 are all positive integers, K>N; ; ; Among them, B and D are both positive integers; B represents the batch size; H represents height; W represents width; and D represents dimension.

2. A method for enhancing long video understanding aligned with audio information according to claim 1, characterized in that: Also includes, Step S8: input the feature R, the text Prompt-P and the answer L to the text Prompt-P into the large language model LLM, train the large language model through the loss function, and form a trained large language model LLM-New.

3. The method for enhancing long video understanding aligned with audio information according to claim 1, characterized in that: In step S4, X is used as the query, M as the key and value, and input into the Resampler module for cross attention processing to form a The feature O specifically includes the following steps: Step S401: pass through three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V; Step S402: Perform Softmax operation on feature Q, feature K and feature V to form feature O.

4. The method for enhancing long video understanding aligned with audio information according to claim 3, characterized in that: In step S401, three linear layers W Q , W K and W V Map feature X and feature M to form feature Q, feature K and feature V, which are specifically expressed as follows: Q=W Q X K=W K M V=W V M Among them, W Q , W K and W V Represent three linear layers respectively.

5. The method for enhancing long video understanding aligned with audio information according to claim 3, characterized in that: In step S402, a Softmax operation is performed on the features Q, K, and V to form a feature O, which is specifically represented as follows: O=Softmax(QK T )V Where T stands for transpose.

6. A device for enhancing understanding of long videos aligned with audio information, characterized in that: The modules include: The video data extraction module is used to obtain the video data DATA1, extract the picture information and audio information of the data DATA1, and form the A-frame video frame data DATA2 and audio data DATA3; The first video encoding module is used to extract K frames of video frames at equal intervals from the data DATA2, and perform feature encoding on the K frames of video frames through a video encoder to form a size of Video coding features M; The second video encoding module is used to extract N frames of video frames at equal intervals from the data DATA2, and perform feature encoding on the N frames of video frames through a video encoder to form a size of Video coding features X; or It is used to obtain the video coding features corresponding to N frames of video frames extracted at equal intervals from the video coding feature M, forming a size of Video coding features X; The cross attention module is used to input X as Query, M as Key and Value into the Resampler module for cross attention processing to form a size of Features O; An audio encoding module, used for encoding the audio data DATA3 to form an audio encoding feature Y; The audio feature sampling module is used to input the audio coding feature Y into the downsampling module to form a size of Audio coding feature P; A feature fusion module, used for superimposing the feature O and the feature P to form a feature R; or Used to perform feature splicing on the feature O and the feature P to form a feature R; Wherein, in step S2, the K video frames include a start frame and an end frame; Wherein, in step S3, the N video frames include a start frame and an end frame; Among them, N, K, K1, N1 are all positive integers, K>N; ; ; Among them, B and D are both positive integers; B represents the batch size; H represents height; W represents width; and D represents dimension.

7. The device for enhancing understanding of long videos aligned with audio information according to claim 6, characterized in that: Also includes, The large language model training module is used to input the feature R, text Prompt-P and the answer L of the text Prompt-P into the large language model LLM, train the large language model through the loss function, and form a trained large language model LLM-New.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements a method for enhancing the understanding of a long video aligned with audio information as described in any one of claims 1-5.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for enhancing the understanding of a long video aligned with audio information as described in any one of claims 1 to 5 is implemented.

Images