Video understanding method, system, electronic device and storage medium

Abstract

The present invention relates to the technical field of multimodal large models, and specifically discloses a video understanding method, system, electronic device, and storage medium, including: extracting frames from a video according to a preset frame extraction interval to obtain multiple video frames and encoding them to obtain multiple initial coding features; calculating the cosine similarity of each two temporally adjacent initial coding features, and determining the initial coding features whose cosine similarity is not greater than a first threshold as target coding features; when the number of target coding features is not less than a second threshold, constructing initial temporal coding features and compressing them to obtain target temporal coding features; mapping the target temporal coding features to obtain mapping coding features, and inputting the mapping coding features and the question text token sequence of the target video into a trained large language model to obtain an answer text. The present invention can effectively extract important change information of the video, avoid information loss, and thus improve the accuracy and efficiency of the large language model for video understanding.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal large models, and in particular to a video understanding method, system, electronic device and storage medium. Background Art

[0002] A large video understanding model is one that, after inputting a video, can provide a textual understanding of the video and, based on the user's questions, provide relevant answers to the video. The main challenge for large video understanding models is how to effectively encode the information in the input video. Specifically, after inputting the video, how to convert the entire video into information that the model can understand. Currently, mainstream large video understanding models typically extract video information by uniformly sampling frames or encoding them frame by frame. Specifically:

[0003] 1) If Figure 2 As shown in FIG, the method of uniformly sampling a video frame is as follows: uniformly sampling the input video to extract a fixed N frames; then sending the frames to the visual encoder for encoding, and finally connecting to the large language model through the projection layer.

[0004] 2) If Figure 3 As shown in the figure, the frame-by-frame encoding method is to first encode each frame of the video, then use a model to compress the encoding information of all frames, and finally connect to the large language model through the projection layer.

[0005] The commonly used method of uniformly sampling video frames is to uniformly sample only eight frames, regardless of the length of the input video. This method does not cause much information loss when the input video is short and simple. However, if the input video is long or densely packed, this method can cause significant information loss, making it difficult for large language models to accurately understand the video.

[0006] Frame-by-frame encoding requires encoding each frame, and the encoded features of each frame are fed into the information compression network. This method provides more information than uniform sampling. Even with long or dense videos, this method does not cause excessive information loss. However, this method has the disadvantages of being slow for long input videos and requiring significant computing resources, potentially causing processing errors.

[0007] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the Invention

[0008] To solve the above technical problems, the present invention provides a video understanding method, system, electronic device and storage medium.

[0009] In a first aspect, the present invention provides a video understanding method, the technical solution of which is as follows:

[0010] Extract frames from the target video according to a preset frame extraction interval to obtain multiple target video frames and encode them respectively to obtain multiple initial video frame encoding features;

[0011] Calculating the cosine similarity between every two temporally adjacent initial video frame coding features, and determining two temporally adjacent initial video frame coding features whose cosine similarity is not greater than a first threshold as target video frame coding features;

[0012] When the number of target video frame coding features is not less than a second threshold, constructing an initial temporal coding feature representing all target video frame coding features and performing feature compression to obtain a target temporal coding feature;

[0013] The target temporal coding features are mapped to obtain mapping coding features, and the mapping coding features and the token sequence of the question text corresponding to the target video are input into the trained large language model to obtain the answer text corresponding to the question text.

[0014] The beneficial effects of a video understanding method of the present invention are as follows:

[0015] The method of the present invention can effectively extract important change information of the video, avoid information loss, and thus improve the accuracy and efficiency of the large language model in understanding the video.

[0016] Based on the above solution, a video understanding method of the present invention can be further improved as follows.

[0017] In an optional manner, the method further includes:

[0018] When the number of target video frame coding features is less than the second threshold, the cosine similarities greater than the first threshold are arranged in ascending order to obtain a target queue;

[0019] Using the first cosine similarity in the target queue as a third threshold, and determining, among the remaining initial video frame coding features, two temporally adjacent initial video frame coding features whose cosine similarity is not greater than the third threshold as target video frame coding features, and determining whether the number of target video frame coding features is less than the second threshold;

[0020] When the number of target video frame coding features is less than the second threshold, the cosine similarities in the target queue are traversed in sequence until the number of target video frame coding features is not less than the second threshold, and then the steps of constructing the initial temporal coding features representing all target video frame coding features and performing feature compression are executed.

[0021] In an optional manner, the step of constructing an initial temporal coding feature that characterizes coding features of all target video frames and performing feature compression to obtain a target temporal coding feature includes:

[0022] The spatiotemporal attention model is used to construct an initial temporal coding feature that characterizes the coding features of all target video frames, and the spatiotemporal attention model is used to perform feature compression on the initial temporal coding feature to obtain the target temporal coding feature of a fixed length.

[0023] In an optional manner, the step of mapping the target temporal coding feature to obtain the mapped coding feature includes:

[0024] The target temporal coding feature is mapped using a projection layer to obtain the mapped coding feature.

[0025] In an optional manner, the training process of the large language model is:

[0026] Obtaining the mapping coding features and the token sequence of the training question text of each training video, and inputting the mapping coding features and the token sequence of the training question text of any training video into the large language model to obtain the training answer text corresponding to the training question text of the training video, and obtaining the loss value of the training video based on the difference between the training answer text and the true answer text of the training video, until the loss value of each training video is obtained;

[0027] According to the loss value of each training video, the parameters of the large language model are optimized to obtain an optimized large language model, and the optimized large language model is used as the large language model and the step of inputting the mapping encoding features of any training video and the token sequence of the training question text into the large language model is returned to execute until the iterative optimization condition is met, and the optimized large language model is determined as the trained large language model.

[0028] In a second aspect, the present invention provides a video understanding system, the technical solution of which is as follows:

[0029] It includes: a first processing module, a calculation module, a second processing module and an operation module;

[0030] The first processing module is used to: extract frames from the target video according to a preset frame extraction interval to obtain multiple target video frames and encode them respectively to obtain multiple initial video frame encoding features;

[0031] The calculation module is used to calculate the cosine similarity between every two temporally adjacent initial video frame coding features, and determine the two temporally adjacent initial video frame coding features whose cosine similarity is not greater than a first threshold as the target video frame coding features;

[0032] The second processing module is configured to: when the number of target video frame coding features is not less than a second threshold, construct an initial temporal coding feature representing all target video frame coding features and perform feature compression to obtain a target temporal coding feature;

[0033] The operation module is used to: map the target temporal coding features to obtain mapping coding features, and input the mapping coding features and the token sequence of the question text corresponding to the target video into the trained large language model to obtain the answer text corresponding to the question text.

[0034] The beneficial effects of a video understanding system of the present invention are as follows:

[0035] The system of the present invention can effectively extract important change information of the video, avoid information loss, and thus improve the accuracy and efficiency of the large language model in understanding the video.

[0036] Based on the above solution, the video understanding system of the present invention can be further improved as follows.

[0037] In an optional manner, the method further includes: a judgment module; the judgment module is configured to:

[0038] When the number of target video frame coding features is less than the second threshold, the cosine similarities greater than the first threshold are arranged in ascending order to obtain a target queue;

[0039] Using the first cosine similarity in the target queue as a third threshold, and determining, among the remaining initial video frame coding features, two temporally adjacent initial video frame coding features whose cosine similarity is not greater than the third threshold as target video frame coding features, and determining whether the number of target video frame coding features is less than the second threshold;

[0040] When the number of target video frame coding features is less than the second threshold, the cosine similarities in the target queue are traversed in sequence until the number of target video frame coding features is not less than the second threshold, and then the second processing module is called.

[0041] In an optional manner, the step of constructing an initial temporal coding feature representing the coding features of all target video frames and performing feature compression to obtain a target temporal coding feature in the second processing module includes:

[0042] The spatiotemporal attention model is used to construct an initial temporal coding feature that characterizes the coding features of all target video frames, and the spatiotemporal attention model is used to perform feature compression on the initial temporal coding feature to obtain the target temporal coding feature of a fixed length.

[0043] In a third aspect, the technical solution of an electronic device of the present invention is as follows:

[0044] The system comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the video understanding method of the present invention are implemented.

[0045] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0046] The computer-readable storage medium stores instructions. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the video understanding method of the present invention.

[0047] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0049] Figure 1 A schematic flow chart of an embodiment of a video understanding method of the present invention;

[0050] Figure 2 Flowchart of the method for uniformly extracting frames from a video;

[0051] Figure 3 Schematic diagram of the flow of the frame-by-frame encoding method;

[0052] Figure 4 This is a flowchart of the second-level frame extraction method;

[0053] Figure 5 A schematic structural diagram of an embodiment of a video understanding system of the present invention;

[0054] Figure 6 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0055] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0056] Figure 1The flowchart of an embodiment of a video understanding method provided by the present invention is shown. The video understanding method can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the video understanding method by calling computer-readable instructions stored in a memory through a processor. Figure 1 As shown, the following steps are included:

[0057] S1. Extract frames from a target video according to a preset frame extraction interval to obtain multiple target video frames and encode them respectively to obtain multiple initial video frame encoding features.

[0058] Among them, the preset frame extraction interval is the video time interval or frame number interval between any two frames. In this embodiment, the default is 1S, that is, one frame is extracted every 1S of video time; the number of frames of the interval can also be determined according to the video frame rate (for example, when the video frame rate is 30FPS, the frame number interval is 30 frames), and there is no restriction here. The target video is the video that needs to be understood according to the question text in this embodiment. The target video frame is the video frame extracted from the target video (if the video is 60S, the number of target video frames is 60). The target video frame is encoded using a visual encoder, and each target video frame corresponds to an initial video frame encoding feature. The feature sequence of multiple initial video frame encoding features is F_all(t), t≤N, N represents the number of initial video frame encoding features, t is the frame index of the initial video frame encoding feature, and F_all(t) is the tth initial video frame encoding feature obtained by sorting by time axis.

[0059] S2. Calculate the cosine similarity between every two temporally adjacent initial video frame coding features, and determine two temporally adjacent initial video frame coding features whose cosine similarity is not greater than a first threshold as target video frame coding features.

[0060] The cosine similarity algorithm is used to calculate the cosine similarity between the coding features of two temporally adjacent initial video frames. The first threshold can be set according to actual conditions and is not limited here. The feature sequence of the target video frame coding feature satisfies the formula: cosine_similarity(F(h), F(h+1))≤th_f; h is the frame index of the target video frame coding feature, F(h) and F(h+1) are the coding features of two temporally adjacent target video frames, th_f is the first threshold, and cosine_similarity(F(h), F(h+1)) represents the cosine similarity between F(h) and F(h+1).

[0061] S3. When the number of target video frame coding features is not less than a second threshold, construct an initial temporal coding feature that characterizes all target video frame coding features and perform feature compression to obtain a target temporal coding feature.

[0062] The second threshold can be set according to the actual situation and is not limited here. The spatiotemporal attention model is used to construct the initial temporal coding features that characterize the coding features of all target video frames, and the spatiotemporal attention model is used to perform feature compression on the initial temporal coding features to obtain the fixed-length target temporal coding features. N t Represents a fixed output length. This value can be set based on the large language model used, and there is no limit here. 257*1024 is the output size specified by the current mainstream spatiotemporal attention model.

[0063] It should be noted that, since there is a time sequence between video frames, in order to obtain temporal coding features, the spatiotemporal attention model is used for extraction in this embodiment.

[0064] S4. Map the target temporal coding features to obtain mapping coding features, and input the mapping coding features and the token sequence of the question text corresponding to the target video into the trained large language model to obtain the answer text corresponding to the question text.

[0065] Among them, the projection layer is used to map the target temporal coding features to obtain the mapping coding features F vo The question text is the text pre-set for the target video. The answer text is the answer content corresponding to the question text output by the large language model.

[0066] It should be noted that in order to enable the large language model to recognize video features, F v Projected to the embedding space of the large language model. The output after the projection layer is the mapping encoding feature F vo. Assuming the dimension of the embedding space of the large language model is D, then we should have: Dim(F vo )[-1]=D;Dim(F vo ) is to obtain F vo The dimension of [-1] is to get the last dimension. The formula represents F vo The last dimension of should be equal to the dimension of the embedding space. Use the tokenizer to divide the sentence in the question text into a string of tokens, each token can be simply understood as a character or word.

[0067] In an optional manner, the method further includes:

[0068] When the number of target video frame coding features is less than the second threshold, the cosine similarities greater than the first threshold are arranged in ascending order to obtain a target queue.

[0069] The first cosine similarity in the target queue is used as a third threshold, and among the remaining initial video frame coding features, two temporally adjacent initial video frame coding features whose cosine similarity is not greater than the third threshold are determined as target video frame coding features, and it is determined whether the number of target video frame coding features is less than the second threshold.

[0070] When the number of target video frame coding features is less than the second threshold, the cosine similarities in the target queue are traversed in sequence until the number of target video frame coding features is not less than the second threshold, and then the steps of constructing the initial temporal coding features representing all target video frame coding features and performing feature compression are executed.

[0071] The third threshold is determined according to the selected cosine similarity, and the third threshold is always greater than the first threshold.

[0072] It should be noted that, for the situation when the number of target video frame coding features is less than the second threshold, a variable threshold method is adopted in this embodiment, the goal of which is to supplement the scenarios that the first threshold cannot cover. In order to avoid information loss, when the length (number) of the acquired target video frame coding features is less than the first threshold, it is necessary to use a third threshold for screening. The first cosine similarity in the target queue is selected as the third threshold. According to the third threshold, the remaining initial video frame coding features are selected and then inserted into the feature sequence F(h) of the target video frame coding features in chronological order. The newly inserted features should satisfy: osine_similarity(F(h ′ ),F(h ′ +1))≤th_c;h ′ is the frame index of the updated target video frame encoding feature, F(h ′ ) and F(h ′+1) is the updated encoding feature of the two temporally adjacent target video frames, th_c is the third threshold, cosine_similarity(F(h ′ ),F(h ′ +1)) indicates F(h ′ ) and F(h ′ +1) between the cosine similarity.

[0073] In an optional manner, the training process of the large language model is:

[0074] Obtain the mapping coding features and the token sequence of the training question text of each training video, and input the mapping coding features and the token sequence of the training question text of any training video into the large language model to obtain the training answer text corresponding to the training question text of the training video, and obtain the loss value of the training video based on the difference between the training answer text and the real answer text of the training video, until the loss value of each training video is obtained.

[0075] According to the loss value of each training video, the parameters of the large language model are optimized to obtain an optimized large language model, and the optimized large language model is used as the large language model and the step of inputting the mapping encoding features of any training video and the token sequence of the training question text into the large language model is returned to execute until the iterative optimization condition is met, and the optimized large language model is determined as the trained large language model.

[0076] The loss function is the cross entropy loss function, which uses the loss of the autoregressive model. Based on this loss function, the difference between the training answer text and the real answer text of the training video is calculated. The loss function is defined as: N is the length of the input sequence (at least including the real answer text), V is the size of the vocabulary; y ij It takes 0 or 1. When the i-th word in the input sequence is the j-th word in the vocabulary, its value is 1, otherwise it is 0. It is the predicted probability determined by the training answer text of the large language model, that is, the large language model predicts the probability that the i-th position is the j-th word in the vocabulary.

[0077] like Figure 4 As shown, compared to traditional video understanding methods, this embodiment first preprocesses the video to extract video frames. The extracted video frames are then fed into a visual encoder for feature extraction. Feature clustering and information compression then proceed. A projection layer is then used to align the data to the word embedding space of the large language model. During training, a cross-entropy loss function is used to implement gradient backpropagation to complete model training.

[0078] The technical solution of this embodiment can effectively extract important change information of the video, avoid information loss, and thus improve the accuracy and efficiency of the large language model in understanding the video.

[0079] Figure 5 FIG. 2 shows a schematic diagram of a structure of an embodiment of a video understanding system 200 provided by the present invention. Figure 5 As shown, the system 200 includes: a first processing module 210, a calculation module 220, a second processing module 230 and an operation module 240;

[0080] The first processing module 210 is configured to: extract frames of the target video according to a preset frame extraction interval to obtain multiple target video frames and encode them respectively to obtain multiple initial video frame encoding features;

[0081] The calculation module 220 is configured to calculate the cosine similarity between each two temporally adjacent initial video frame coding features, and determine the two temporally adjacent initial video frame coding features whose cosine similarity is not greater than a first threshold as target video frame coding features;

[0082] The second processing module 230 is configured to: when the number of target video frame coding features is not less than a second threshold, construct an initial temporal coding feature representing all target video frame coding features and perform feature compression to obtain a target temporal coding feature;

[0083] The operation module 240 is used to: map the target temporal coding features to obtain mapping coding features, and input the mapping coding features and the token sequence of the question text corresponding to the target video into the trained large language model to obtain the answer text corresponding to the question text.

[0084] In an optional manner, the method further includes: a judgment module; the judgment module is configured to:

[0085] When the number of target video frame coding features is less than the second threshold, the cosine similarities greater than the first threshold are arranged in ascending order to obtain a target queue;

[0086] Using the first cosine similarity in the target queue as a third threshold, and determining, among the remaining initial video frame coding features, two temporally adjacent initial video frame coding features whose cosine similarity is not greater than the third threshold as target video frame coding features, and determining whether the number of target video frame coding features is less than the second threshold;

[0087] When the number of target video frame coding features is less than the second threshold, the cosine similarities in the target queue are traversed in sequence until the number of target video frame coding features is not less than the second threshold, and then the steps of constructing the initial temporal coding features representing all target video frame coding features and performing feature compression are executed.

[0088] In an optional manner, the step of constructing initial temporal coding features representing coding features of all target video frames and performing feature compression to obtain target temporal coding features in the second processing module 230 includes:

[0089] The spatiotemporal attention model is used to construct an initial temporal coding feature that characterizes the coding features of all target video frames, and the spatiotemporal attention model is used to perform feature compression on the initial temporal coding feature to obtain the target temporal coding feature of a fixed length.

[0090] In an optional manner, the step of mapping the target temporal coding feature to obtain the mapped coding feature in the running module 240 includes:

[0091] The target temporal coding feature is mapped using a projection layer to obtain the mapped coding feature.

[0092] In an optional manner, the training process of the large language model is:

[0093] Obtaining the mapping coding features and the token sequence of the training question text of each training video, and inputting the mapping coding features and the token sequence of the training question text of any training video into the large language model to obtain the training answer text corresponding to the training question text of the training video, and obtaining the loss value of the training video based on the difference between the training answer text and the true answer text of the training video, until the loss value of each training video is obtained;

[0094] According to the loss value of each training video, the parameters of the large language model are optimized to obtain an optimized large language model, and the optimized large language model is used as the large language model and the step of inputting the mapping encoding features of any training video and the token sequence of the training question text into the large language model is returned to execute until the iterative optimization condition is met, and the optimized large language model is determined as the trained large language model.

[0095] The technical solution of this embodiment can effectively extract important change information of the video, avoid information loss, and thus improve the accuracy and efficiency of the large language model in understanding the video.

[0096] The above parameters and steps for each module to implement corresponding functions in the video understanding system 200 of this embodiment can refer to the parameters and steps in the embodiment of the video understanding method above, and will not be repeated here.

[0097] like Figure 6 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above-mentioned video understanding methods. Specifically:

[0098] The electronic device 300 may vary significantly due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, which is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement any of the video understanding methods provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0099] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-mentioned video understanding methods.

[0100] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0101] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the above-described video understanding methods.

[0102] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0103] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0104] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0105] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A video understanding method, characterized in that: include: Extract frames from the target video according to a preset frame extraction interval to obtain multiple target video frames and encode them separately to obtain multiple initial video frame coding features; wherein the feature sequence of the multiple initial video frame coding features is F_all(t), t≤N, N represents the number of initial video frame coding features, t is the frame index of the initial video frame coding features, and F_all(t) is the t-th initial video frame coding feature obtained by sorting according to the time axis; Calculate the cosine similarity between every two temporally adjacent initial video frame coding features, and determine the two temporally adjacent initial video frame coding features whose cosine similarity is not greater than a first threshold as the target video frame coding features; wherein the feature sequence of the target video frame coding features satisfies the formula: cosine_similarity(F(h),F(h+1))≤th_f; h is the frame index of the target video frame coding feature, F(h) and F(h+1) are two temporally adjacent target video frame coding features, th_f is the first threshold, and cosine_similarity(F(h),F(h+1)) represents the cosine similarity between F(h) and F(h+1); When the number of target video frame coding features is not less than a second threshold, constructing an initial temporal coding feature representing all target video frame coding features and performing feature compression to obtain a target temporal coding feature; wherein, using a spatiotemporal attention model, constructing an initial temporal coding feature representing all target video frame coding features, and using the spatiotemporal attention model, performing feature compression on the initial temporal coding feature to obtain the target temporal coding feature of a fixed length N t Represents a fixed output length, 257*1024 is the output size specified by the spatiotemporal attention model; The target temporal coding feature is mapped to obtain a mapping coding feature, and the mapping coding feature and the token sequence of the question text corresponding to the target video are input into the trained large language model to obtain the answer text corresponding to the question text; wherein, the target temporal coding feature is mapped to obtain the mapping coding feature by using the projection layer; the target temporal coding feature F is mapped to obtain the mapping coding feature ... v Projected to the embedding space of the large language model, the output after the projection layer is the mapping encoding feature F vo ; Assume that the dimension of the embedding space of the large language model is D, then we should have: Dim(F vo )[-1]=D;Dim(F vo ) is to obtain F vo The dimension of [-1] is to get the last dimension; the formula represents F vo The last dimension of should be equal to the dimension of the embedding space; use the tokenizer to divide the sentence in the question text into a string of tokens, each token represents a word or phrase; The training process of the large language model is as follows: Obtaining the mapping coding features and the token sequence of the training question text of each training video, and inputting the mapping coding features and the token sequence of the training question text of any training video into the large language model to obtain the training answer text corresponding to the training question text of the training video, and obtaining the loss value of the training video based on the difference between the training answer text and the true answer text of the training video, until the loss value of each training video is obtained; According to the loss value of each training video, the parameters of the large language model are optimized to obtain an optimized large language model, and the optimized large language model is used as the large language model and the step of inputting the mapping coding features of any training video and the token sequence of the training question text into the large language model is returned until the iterative optimization condition is met, and the optimized large language model is determined as the trained large language model; wherein the loss function of the large language model is defined as: N is the length of the input sequence containing at least the true answer text, V is the size of the vocabulary; y ij It takes 0 or 1. When the i-th word in the input sequence is the j-th word in the vocabulary, its value is 1, otherwise it is 0. It is the predicted probability determined by the training answer text of the large language model, that is, the large language model predicts the probability that the i-th position is the j-th word in the vocabulary.

2. The video understanding method according to claim 1, characterized in that Also includes: When the number of target video frame coding features is less than the second threshold, the cosine similarities greater than the first threshold are arranged in ascending order to obtain a target queue; Using the first cosine similarity in the target queue as a third threshold, and determining, among the remaining initial video frame coding features, two temporally adjacent initial video frame coding features whose cosine similarity is not greater than the third threshold as target video frame coding features, and determining whether the number of target video frame coding features is less than the second threshold; When the number of target video frame coding features is less than the second threshold, the cosine similarities in the target queue are traversed in sequence until the number of target video frame coding features is not less than the second threshold, and then the steps of constructing the initial temporal coding features representing all target video frame coding features and performing feature compression are executed.

3. A video understanding system, characterized in that: include: a first processing module, a calculation module, a second processing module and an operation module; The first processing module is configured to extract frames from a target video according to a preset frame extraction interval to obtain a plurality of target video frames and encode them respectively to obtain a plurality of initial video frame coding features; wherein a feature sequence of the plurality of initial video frame coding features is F_all(t), t≤N, N represents the number of initial video frame coding features, t is the frame index of the initial video frame coding features, and F_all(t) is the t-th initial video frame coding feature obtained by sorting according to the time axis; The calculation module is used to calculate the cosine similarity between every two temporally adjacent initial video frame coding features, and determine the two temporally adjacent initial video frame coding features whose cosine similarity is not greater than a first threshold as the target video frame coding features; wherein the feature sequence of the target video frame coding features satisfies the formula: cosine_similarity(F(h), F(h+1))≤th_f; h is the frame index of the target video frame coding feature, F(h) and F(h+1) are two temporally adjacent target video frame coding features, th_f is the first threshold, and cosine_similarity(F(h), F(h+1)) represents the cosine similarity between F(h) and F(h+1); The second processing module is used to: when the number of target video frame coding features is not less than a second threshold, construct an initial temporal coding feature that characterizes all target video frame coding features and perform feature compression to obtain a target temporal coding feature; wherein, the spatiotemporal attention model is used to construct the initial temporal coding feature that characterizes all target video frame coding features, and the spatiotemporal attention model is used to perform feature compression on the initial temporal coding feature to obtain the target temporal coding feature of fixed length N t Represents a fixed output length, 257*1024 is the output size specified by the spatiotemporal attention model; The operation module is used to: map the target temporal coding features to obtain mapping coding features, and input the mapping coding features and the token sequence of the question text corresponding to the target video into the trained large language model to obtain the answer text corresponding to the question text; The target temporal coding feature is mapped using the projection layer to obtain the mapped coding feature; the target temporal coding feature F v Projected to the embedding space of the large language model, the output after the projection layer is the mapping encoding feature F vo ; Assume that the dimension of the embedding space of the large language model is D, then we should have: Dim(F vo )[-1]=D;Dim(F vo ) is to obtain F vo The dimension of [-1] is to get the last dimension; the formula represents F vo The last dimension of should be equal to the dimension of the embedding space; use the tokenizer to divide the sentence in the question text into a string of tokens, each token represents a word or phrase; The training process of the large language model is as follows: Obtaining the mapping coding features and the token sequence of the training question text of each training video, and inputting the mapping coding features and the token sequence of the training question text of any training video into the large language model to obtain the training answer text corresponding to the training question text of the training video, and obtaining the loss value of the training video based on the difference between the training answer text and the true answer text of the training video, until the loss value of each training video is obtained; According to the loss value of each training video, the parameters of the large language model are optimized to obtain an optimized large language model, and the optimized large language model is used as the large language model and the step of inputting the mapping coding features of any training video and the token sequence of the training question text into the large language model is returned until the iterative optimization condition is met, and the optimized large language model is determined as the trained large language model; wherein the loss function of the large language model is defined as: N is the length of the input sequence containing at least the true answer text, V is the size of the vocabulary; y ij It takes 0 or 1. When the i-th word in the input sequence is the j-th word in the vocabulary, its value is 1, otherwise it is 0. It is the predicted probability determined by the training answer text of the large language model, that is, the large language model predicts the probability that the i-th position is the j-th word in the vocabulary.

4. The video understanding system according to claim 3, characterized in that Also includes: Judgment module; the judgment module is used to: When the number of target video frame coding features is less than the second threshold, the cosine similarities greater than the first threshold are arranged in ascending order to obtain a target queue; Using the first cosine similarity in the target queue as a third threshold, and determining, among the remaining initial video frame coding features, two temporally adjacent initial video frame coding features whose cosine similarity is not greater than the third threshold as target video frame coding features, and determining whether the number of target video frame coding features is less than the second threshold; When the number of target video frame coding features is less than the second threshold, the cosine similarities in the target queue are traversed in sequence until the number of target video frame coding features is not less than the second threshold, and then the second processing module is called.

5. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the video understanding method according to claim 1 or 2.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor so that the computer-readable storage medium implements the video understanding method according to claim 1 or 2.

Images