Video analysis result output method, device, equipment and medium

Abstract

The present application discloses a method, apparatus, device, and medium for outputting video analysis results, which are used to achieve more efficient video analysis. The method provided in the present application includes: obtaining a video to be analyzed and a text prompt word input by a user; based on the video, obtaining a video frame sequence, wherein the video frame sequence includes at least one frame of image; inputting the video frame sequence into a visual encoder for feature extraction, and outputting a feature matrix for each frame of image; based on the feature matrix of each frame of image, determining at least one similarity group, each similarity group includes a feature matrix of at least one image, and the similarity between the feature matrices of images in the same similarity group is greater than a first preset threshold; for each similarity group, sampling the feature matrices of the images in the similarity group, and determining the sampling results corresponding to the similarity group; using the sampling results and text prompt words corresponding to each similarity group, outputting the video analysis results through a large language model.

Description

Technical Field

[0001] The present application relates to the field of visual analysis technology, and in particular to a method, device, equipment and medium for outputting video analysis results. Background Art

[0002] With the development of basic visual models, the focus of computer vision research has begun to shift to video understanding. The ultimate goal of computer vision is to achieve artificial intelligence similar to human vision. The transition from static vision (images) to dynamic vision (videos) is a critical and challenging step. Summary of the Invention

[0003] The embodiments of the present application provide a method, apparatus, device, and medium for outputting video analysis results, which are used to achieve more efficient video analysis based on text prompts input by users and output accurate video analysis results to users.

[0004] The present invention provides a method for outputting video analysis results, including:

[0005] Obtain the video to be analyzed and the text prompt words entered by the user;

[0006] Based on the video, a video frame sequence is obtained, wherein the video frame sequence includes at least one frame of image;

[0007] Input the video frame sequence into a preset visual encoder, perform feature extraction through the visual encoder, and output a feature matrix for each frame image;

[0008] Determining at least one similarity group based on the feature matrix of each frame of image, wherein each similarity group includes a feature matrix of at least one image, and similarities between the feature matrices of images in the same similarity group are greater than a first preset threshold;

[0009] For each similarity group, sampling the feature matrix of the images in the similarity group to determine a sampling result corresponding to the similarity group;

[0010] The sampling results corresponding to each similarity group and the text prompt words are used to output the video analysis results through a preset large language model.

[0011] The video analysis result output method provided in the embodiment of the present application obtains a video to be analyzed and a text prompt word input by a user, obtains a video frame sequence based on the video, and inputs the video frame sequence into a preset visual encoder, and performs feature extraction through the visual encoder to obtain a feature matrix of each frame image; further, in order to improve the efficiency of visual analysis, at least one similarity group is determined based on the feature matrix of each frame image, wherein each similarity group includes a feature matrix of at least one image, and the similarity between the feature matrices of images in the same similarity group is greater than a first preset threshold; thus, for each similarity group, the feature matrices of the images in the similarity group can be sampled to determine the sampling results corresponding to the similarity group; using the sampling results corresponding to each similarity group and the text prompt word, the video analysis results are output through a preset large language model. Therefore, not only can accurate video analysis results be output to the user, but also because the entire process has less computational complexity, the computational efficiency is further improved, thereby achieving more efficient video analysis and video result output.

[0012] In some embodiments, the method further comprises:

[0013] The feature matrix of each frame of image output by the visual encoder is used as a first feature matrix, inputted into a preset visual adapter, and feature extraction is performed through the visual adapter, and a second feature matrix of each frame of image is outputted;

[0014] Determining at least one similarity group based on the feature matrix of each frame of image includes:

[0015] The second feature matrix of each frame image is input into a preset similarity group module for determining at least one similarity group, and at least one similarity group is obtained through the similarity group module.

[0016] In some embodiments, for each similarity group, sampling a feature matrix of images in the similarity group and determining a sampling result corresponding to the similarity group includes:

[0017] For each similarity group, the feature matrix of an image is selected from the feature matrices of the images in the similarity group in the following manner as the sampling result corresponding to the similarity group:

[0018] Select the feature matrix of the video frame that is closest in time to the current frame;

[0019] And / or, selecting the feature matrix of the video frame having the lowest similarity to the feature matrix of the current frame.

[0020] In some embodiments, the sampling results corresponding to each similarity group and the text prompt word are used to output a video analysis result through a preset large language model, including:

[0021] Generating a current event feature group using the sampling results corresponding to each similarity group; the current event feature group includes a feature matrix of at least one image, and the similarity between the feature matrices of the images in the current event feature group is greater than a second preset threshold;

[0022] Inputting the current event feature group and the text prompt word into a preset large language model;

[0023] The video analysis result is obtained through the large language model.

[0024] In some embodiments, the visual encoder extracts features from each frame of image through at least one multi-head self-attention layer;

[0025] Feature extraction is performed using the visual encoder, including:

[0026] For each frame of image, after each layer of multi-head self-attention layer outputs the feature extraction result, the feature extraction result output by the multi-head self-attention layer of this layer is subjected to sparse processing;

[0027] Alternatively, for each frame of image, after the preset multi-head self-attention layer outputs the feature extraction result, sparse processing is performed on the feature extraction result output by the multi-head self-attention layer of this layer.

[0028] In some embodiments, the method further comprises:

[0029] For any of the similarity groups, if the correlation between the similarity group and the current event is less than a second preset threshold, and the time interval between the generation time of the similarity group and the current event is greater than a third preset threshold, the similarity group is cleared.

[0030] In some embodiments, the large language model includes a low-rank adaptation LoRA module of the large language model.

[0031] An embodiment of the present application provides a video analysis result output device, comprising:

[0032] The input module is used to obtain the video to be analyzed and the text prompt words entered by the user;

[0033] A decoding module, configured to obtain a video frame sequence based on the video, wherein the video frame sequence includes at least one frame of image;

[0034] A visual encoder is used to extract features from the video frame sequence and output a feature matrix for each frame image;

[0035] a similarity group module, configured to determine at least one similarity group based on the feature matrix of each frame of image, wherein each similarity group includes a feature matrix of at least one image, and similarities between the feature matrices of images in the same similarity group are greater than a first preset threshold;

[0036] a sampling module, configured to sample, for each similarity group, a feature matrix of images in the similarity group and determine a sampling result corresponding to the similarity group;

[0037] The large language model is used to output a video analysis result based on the sampling result corresponding to each similarity group and the text prompt word.

[0038] In some embodiments, between the visual encoder and the similarity group module, the apparatus further comprises:

[0039] A visual adapter, configured to obtain a feature matrix of each frame of image output by the visual encoder and use it as a first feature matrix, perform feature extraction on the first feature matrix of each frame of image, and output a second feature matrix of each frame of image;

[0040] The similarity group module obtains at least one similarity group based on the second feature matrix of each frame of image.

[0041] Another embodiment of the present application provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions, and the processor is used to call the program instructions stored in the memory and execute any of the above methods according to the obtained program.

[0042] Another embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of the overall process of the video analysis result output method provided in an embodiment of the present application;

[0045] Figure 2A schematic diagram of sparsification of features for multiple video frames provided in an embodiment of the present application;

[0046] Figure 3 A schematic diagram of a specific process of a method for outputting video analysis results provided in an embodiment of the present application;

[0047] Figure 4 A schematic diagram of the video analysis process provided in an embodiment of the present application;

[0048] Figure 5 A schematic diagram of sparsification of features for the same video frame provided in an embodiment of the present application;

[0049] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 7 A schematic diagram of the structure of a video analysis result output device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] The embodiments of the present application provide a method, apparatus, device, and medium for outputting video analysis results, which are used to achieve more efficient video analysis based on text prompts input by users and output accurate video analysis results to users.

[0053] Among them, the method and the device, equipment, and medium are based on the same application concept. Since the principles of solving problems by the method and the device, equipment, and medium are similar, the implementation of the device, equipment, medium, and method can refer to each other, and the repeated parts will not be repeated.

[0054] The terms "first", "second", etc. (if any) in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] The following examples and embodiments are to be understood as illustrative examples only. Although this specification may refer to "one," "an," or "some" examples or embodiments at several points, this does not mean that each such reference relates to the same example or embodiment, nor does it mean that the feature applies only to a single example or embodiment. Individual features of different embodiments may also be combined to provide further embodiments. Furthermore, terms such as "include" and "comprising" should be understood as not limiting the described embodiments to consisting only of those features mentioned; such examples and embodiments may also include features, structures, units, modules, etc. that are not specifically mentioned.

[0056] The following describes in detail the various embodiments of the present application in conjunction with the accompanying drawings. It should be noted that the order in which the embodiments of the present application are presented only represents the order of the embodiments, and does not represent the advantages or disadvantages of the technical solutions provided by the embodiments.

[0057] See also Figure 1 , a method for outputting video analysis results provided by an embodiment of the present application includes:

[0058] S101, obtaining a video to be analyzed and a text prompt word input by a user;

[0059] The video to be analyzed may include, for example, a video stream or a video clip collected in real time;

[0060] The text prompt word input by the user, for example: "Can you describe what happened in the video?"

[0061] Get the video to be analyzed and the text prompt word entered by the user, for example:

[0062] Method 1: The real-time collected video stream can be directly connected to the video understanding large model provided in the embodiment of the present application (that is, the model used in the embodiment of the present application to implement the video analysis result output method, including the large visual model and the large language model).

[0063] Method 2: Alternatively, the target detection model can be used to process the video stream in real time. When a warning target appears, a pre-captured video clip is saved. Subsequently, the pre-captured video clip is analyzed using the video understanding model provided in the embodiments of this application to identify events in the video clip, such as vehicle events, pedestrian events, road events, and weather events, and output the video analysis results.

[0064] The target detection model has a fast processing speed and high real-time performance.

[0065] For example, the downstream task is to analyze vehicle violations in the video (crossing the line, changing lanes, reversing, driving in the wrong direction, etc.). When there is a car, the video stream is connected to the video understanding model for processing, which can save performance consumption. The target detection model (that is, the small model) can be used to detect vehicle targets in real time. When the vehicle target appears in the specified area, it is connected to the video understanding model for video analysis, or the corresponding video clip is saved and then sent to the video understanding model.

[0066] S102: obtaining a video frame sequence based on the video, wherein the video frame sequence includes at least one frame of image;

[0067] S103, inputting the video frame sequence into a preset visual encoder, performing feature extraction through the visual encoder, and outputting a feature matrix for each frame image;

[0068] The feature matrix of each frame of image includes at least one feature;

[0069] Regarding the selection of encoders in the embodiments of this application, in order to improve the matching between visual features and language features, the embodiments of this application may use a network encoder based on a transformer (a transformer is a sequence model based on an attention mechanism and a neural network) for encoding visual and language features. Among them:

[0070] The visual encoder provided in the embodiments of the present application is, for example, CLIP (Contrastive Language-Image Pre-training, a large multimodal model) or SigLIP (i.e., a visual understanding encoder, which can be used for image-to-text translation, enhances multimodal translation capabilities, and is suitable for tasks such as visual question answering and image captioning);

[0071] The language encoder provided in the embodiments of the present application is, for example, the LLAMA series of large language models (LLMs). Since large language models have excellent multimodal feature extraction capabilities, the visual and language combined feature encoder can also use the LLAMA series of large language models.

[0072] S104. Determine at least one similarity group based on the feature matrix of each frame of image, wherein each similarity group includes a feature matrix of at least one image, and similarities between feature matrices of images in the same similarity group are greater than a first preset threshold;

[0073] S105 , for each similarity group, sampling the feature matrix of the images in the similarity group to determine a sampling result corresponding to the similarity group;

[0074] S106 , using the sampling results corresponding to each similarity group and the text prompt words, and outputting a video analysis result through a preset large language model.

[0075] In some embodiments, the sampling results corresponding to each similarity group and the text prompt word are used to output a video analysis result through a preset large language model, including:

[0076] Generating a current event feature group using the sampling results corresponding to each similarity group; the current event feature group includes a feature matrix of at least one image, and the similarity between the feature matrices of the images in the current event feature group is greater than a second preset threshold;

[0077] Inputting the current event feature group and the text prompt word into a preset large language model;

[0078] The video analysis result is obtained through the large language model.

[0079] In the embodiment of the present application, the features of the video stream are screened to select the features of representative video frames, which are then fed into the large language model. This is equivalent to thinning the visual features of multiple video frames, thereby reducing the amount of calculation and improving computational efficiency.

[0080] For features of selecting representative video frames, see Figure 2 For example, first calculate the similarity between the features of each video frame. For features whose feature similarity satisfies a similarity threshold (for example, 90), put the features of these frames into a group (called a similarity group), and select the features of a frame from the group as the representative frame feature.

[0081] In some embodiments, a similarity threshold is preset, i.e., a first preset threshold is T1. In the embodiment of the present application, when the feature matrices of the obtained continuous video frames are grouped by similarity, when the similarity between the feature matrix of any video frame and the feature matrices in other groups is lower than the threshold T1, a new similarity group g is created to store the feature matrix of the video frame. In other words, the similarity between the feature matrices in the same group is greater than the threshold T1. The similarity between the feature matrices of different groups is lower than the threshold T1. Therefore, when a new video frame has a feature matrix lower than the threshold T1, a new group is created to store the feature matrix of the video frame.

[0082] like Figure 2 As shown in Figure 1, due to the low similarity with the current event, the features in Group 1 and Group 2 are not sampled for video analysis. For example, the features in Group 1 and Group 2 can be compared with the features in the current event feature group. If the correlation between the features in Group 1 and Group 2 and the features in the current event feature group is less than the correlation threshold T2, they will not be sampled.

[0083] The current event is the target event recognized by the large language model, such as a traffic event.

[0084] In some embodiments, the method further comprises:

[0085] The feature matrix of each frame of image output by the visual encoder is used as a first feature matrix (for example, represented as Fv), and is input into a preset visual adapter. The visual adapter performs feature extraction and outputs a second feature matrix (for example, represented as Fm) for each frame of image;

[0086] Determining at least one similarity group based on the feature matrix of each frame of image includes:

[0087] The second feature matrix of each frame image is input into a preset similarity group module for determining at least one similarity group, and at least one similarity group is obtained through the similarity group module.

[0088] In some embodiments, the method further comprises:

[0089] For any of the similarity groups, if the correlation between the similarity group and the current event is less than a second preset threshold, and the time interval between the generation time of the similarity group and the current event is greater than a third preset threshold, the similarity group is cleared.

[0090] It can be seen that the embodiments of the present application take into account the problem that the video understanding solution forgets past events too quickly when processing real-time streams. Through the lifecycle management of similarity groups, it is more friendly to long videos and real-time streams, and can maximize the maintenance of long video features, reduce the speed at which large models forget video content, enhance the ability to associate past events and current events in the video, and better understand the video content.

[0091] Therefore, in some embodiments, see Figure 3 , a method for outputting video analysis results provided by an embodiment of the present application includes:

[0092] S201, obtaining a video to be analyzed and a text prompt word input by a user;

[0093] S202: obtaining a video frame sequence based on the video, wherein the video frame sequence includes at least one frame of image;

[0094] S203, inputting the video frame sequence into a preset visual encoder, performing feature extraction through the visual encoder, and outputting a first feature matrix for each frame image;

[0095] S204, inputting the first feature matrix of each frame of image output by the visual encoder into a preset visual adapter, performing feature extraction through the visual adapter, and outputting a second feature matrix of each frame of image;

[0096] In order to make the visual features extracted by the visual encoder more suitable for video understanding tasks, the embodiment of the present application adds a visual adapter (MLP layer, which can be one or more layers of visual adaptation layer) after the visual encoder to learn to fine-tune the effect of the visual encoder (CLIP model), and sends the first feature matrix output by the visual encoder (for example, expressed as Fv features) to the visual adapter to obtain the second feature matrix (which can also be expressed as Fm features).

[0097] That is, by adding a visual adapter, the weights of the CLIP model are frozen and not updated, and only the weights of the visual adapter are learned and updated (that is, only the visual adapter is learned and trained). In this way, the video encoder can learn the data features used in the implementation of this application, achieve the effect of fine-tuning the CLIP model, and does not affect the CLIP model. That is, by adding a visual adapter, the output second feature matrix is more targeted for the current task. Different tasks can be set with corresponding different visual adapters to obtain visual features that are more suitable for the current task.

[0098] S205. Determine at least one similarity group based on the second feature matrix of each frame of image, wherein each similarity group includes the second feature matrix of at least one image, and the similarity between the second feature matrices of images in the same similarity group is greater than a first preset threshold;

[0099] S206 , for each similarity group, sampling the feature matrix of the images in the similarity group to determine a sampling result corresponding to the similarity group;

[0100] That is to say, in the embodiment of the present application, frame features within the similarity group are sampled, one feature is extracted from each group, and the sampling results are summarized as video features and combined with the text prompt word P1 (for example, "Can you describe what happened in the video?"), and then sent to the large language model to complete multimodal feature extraction and output the answer result for the P1 prompt word.

[0101] In some embodiments, the method of sampling frame features within a similarity group includes, for example:

[0102] Method 1: Sampling based on the principle of recent time:

[0103] For each group of video frame features, extract the features of the video frame closest to the current frame (i.e., the end frame of the time window, or the last frame of the video) in chronological order, and the time interval is less than the preset threshold T3. The extracted frame is used as the representative frame of the group;

[0104] Method 2: Principle of Maximum Difference:

[0105] Within each group, the features of the frame with the lowest similarity to the features of the current frame are extracted.

[0106] S207 : Utilize the sampling results corresponding to each similarity group and the text prompt words and output the video analysis results through a preset large language model.

[0107] It can be seen that the embodiment of the present application takes into account the limitation of the large language model on the number of tokens, and will process each frame of the input video. However, the frame features input to the large language model will filter representative frame features (i.e., the sampling results) according to the feature similarity group strategy, thereby not only supporting the visual analysis of long videos, but also avoiding the problem of incorrect video analysis caused by discarding key frames, etc., making the output video analysis results more accurate and more efficient.

[0108] In summary, the input to the large language model provided in the embodiments of this application consists of two components: video features (i.e., feature matrix) and text descriptions (i.e., text prompts entered by the user). The video features are processed by the visual encoder, and because multiple similarity groups are sampled over a large time span, including past and current video features, the visual feature similarity group module can output a summary of past and current video features.

[0109] The video is the video that needs to be identified in downstream tasks (traffic event identification, equipment operation status identification, process flow specification identification, etc.);

[0110] The text description is the text description corresponding to the event recognition set that needs to be completed in the task. For example, regarding traffic event recognition, the text description is as follows:

[0111] “Can you describe the congestion in the video?”;

[0112] “The illegal driving behavior of the vehicle shown in the video”;

[0113] “Describe when in the video the traffic accident occurred?”.

[0114] After combining video features and text descriptions into a large language model, <video>This special token represents the video feature ( <video>Similar to a placeholder, it will eventually be replaced by sampled video features);

[0115] The input of the large language model is: "Can you describe the video <video>Is there congestion in the city? ";

[0116] The output of the large language model is: "From frames 30 to 80, the vehicles were traveling at a very slow speed. Because the bus slowed down first, a long queue of vehicles formed behind it, causing congestion. From frame 90 onwards, vehicles in the lane were running smoothly."

[0117] For example, see Figure 4 When the text prompt word input by the user is: "Can you describe the congestion in the video?", the method provided in the embodiment of the present application can output the following video analysis results: "The video shows a crowded road scene. Vehicles are lined up in a long queue and moving slowly. The bus in the front stops first, and pedestrians cross the road. From the 30th to the 80th frame, the vehicles are driving at a very slow speed. Because the bus slows down and stops first, a long queue of vehicles is formed behind it, causing congestion. From the 90th frame, vehicles in the lane are running smoothly."

[0118] In some embodiments, for each similarity group, sampling a feature matrix of images in the similarity group and determining a sampling result corresponding to the similarity group includes:

[0119] For each similarity group, the feature matrix of an image (feature matrix of one or more video frames) is selected from the feature matrices of the images in the similarity group in the following manner as the sampling result corresponding to the similarity group:

[0120] Select the feature matrix of the video frame that is closest in time to the current frame;

[0121] And / or, selecting the feature matrix of the video frame having the lowest similarity to the feature matrix of the current frame.

[0122] In the visual encoder, after each video frame passes through different transformer layers (multi-head self-attention layers, i.e., self-attention networks), a visual token sparsification module is set up to optimize the visual tokens and adaptively retain representative tokens. This step is equivalent to sparsifying the visual features of a single video frame, thereby reducing the amount of computation and improving computational efficiency. The representative tokens, including multiple tokens with high similarity, are retained as representative tokens through feature fusion. These tokens are components of the visual features of the video frame.

[0123] The visual token sparsification method adopted in the embodiment of the present application does not require training and is plug-and-play. Regarding the visual token sparsification provided in the embodiment of the present application, the specific implementation is to calculate the similarity between each token through dot product similarity. The larger the dot product value, the higher the similarity. Tokens with high similarity can be fused to achieve token sparsification.

[0124] Therefore, in some embodiments, the visual encoder extracts features from each frame of image through at least one multi-head self-attention layer (transformer layer);

[0125] See also Figure 5 , feature extraction is performed through the visual encoder, including:

[0126] For each frame of image, after each layer of multi-head self-attention layer outputs the feature extraction result, the feature extraction result output by the multi-head self-attention layer of this layer is subjected to sparse processing;

[0127] Alternatively, for each frame of image, after the preset multi-head self-attention layer outputs the feature extraction result, sparse processing is performed on the feature extraction result output by the multi-head self-attention layer of this layer.

[0128] The sparse processing, such as feature fusion, is the sparse processing for a single frame image.

[0129] That is to say, the sparsification described in the embodiments of the present application can adopt different strategies. Sparsification can be performed once after the video frame passes through each transformer layer; or sparsification can be performed once after the video frame passes through multiple transformer layers. For example, sparsification operations can be performed after the 3rd, 6th, and 9th transformer layers respectively, and finally the feature matrix Fv is obtained.

[0130] For example, if a video frame has 512 features, and 100 of them are similar, the features of the video frame can be sparsified, for example, by fusing these 100 features into one feature. This reduces the computational effort of subsequent transformer layers.

[0131] It can be seen that the embodiment of the present application takes into account that the transformer layer in the video understanding solution will generate a large number of visual tokens when processing video data. Each layer of transformer processes these large number of tokens, which results in high performance consumption and slow processing speed. Therefore, the embodiment of the present application will set up visual token sparse processing between different transformer layers. The sparse strategy will reduce visual tokens, improve the processing speed of the transformer layer, and reduce performance consumption.

[0132] In some embodiments, the large language model includes a low-rank adaptation LoRA module of the large language model.

[0133] In other words, in some embodiments, to make the multimodal features extracted by the large language model more suitable for video understanding tasks, a LoRA (Low-Rank Adaptation of Large Language Models) module is added (i.e., low-rank adaptation of large language models, a technique that simulates the effect of full parameter fine-tuning by adding low-rank matrices) to fine-tune the large language model and optimize its ability to extract multimodal features of video language. To better adapt to downstream tasks, the large language model can be fine-tuned, for example, including:

[0134] In order to improve the large language model's ability to perceive the start, end, and duration of events, open source and private datasets were sorted out, and the start and end time points of the events were supplemented to the original video description content.

[0135] The first fine-tuning was performed using an open-source dataset. The open-source dataset contains a large amount of material data, numerous scenes, and relatively rough annotations. Therefore, the main focus was on improving the large model's ability to understand video events and its general video understanding ability.

[0136] Freeze the training weights of the first fine-tuning and fine-tune the second training weights. This fine-tuning can use manually labeled, accurate, and downstream task-appropriate event annotations for training fine-tuning.

[0137] The following is an introduction to the equipment or device provided in the embodiments of the present application, in which the explanations or examples of technical features that are the same as or corresponding to those described in the above method are not repeated hereafter.

[0138] An electronic device provided in an embodiment of the present application, see Figure 6 , for example:

[0139] The processor 600 is configured to read the program in the memory 620 and execute the following process:

[0140] Obtain the video to be analyzed and the text prompt words entered by the user;

[0141] Based on the video, a video frame sequence is obtained, wherein the video frame sequence includes at least one frame of image;

[0142] Input the video frame sequence into a preset visual encoder, perform feature extraction through the visual encoder, and output a feature matrix for each frame image;

[0143] Determining at least one similarity group based on the feature matrix of each frame of image, wherein each similarity group includes a feature matrix of at least one image, and similarities between the feature matrices of images in the same similarity group are greater than a first preset threshold;

[0144] For each similarity group, sampling the feature matrix of the images in the similarity group to determine a sampling result corresponding to the similarity group;

[0145] The sampling results corresponding to each similarity group and the text prompt words are used to output the video analysis results through a preset large language model.

[0146] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0147] The feature matrix of each frame of image output by the visual encoder is used as a first feature matrix, inputted into a preset visual adapter, and feature extraction is performed through the visual adapter, and a second feature matrix of each frame of image is outputted;

[0148] Determining at least one similarity group based on the feature matrix of each frame of image includes:

[0149] The second feature matrix of each frame image is input into a preset similarity group module for determining at least one similarity group, and at least one similarity group is obtained through the similarity group module.

[0150] In some embodiments, for each similarity group, sampling a feature matrix of images in the similarity group and determining a sampling result corresponding to the similarity group includes:

[0151] For each similarity group, the feature matrix of an image is selected from the feature matrices of the images in the similarity group in the following manner as the sampling result corresponding to the similarity group:

[0152] Select the feature matrix of the video frame that is closest in time to the current frame;

[0153] And / or, selecting the feature matrix of the video frame having the lowest similarity to the feature matrix of the current frame.

[0154] In some embodiments, the sampling results corresponding to each similarity group and the text prompt words are used to output video analysis results through a preset large language model, including:

[0155] Generating a current event feature group using the sampling results corresponding to each similarity group; the current event feature group includes a feature matrix of at least one image, and the similarity between the feature matrices of the images in the current event feature group is greater than a second preset threshold;

[0156] Inputting the current event feature group and the text prompt word into a preset large language model;

[0157] The video analysis result is obtained through the large language model.

[0158] In some embodiments, the visual encoder extracts features from each frame of image through at least one multi-head self-attention layer;

[0159] Feature extraction is performed using the visual encoder, including:

[0160] For each frame of image, after each layer of multi-head self-attention layer outputs the feature extraction result, the feature extraction result output by the multi-head self-attention layer of this layer is subjected to sparse processing;

[0161] Alternatively, for each frame of image, after the preset multi-head self-attention layer outputs the feature extraction result, sparse processing is performed on the feature extraction result output by the multi-head self-attention layer of this layer.

[0162] In some embodiments, the processor 600 is further configured to read a program in the memory 620 and execute the following process:

[0163] For any of the similarity groups, if the correlation between the similarity group and the current event is less than a second preset threshold, and the time interval between the generation time of the similarity group and the current event is greater than a third preset threshold, the similarity group is cleared.

[0164] The transceiver 610 is configured to receive and send data under the control of the processor 600 .

[0165] Among them, Figure 6 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking together various circuits of one or more processors represented by processor 600 and memory represented by memory 620. The bus architecture may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 610 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium, such as a wireless channel, a wired channel, an optical cable, and the like. For different user devices, the user interface 630 may also be an interface capable of connecting external or internal devices as required, and the connected devices include but are not limited to a keypad, a display, a speaker, a microphone, a joystick, and the like.

[0166] The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 can store data used by the processor 600 when performing operations.

[0167] In some embodiments, the processor 600 may be a CPU (central processing unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a CPLD (Complex Programmable Logic Device), and the processor may also adopt a multi-core architecture.

[0168] The processor calls the computer program stored in the memory to execute any of the methods provided in the embodiments of the present application according to the obtained executable instructions. The processor and the memory can also be arranged physically separately.

[0169] It should be noted here that the above-mentioned device provided in the embodiment of the present application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effect. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be described in detail here.

[0170] See also Figure 7 , a video analysis result output device provided in an embodiment of the present application (for example, the video understanding large model provided in the above embodiment of the present application) includes:

[0171] Input module 11, used to obtain the video to be analyzed and the text prompt word input by the user;

[0172] A decoding module 12 is configured to obtain a video frame sequence based on the video, wherein the video frame sequence includes at least one frame of image;

[0173] A visual encoder 13 is used to extract features from the video frame sequence and output a feature matrix for each frame image;

[0174] a similarity group module 14, configured to determine at least one similarity group based on the feature matrix of each frame of image, wherein each similarity group includes a feature matrix of at least one image, and the similarity between the feature matrices of images in the same similarity group is greater than a first preset threshold;

[0175] A sampling module 15 is configured to sample, for each similarity group, a feature matrix of images in the similarity group and determine a sampling result corresponding to the similarity group;

[0176] The large language model 16 is configured to output a video analysis result based on the sampling result corresponding to each similarity group and the text prompt word.

[0177] In some embodiments, between the visual encoder and the similarity group module, the apparatus further comprises:

[0178] A visual adapter 17 is configured to obtain a feature matrix of each frame of image output by the visual encoder and use it as a first feature matrix, perform feature extraction on the first feature matrix of each frame of image, and output a second feature matrix of each frame of image;

[0179] The similarity group module 14 obtains at least one similarity group based on the second feature matrix of each frame of image.

[0180] The video analysis result output device provided in the embodiments of this application can be considered as the video understanding large model provided in the embodiments of this application, comprising a large visual model and a large language model. The large visual model includes a decoding module 12, a visual encoder 13, a visual adapter 17, a similarity group module 14, and a sampling module 15. The large language model is also known as the large language model 16. The input module 11 can be considered as an object detection model. It can also belong to the large visual model.

[0181] In some embodiments, for each similarity group, sampling a feature matrix of images in the similarity group and determining a sampling result corresponding to the similarity group includes:

[0182] For each similarity group, the feature matrix of an image is selected from the feature matrices of the images in the similarity group in the following manner as the sampling result corresponding to the similarity group:

[0183] Select the feature matrix of the video frame that is closest in time to the current frame;

[0184] And / or, selecting the feature matrix of the video frame having the lowest similarity to the feature matrix of the current frame.

[0185] In some embodiments, the sampling results corresponding to each similarity group and the text prompt words are used to output video analysis results through a preset large language model, including:

[0186] Generating a current event feature group using the sampling results corresponding to each similarity group; the current event feature group includes a feature matrix of at least one image, and the similarity between the feature matrices of the images in the current event feature group is greater than a second preset threshold;

[0187] Inputting the current event feature group and the text prompt word into a preset large language model;

[0188] The video analysis result is obtained through the large language model.

[0189] In some embodiments, the visual encoder extracts features from each frame of image through at least one multi-head self-attention layer;

[0190] Feature extraction is performed using the visual encoder, including:

[0191] For each frame of image, after each layer of multi-head self-attention layer outputs the feature extraction result, the feature extraction result output by the multi-head self-attention layer of this layer is subjected to sparse processing;

[0192] Alternatively, for each frame of image, after the preset multi-head self-attention layer outputs the feature extraction result, sparse processing is performed on the feature extraction result output by the multi-head self-attention layer of this layer.

[0193] In some embodiments, the similarity group module 14 is further configured to:

[0194] For any of the similarity groups, if the correlation between the similarity group and the current event is less than a second preset threshold, and the time interval between the generation time of the similarity group and the current event is greater than a third preset threshold, the similarity group is cleared.

[0195] In some embodiments, the large language model includes a low-rank adaptation LoRA module of the large language model.

[0196] It should be noted that the division of units in the embodiments of the present application is schematic and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program code.

[0198] Any of the devices or apparatuses provided in the embodiments of the present application may specifically be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), etc. These may include a central processing unit (CPU), a memory, input / output devices, etc. Input devices may include a keyboard, a mouse, a touch screen, etc. Output devices may include a display device such as a liquid crystal display (LCD) or a cathode ray tube (CRT).

[0199] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides the processor with program instructions and data stored in the memory. In the embodiment of the present application, the memory may be used to store the program of any of the methods provided in the embodiment of the present application.

[0200] The processor calls the program instructions stored in the memory, and the processor is used to execute any of the methods provided in the embodiments of the present application according to the obtained program instructions.

[0201] Embodiments of the present application also provide a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the methods described in the above embodiments. The program product may utilize any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk-read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0202] The present invention provides a computer-readable storage medium for storing computer program instructions used by the apparatus provided in the above embodiments of the present invention, which includes a program for executing any of the methods provided in the above embodiments of the present invention. The computer-readable storage medium may be a non-transitory computer-readable medium.

[0203] The computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (such as ROMs, EPROMs, EEPROMs, non-volatile memories (NANDFLASH), solid-state drives (SSDs)), etc.

[0204] It should be understood that:

[0205] The access technology through which entities in the communication network transmit traffic can be any suitable current or future technology, such as WLAN (Wireless Local Access Network), WiMAX (Worldwide Interoperability for Microwave Access), LTE, LTE-A, 5G, Bluetooth, infrared, etc.; in addition, the embodiments can also apply wired technology, for example, IP-based access technology, such as a wired network or a fixed line.

[0206] Embodiments suitable for being implemented as software code or a portion thereof and run using a processor or processing functionality are independent of the software code and may be specified using any known or future developed programming language, such as a high-level programming language such as objective-C, C, C++, C#, Java, Python, Javascript, other scripting languages, etc., or a low-level programming language such as machine language or assembler.

[0207] The implementation of the embodiments is hardware independent and may be implemented using any known or future developed hardware technology or any mixture thereof, such as a microprocessor or CPU (Central Processing Unit), MOS (Metal Oxide Semiconductor), CMOS (Complementary MOS), BiMOS (Bipolar MOS), BiCMOS (Bipolar CMOS), ECL (Emitter Coupled Logic) and / or TTL (Transistor-Transistor Logic).

[0208] Embodiments may be implemented as separate devices, apparatuses, units, components or functions, or in a distributed manner, for example, one or more processors or processing functions may be used or shared in a process, or one or more processing segments or processing portions may be used and shared in a process, where one physical processor or more than one physical processor may be used to implement one or more processing portions dedicated to a particular process as described.

[0209] The apparatus may be implemented by a semiconductor chip, a chipset, or a (hardware) module including such a chip or chipset.

[0210] The embodiments may also be implemented as any combination of hardware and software, such as ASIC (Application Specific IC (Integrated Circuit)) components, FPGA (Field Programmable Gate Array) or CPLD (Complex Programmable Logic Device) components or DSP (Digital Signal Processor) components.

[0211] The embodiments may also be implemented as a computer program product including a computer usable medium having computer readable program code embodied therein, the computer readable program code being adapted to perform the processes as described in the embodiments, wherein the computer usable medium may be a non-transitory medium.

[0212] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage) containing computer-usable program code.

[0213] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0214] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0215] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0216] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.< / video> < / video> < / video>

Claims

1. A method for outputting video analysis results, characterized in that: The method comprises: Obtain the video to be analyzed and the text prompt words entered by the user; Based on the video, a video frame sequence is obtained, wherein the video frame sequence includes at least one frame of image; Input the video frame sequence into a preset visual encoder, perform feature extraction through the visual encoder, and output a feature matrix for each frame image; determining, based on the feature matrix of each frame of image, at least one similarity group by a preset similarity group module for determining at least one similarity group, wherein each similarity group includes a feature matrix of at least one image, and the similarity between the feature matrices of images in the same similarity group is greater than a first preset threshold; and the similarity between the feature matrices of images in different similarity groups is less than the first preset threshold; For each similarity group, sampling a feature matrix of images in the similarity group to determine a sampling result corresponding to the similarity group; wherein the sampling result is a feature of a video frame whose time interval from the current frame is less than a preset threshold; Determining a current event feature group using the sampling results corresponding to each of the similarity groups; wherein the current event feature group includes at least one sampling result corresponding to the similarity group, and the correlation between the sampling results corresponding to each similarity group included in the current event feature group and the current event is greater than a preset correlation threshold; The current event feature group and the text prompt words are used to output a video analysis result for the text prompt words through a preset large language model.

2. The method according to claim 1, characterized in that The method further comprises: The feature matrix of each frame of image output by the visual encoder is used as a first feature matrix, inputted into a preset visual adapter, and feature extraction is performed through the visual adapter, and a second feature matrix of each frame of image is outputted; Determining at least one similarity group based on the feature matrix of each frame of image by a preset similarity group module for determining at least one similarity group includes: The second feature matrix of each frame image is input into a preset similarity group module for determining at least one similarity group, and at least one similarity group is obtained through the similarity group module.

3. The method according to claim 1, characterized in that For each similarity group, sampling the feature matrix of the images in the similarity group and determining the sampling result corresponding to the similarity group includes: For each similarity group, the feature matrix of an image is selected from the feature matrices of the images in the similarity group in the following manner as the sampling result corresponding to the similarity group: Select the feature matrix of the video frame that is closest in time to the current frame; And / or, selecting the feature matrix of the video frame having the lowest similarity to the feature matrix of the current frame.

4. The method according to claim 1, wherein The similarity between the feature matrices of the images in the current event feature group is greater than a second preset threshold.

5. The method according to claim 1, characterized in that The visual encoder extracts features from each frame of image through at least one multi-head self-attention layer; Feature extraction is performed using the visual encoder, including: For each frame of image, after each layer of multi-head self-attention layer outputs the feature extraction result, the feature extraction result output by the multi-head self-attention layer of this layer is subjected to sparse processing; Alternatively, for each frame of image, after the preset multi-head self-attention layer outputs the feature extraction result, sparse processing is performed on the feature extraction result output by the multi-head self-attention layer of this layer.

6. The method according to claim 1, characterized in that The method further comprises: For any of the similarity groups, if the correlation between the similarity group and the current event is less than a second preset threshold, and the time interval between the generation time of the similarity group and the current event is greater than a third preset threshold, the similarity group is cleared.

7. The method according to claim 1, characterized in that The large language model includes a low-rank adaptation LoRA module of the large language model.

8. A video analysis result output device, characterized in that: The device comprises: The input module is used to obtain the video to be analyzed and the text prompt words entered by the user; A decoding module, configured to obtain a video frame sequence based on the video, wherein the video frame sequence includes at least one frame of image; A visual encoder is used to extract features from the video frame sequence and output a feature matrix for each frame image; a similarity group module, configured to determine at least one similarity group based on the feature matrix of each frame of image, wherein each similarity group includes a feature matrix of at least one image, and the similarity between the feature matrices of images in the same similarity group is greater than a first preset threshold; and the similarity between the feature matrices of images in different similarity groups is less than the first preset threshold; a sampling module configured to sample, for each similarity group, a feature matrix of images in the similarity group and determine a sampling result corresponding to the similarity group; wherein the sampling result is a feature of a video frame whose time interval from a current frame is less than a preset threshold; and to determine a current event feature group using the sampling result corresponding to each similarity group; wherein the current event feature group includes a sampling result corresponding to at least one of the similarity groups, and wherein the correlation between the sampling result corresponding to each similarity group included in the current event feature group and the current event is greater than a preset correlation threshold; The large language model is used to utilize the current event feature group and the text prompt word to output a video analysis result for the text prompt word.

9. The device according to claim 8, characterized in that Between the visual encoder and the similarity group module, the apparatus further comprises: A visual adapter, configured to obtain a feature matrix of each frame of image output by the visual encoder and use it as a first feature matrix, perform feature extraction on the first feature matrix of each frame of image, and output a second feature matrix of each frame of image; The similarity group module obtains at least one similarity group based on the second feature matrix of each frame of image.

10. An electronic device, characterized in that: include: a memory for storing program instructions; A processor, configured to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 7 according to the obtained program.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable the computer to execute the method according to any one of claims 1 to 7.

Images