A video intelligent description method, system and readable storage medium based on size model collaborative fusion

Abstract

The application belongs to the technical field of artificial intelligence video analysis, and relates to a video intelligent description method and system based on size model collaborative fusion and a readable storage medium, which comprises the following steps: extracting key frames from an input video; inputting visual features of the video into a lightweight pre-training model to obtain global feature encoding of the video and a coarse-grained summary of the video; inputting the key frames into a large model to generate fine-grained behavior and interaction detail encoding of the video; setting a prompt word for size model collaborative fusion; inputting the prompt word for size model collaborative fusion, the global feature encoding and the fine-grained behavior and interaction detail encoding into the large model to generate video natural language description with macroscopic integrity and microscopic accuracy. The application has high precision and good integrity in generating multi-granularity semantic description content, and ensures the overall summary of the video content; effectively solves the trade-off contradiction between generalization and detail fidelity of a single model, and significantly improves semantic integrity and scene adaptability.

Description

TECHNICAL FIELD

[0001] The application relates to a video intelligent description method, system and readable storage medium based on size model collaborative fusion, and belongs to the technical field of artificial intelligence video analysis. BACKGROUND

[0002] Video caption generation technology, as a core topic of artificial intelligence, is widely used in security monitoring, content retrieval and other fields. With the breakthrough of generative AI technology, large models represented by GPT-4 gradually become the mainstream solution for video description tasks due to their excellent semantic understanding ability. However, two major challenges have been found in practical applications: first, large models are prone to generating redundant details in complex scenarios (such as over-describing the color of a worker's gloves or the reflection of a toolbox), leading to semantic focus shift; second, they lack sufficient modeling of temporal logic for long videos, which may reverse the order of key actions (such as mistakenly placing "disassembling equipment" after "installing components"). Existing methods face the challenge of synergistic optimization of controllability, semantic focus, and computational efficiency. Lightweight pre-trained models are computationally efficient but lack the ability to distinguish between actions, while large models can capture fine actions such as "tightening bolts" but have the problem of over-generation, which restricts the practical application effect in industrial scenarios. SUMMARY

[0003] To solve the above problems, the application provides a video intelligent description method, system and readable storage medium based on size model collaborative fusion, which can capture details and improve semantic condensation, especially in high-risk scenarios such as security monitoring.

[0004] To achieve the above purpose, the application provides the following technical solution: a video intelligent description method based on size model collaborative fusion, comprising the following steps: extracting key frames from an input video; inputting visual features of the video into a lightweight pre-trained model to obtain global feature encoding of the video and a coarse-grained summary of the video; inputting the key frames into a large model to generate fine-grained behavior and interaction detail encoding of the video; setting a prompt word for size model collaborative fusion, inputting the prompt word for size model collaborative fusion, global feature encoding, and fine-grained behavior and interaction detail encoding into the large model to generate a video natural language description with macroscopic integrity and microscopic accuracy.

[0005] Further, a frame difference dynamic sampling algorithm is used to filter key frames from the input video and construct a key frame sequence representing the video semantics.

[0006] Further, the method for obtaining the global feature code of the video is as follows: inputting a video, performing equidistant time sampling on the video; obtaining visual features of video frames according to the sampling result, inputting the visual features into a lightweight pre-training model, wherein an input layer of the lightweight pre-training model comprises a visual feature encoder, and an output of the lightweight pre-training model comprises a global feature code of a coarse-grained summary description of a video subject, which can reflect the global feature code of the video theme and the coarse-grained summary content of the video, i.e., a simple summary of the video content.

[0007] Further, the method for generating the fine-grained behavior and interaction detail code of the video is as follows: generating prompt words of fine-grained semantic content based on the key frames; inputting the prompt words of fine-grained semantic content and the key frames into the large model to generate the fine-grained behavior and interaction detail code of the video.

[0008] Further, the prompt words of fine-grained semantic content comprise behavior analysis instructions for constraining the large model, scene element constraints, and syntax specification templates, and the prompt words of fine-grained semantic content are used to guide the large model to generate fine-grained content.

[0009] Further, the method for generating the natural language description of the video is as follows: under the constraint of the prompt words of the large-small model collaborative fusion, generating a multi-granularity natural language description of the video, matching the coarse-grained features generated by the small model with the fine-grained features generated by the large model, eliminating logical contradictions between the two, and generating a multi-granularity natural language description with unified semantics.

[0010] Further, the prompt words of the large-small model collaborative fusion comprise specification and syntax templates for fusion and optimization of multi-granularity semantic content output by the large-small model understanding video content.

[0011] The application further discloses a video intelligent description system based on large-small model collaborative fusion, comprising: a key frame extraction module configured to extract key frames from an input video; a small model module configured to input visual features of the video into a lightweight pre-training model to obtain a global feature code of the video; a large model module configured to input the key frames into a large model to generate a fine-grained behavior and interaction detail code of the video; and a large-small model fusion module configured to set prompt words of large-small model collaborative fusion, input the prompt words of large-small model collaborative fusion, the global feature code, and the fine-grained behavior and interaction detail code into the large model, and generate a natural language description of the video with macroscopic integrity and microscopic accuracy.

[0012] Further, the method for generating the natural language description of the video is as follows: under the constraint of the prompt words of the large-small model collaborative fusion, generating a multi-granularity natural language description of the video, matching the coarse-grained features generated by the small model with the fine-grained features generated by the large model, eliminating logical contradictions between the two, and generating a multi-granularity natural language description with unified semantics.

[0013] The application further discloses a computer readable storage medium, and the computer readable storage medium stores a computer program.

[0014] The technical scheme of the application has at least the following technical effects or advantages:

[0015] 1. The application generates multi-granularity semantic content based on the cooperation of the large and small models, improves the precision and completeness of video semantic description, generates coarse-granularity global description through a lightweight pre-training model to ensure the overall summary of video content, and analyzes fine-granularity behaviors and scene details under the guidance of a prompt word through a large model, and through a semantic fusion engine, the multi-level description is subjected to conflict resolution and logical integration to generate a text with both summarization and detail depth.

[0016] 2. The application uses a prompt word to constrain the large model, and through pre-defined behavior analysis instructions, scene element constraints and grammar templates, the large model is accurately guided to output content conforming to industry specifications, compared with the generalization description or too miscellaneous content often occurring in a single model, the application forcibly constrains the output logic and content range through the prompt word. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the video intelligent description method based on the cooperation and fusion of the large and small models in an embodiment of the application. DETAILED DESCRIPTION

[0018] In order to make the skilled in the art better understand the technical scheme of the application, the application is described in detail through specific embodiments. However, it should be understood that the provision of the specific embodiments is only for better understanding of the application, and they should not be understood as a limitation on the application. In the description of the application, it should be understood that the terms used are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0019] The large model in the present application, i.e., the artificial intelligence large model, refers to a class of artificial intelligence models with a large number of parameters constructed by artificial neural networks. The artificial intelligence large model is a new concept that has emerged in the past decade. It is usually pre-trained on a large amount of data through self-supervised learning or semi-supervised learning, and then further optimized its performance and ability through instruction fine-tuning and human alignment. Large models have large parameter quantity, large training data, and large computing resources, and have the ability to solve general tasks, follow human instructions, and perform complex reasoning. The main categories of artificial intelligence large models include large language models, visual large models, multi-modal large models, and basic scientific large models, etc. At present, large models have been widely applied in many fields, including search engines, agents, related vertical industries, and basic sciences, etc., and have promoted the intelligent development of various industries.

[0020] The small model, i.e., the lightweight pre-training model, is a model opposite to the large model. The neural network architecture it usually uses includes shallow neural networks, lightweight convolutional neural networks (such as MobileNet) or small recurrent neural networks (such as LSTM or GRU), etc. The small model may have only a few layers and relatively fewer parameters to maintain the lightweight characteristics of the model. These models require less training time and are usually used for tasks with lower computing resource requirements, such as image recognition, text classification, etc. in mobile applications or embedded systems. The most important difference compared to large models is that there are fewer parameters, the processing task is single, the volume is smaller, and it is easier to deploy in resource-limited environments. Large models usually use deeper and wider neural networks, such as deep convolutional neural networks (such as ResNet, Inception, EfficientNet) or large transformer models (such as BERT, GPT), etc. These models have millions to tens of billions, hundreds of billions of parameters, which can capture more features and complexity.

[0021] To solve the problems in the prior art that using a lightweight pre-training model has high computational efficiency and strong stability, but can only generate a general description and cannot accurately express detailed actions in a video, such as being unable to distinguish between "tightening a bolt" and "removing a bolt"; and using a large model (such as GPT-4) has strong detail capturing capability, but in actual application, it can generate too many tedious or unnecessary details, affecting information summarization and semantic clarity, etc. The present application proposes a video content multi-granularity intelligent description method, system and readable storage medium based on large and small model collaborative fusion. On the basis of existing lightweight pre-training models and large models, first, a key frame sequence representing video semantics is constructed by filtering key frames from the input video based on an inter-frame difference dynamic sampling algorithm; then, a large and small model is established to respectively understand the video content generation mechanism, and a prompt word for large and small model collaborative fusion is set; based on the large model, semantic alignment and conflict resolution of double-path output content are realized under the constraint of the prompt word, and a video content multi-granularity natural language description with macroscopic integrity and microscopic accuracy is generated. The present application integrates the efficient inference capability of the lightweight pre-training model and the deep semantic understanding advantage of the large model through a key frame driven hierarchical semantic architecture, solves the performance imbalance between description generality and detail fidelity in the video description task of the existing single model, and improves the description accuracy and semantic conciseness of the video description task in different fields such as security monitoring, content retrieval and decision support.

[0022] Embodiment one

[0023] The present embodiment discloses a video intelligent description method based on large and small model collaborative fusion, comprising the following steps:

[0024] S1 extracts key frames from the input video.

[0025] The input video is denoted as {video1, video2, video3, …, video n}; each video filters key frames from the input video based on an inter-frame difference dynamic sampling algorithm to construct a key frame sequence representing video semantics, denoted as {video1_keyframe1, video1_keyframe2, video1_keyframe3, …, video1_keyframe n}, {video2_keyframe1, video2_keyframe2, video2_keyframe3, …, video2_keyframe n}, {video3_keyframe1, video3_keyframe2, video3_keyframe3, …, video3_keyframe n}, … {video n _keyframe1, video n _keyframe2, video n _keyframe3, …, video n _keyframen}。

[0026] The key frame is a video frame in which the video content changes significantly, and the specific method for extracting the key frame is as follows: extracting a video frame, that is, screening the video into one image after another, in all video frames, extracting a video frame at the same distance interval, converting adjacent two video frames into gray scale images, subtracting the corresponding pixel points in the two gray scale images, comparing the absolute value of the gray scale difference, judging the change between the two video frames according to the absolute value, and defining the corresponding video frame as a key frame when the change between the two video frames exceeds a threshold.

[0027] The inter-frame difference dynamic sampling algorithm is a method for obtaining the contour of a moving target by performing difference operation on adjacent two frames in a video image sequence. It can be well applied to the case where there are multiple moving targets and camera movement. When an abnormal object moves in the monitored scene, there will be a relatively obvious difference between frames. The absolute value of the brightness difference between two frames is obtained by subtracting the two frames, and it is judged whether it is greater than a threshold to analyze the motion characteristics of the video or image sequence and determine whether there is object movement in the image sequence. The difference between the image sequence frame by frame is equivalent to high-pass filtering of the image sequence in the time domain.

[0028] S2 inputs the visual features of the video into the lightweight pre-training model to obtain the global feature encoding of the video and the coarse-grained summary of the video.

[0029] In the above step, a video frame is extracted at the same distance interval between all video frames. The visual feature is a fixed-dimensional feature vector output after abstract coding of the extracted video frame. Since the lightweight pre-training model cannot directly understand the image, the visual feature of the video needs to be obtained.

[0030] The method for obtaining the global feature encoding of the video is as follows: inputting the video {video1, video2, video3, …, video n}, performing equidistant time sequence sampling on the video; obtaining the visual features of the video frames according to the sampling result, inputting the visual features into the lightweight pre-training model, the input layer of the lightweight pre-training model including a visual feature encoder, and the output of the lightweight pre-training model including the global feature encoding of the coarse-grained summary of the video subject, denoted as {coarse-grained semantic content}, which can reflect the global feature encoding of the video theme and the coarse-grained summary content of the video, that is, the simple summary of the video content.

[0031] S3 inputs the key frame into the large model to generate the fine-grained behavior and interaction detail encoding of the video.

[0032] The method for generating the fine-grained behavior and interaction detail encoding of the video is: generating prompt words of fine-grained semantic content based on key frames; inputting the prompt words of fine-grained semantic content and the key frames {video1_keyframe1, video1_keyframe2, video1_keyframe3, …, video1_keyframe n}, {video2_keyframe1, video2_keyframe2, video2_keyframe3, …, video2_keyframe n}, {video3_keyframe1, video3_keyframe2, video3_keyframe3, …, video3_keyframe n}, … {video n _keyframe1, video n _keyframe2, video n _keyframe3, …, video n _keyframe n} into a large model to generate fine-grained behavior and interaction detail encoding of the video, denoted as {fine-grained semantic content}.

[0033] The prompt words of fine-grained semantic content include behavior analysis instructions for constraining the large model, scene element constraints, and syntax specification templates, and the prompt words of fine-grained semantic content are used to guide the large model to generate fine-grained content.

[0034] The behavior analysis instructions are, for example: “These key frames are input in sequence and come from the same video. Please analyze these key frames, focusing on objects / organisms and their actions with interactive objects”; and the syntax specification templates are, for example: “Describe these details and actions in a time-ordered sentence (directly output the generated result, without adding words such as ‘in the video’ or ‘in these key frames’ at the beginning)”.

[0035] S4 sets the prompt words for the size model collaborative fusion, inputs the prompt words for the size model collaborative fusion and the global feature encoding and the fine-grained behavior and interaction detail encoding into the large model, and generates a video natural language description with macroscopic integrity and microscopic accuracy.

[0036] The method for generating the video natural language description is: under the constraint of the prompt words for the size model collaborative fusion, generating a multi-granularity natural language description of the video, matching the coarse-grained features generated by the small model with the fine-grained features generated by the large model, eliminating logical contradictions between the two, and generating a multi-granularity natural language description with unified semantics.

[0037] The prompt word of the size model collaborative fusion includes the specification and syntax template of the fusion and optimization of the multi-granularity semantic content of the size model understanding video content output. The specification of the fusion and optimization of the multi-granularity semantic content of the size model understanding video content output is, for example, "find the key actions and details in the fine-grained content that are missing in the coarse-grained content, and perform semantic fusion while preserving the coarse-grained content". The syntax template is, for example, "generate a logically consistent sentence".

[0038] To verify the feasibility of the method in the embodiment, a dataset for testing the video caption task is introduced, which includes 41250 video clips covering a wide range of human behaviors and interactions, and each video corresponds to 10 Chinese and English annotations, totaling 825000 annotations. The score uses the generated text quality score (ClipScore), which can reflect the similarity between the generated text and the visual features extracted from the picture, and can measure the semantic similarity between the text and the video content. The comparison results of the method in the embodiment and the method in the prior art are shown in Table 1.

[0039] Table 1 Comparison of the method in the embodiment and the method in the prior art

[0040]

[0041] As can be seen from Table 1, the method in the embodiment is significantly better than the method of using only a lightweight pre-training model or a large model in terms of the generated text quality score. This shows that the method in the embodiment can effectively balance the global generality and detail fidelity of video description, and significantly improve the description accuracy and semantic conciseness of the video caption task in different scenarios.

[0042] The multi-granularity natural language description of the video content generated in the embodiment has macroscopic integrity and microscopic accuracy, which improves the description accuracy and semantic conciseness of the video description task in different scenarios.

[0043] Embodiment Two

[0044] Based on the same inventive concept, the embodiment discloses a video intelligent description system based on size model collaborative fusion, which comprises:

[0045] A key frame extraction module is configured to extract key frames from an input video.

[0046] A small model module is configured to input visual features of the video into a lightweight pre-training model to obtain global feature encoding of the video.

[0047] A large model module is configured to input the key frames into a large model to generate fine-grained behavior and interaction detail encoding of the video.

[0048] The size model fusion module is configured to set a prompt word for size model collaborative fusion, input the prompt word for size model collaborative fusion, global feature encoding, and fine-grained behavior and interaction detail encoding into a large model, and generate a video natural language description with macro integrity and micro accuracy.

[0049] The method for generating a video natural language description is as follows: under the constraint of the prompt word for size model collaborative fusion, a multi-granularity natural language description of the video is generated, coarse-grained features generated by a small model are matched with fine-grained features generated by a large model, logical contradictions between the two are eliminated, and a multi-granularity natural language description with unified semantics is generated.

[0050] Embodiment three

[0051] Based on the same inventive concept, the embodiment discloses a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the video intelligent description method based on size model collaborative fusion.

[0052] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0053] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the flow Figure 1 one or more flows and / or blocks Figure 1 an apparatus with the function specified in one or more blocks.

[0054] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction apparatus, which implements the flow Figure 1 one or more flows and / or blocks Figure 1 an apparatus with the function specified in one or more blocks.

[0055] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 Figure 1 The flowchart blocks or blocks can be represented by a computer program instruction or a plurality of computer program instructions.

[0056] Finally, it should be noted that the above examples are merely intended to describe the technical solutions of the present application, rather than limiting it. Although the present application has been described in detail with reference to the above examples, those skilled in the art should understand that the specific embodiments of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and any modification or equivalent replacement without departing from the spirit and scope of the present application should be covered in the protection scope of the claims of the present application. The above content is merely a specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​

Claims

1. A video intelligent description method based on the collaborative fusion of size models, characterized in that, Includes the following steps: Extract keyframes from the input video; The visual features of the video are input into a lightweight pre-trained model to obtain the global feature encoding and coarse-grained overview of the video. The keyframes are input into a large model to generate fine-grained behavior and interaction detail encodings for the video. Set prompt words for collaborative fusion of large and small models, and input the prompt words for collaborative fusion of large and small models, global feature encoding, fine-grained behavior and interaction detail encoding into the large model to generate video natural language descriptions that have both macro-level completeness and micro-level accuracy; The method for generating the video natural language description is as follows: under the constraint of the prompt words of the collaborative fusion of the small and large models, a multi-granular natural language description of the video is generated. The coarse-grained features generated by the small model are matched with the fine-grained features produced by the large model to eliminate the logical contradiction between the two and generate a semantically unified multi-granular natural language description. The prompt words for the collaborative fusion of large and small models include specifications and grammatical templates for the fusion and optimization of multi-granular semantic content output by large and small models in understanding video content.

2. The video intelligent description method based on size model collaborative fusion as described in claim 1, characterized in that, The keyframe sequence representing the semantics of the video is constructed by selecting keyframes from the input video using the inter-frame difference dynamic sampling algorithm.

3. The video intelligent description method based on size model collaborative fusion as described in claim 1, characterized in that, The method for obtaining the global feature encoding of a video is as follows: inputting a video and performing equidistant temporal sampling on the video; obtaining the visual features of the video frames based on the sampling results; inputting the visual features into a lightweight pre-trained model, wherein the input layer of the lightweight pre-trained model includes a visual feature encoder, and the output of the lightweight pre-trained model contains a global feature encoding that provides a coarse-grained general description of the main body of the video.

4. The video intelligent description method based on size model collaborative fusion as described in claim 1, characterized in that, The method for generating fine-grained behavior and interaction detail encoding in videos is as follows: Based on the keyframes, generate prompts with fine-grained semantic content; The fine-grained semantic content prompts and keyframes are input into the large model to generate fine-grained behavior and interaction detail encodings for the video.

5. The video intelligent description method based on size model collaborative fusion as described in claim 4, characterized in that, The prompts for the fine-grained semantic content include behavioral analysis instructions that constrain the large model, scene element constraints, and grammatical specification templates. These prompts are used to guide the large model in generating fine-grained content.

6. A video intelligent description system based on the collaborative fusion of large and small models, characterized in that, include: The keyframe extraction module is used to extract keyframes from the input video. The small model module is used to input the visual features of the video into a lightweight pre-trained model to obtain the global feature encoding of the video. The large model module is used to input the keyframes into the large model to generate fine-grained behavior and interaction detail encodings for the video; The large and small model fusion module is used to set prompt words for the collaborative fusion of large and small models. The prompt words for the collaborative fusion of large and small models, along with global feature encoding, fine-grained behavior and interaction detail encoding, are input into the large model to generate a video natural language description that has both macroscopic completeness and microscopic accuracy. The method for generating the video natural language description is as follows: under the constraint of the prompt words of the collaborative fusion of the small and large models, a multi-granular natural language description of the video is generated. The coarse-grained features generated by the small model are matched with the fine-grained features produced by the large model to eliminate the logical contradiction between the two and generate a semantically unified multi-granular natural language description. The prompt words for the collaborative fusion of large and small models include specifications and grammatical templates for the fusion and optimization of multi-granular semantic content output by large and small models in understanding video content.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor to implement the video intelligent description method based on size model collaborative fusion as described in any one of claims 1-5.

Images