Information generation method, device, computer program product, and computer-readable storage medium

Abstract

The application provides an information generation method, device, computer program product and computer readable storage medium; the method comprises: obtaining a to-be-processed video, and extracting a target frame sequence from the to-be-processed video; based on the target frame sequence, determining a multi-scale feature of the to-be-processed video; wherein the multi-scale feature at least comprises a time sequence motion feature reflecting dynamic changes of the to-be-processed video, a global feature reflecting overall content of the to-be-processed video and a local feature reflecting detailed content of the to-be-processed video; and integrating the time sequence motion feature, the global feature and the local feature to obtain target description information of the to-be-processed video.

Description

Technical Field

[0001] This application relates to video text generation technology, and more particularly to an information generation method, device, computer program product, and computer-readable storage medium. Background Technology

[0002] Video description technology aims to automatically convert video content into natural language descriptions, and is widely used in scenarios such as content understanding, assisted viewing, and video retrieval. However, existing technologies have common limitations when processing video content: on the one hand, many models tend to process video frames globally or in fixed blocks, making it difficult to fully capture and describe the fine semantic information of local areas in the image (such as facial expressions, small text, and the texture of specific objects), resulting in insufficient detail and accuracy in the generated descriptions; on the other hand, video data itself contains a large amount of redundant information. Directly performing frame-by-frame deep analysis on the original high frame rate, long-duration video stream will generate huge computational and storage overheads, making it difficult to balance description quality and processing efficiency, thus limiting the application of the technology in real-time or resource-constrained scenarios. Therefore, there is an urgent need for a novel video description method that can collaboratively understand the global scene, local details, and dynamic temporal information of a video, and is computationally efficient. Summary of the Invention

[0003] This application provides an information generation method, device, computer program product, and computer-readable storage medium, which significantly improves processing efficiency while ensuring that timing information is not lost, making real-time or near-real-time description of video possible.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an information generation method, the method comprising: Acquire the video to be processed, and extract the target frame sequence from the video to be processed; Based on the target frame sequence, multi-scale features of the video to be processed are determined; wherein, the multi-scale features include at least temporal motion features reflecting the dynamic changes of the video to be processed, global features reflecting the overall content of the video to be processed, and local features reflecting the detailed content of the video to be processed. The temporal motion features, global features, and local features are integrated to obtain the target description information of the video to be processed.

[0005] In the above scheme, extracting the target frame sequence from the video to be processed includes: Determine the first motion change information between adjacent frames in the video to be processed; A similarity algorithm is used to determine the content similarity between consecutive frames in the video to be processed; Based on the first action change information and the content similarity, the target frame sequence is determined from the video to be processed.

[0006] In the above scheme, determining the multi-scale features of the video to be processed based on the target frame sequence includes: The temporal motion features are determined based on the second action change information between adjacent frames in the target frame sequence; For each target frame in the target frame sequence, the target frame is downsampled and its features are extracted to obtain the global features; For each target frame, a semantic segmentation model is used to segment the target frame to obtain multiple first regions, and the multiple first regions are processed to obtain the local features.

[0007] In the above scheme, determining the temporal motion features based on the second action change information between adjacent frames in the target frame sequence includes: Determine the second action change information between adjacent frames in the target frame sequence; The temporal motion features are obtained by processing each second action change information using a multimodal large language model and a first cue word.

[0008] In the above scheme, the step of downsampling and feature extraction of the target frame to obtain the global features includes: For each target frame, the target frame is downsampled to obtain a target resolution image of the target frame; wherein, the target resolution image is an image with a resolution less than a resolution threshold; A multimodal large language model and a second cue word are used to encode each target resolution image to obtain multiple global features.

[0009] In the above scheme, the step of processing the plurality of first regions to obtain the local features includes: The plurality of first regions are identified and filtered to obtain a second region with a non-background region; For each of the second regions, the second region is segmented based on its size to obtain at least one initial image block; wherein, the second regions with a size greater than a first threshold are segmented using a first block density, and the second regions with a size less than or equal to the first threshold are segmented using a second block density; the second block density is greater than the first block density; The local features are determined based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches.

[0010] In the above scheme, determining the local features based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches includes: For each initial image patch, the hierarchical sampling strategy is used to perform multi-scale sampling on the initial image patch to obtain an image patch set containing at least three different scales; wherein, the first scale is used to provide the spatial location and range information of the second region in the target frame, the second scale is used to identify the action state of the object in the second region or its interaction with other objects in the scene, and the third scale is used to obtain detailed information in the second region. For the image patch set at the first scale and the image patch set at the second scale, a first type of multimodal large language model is used for feature extraction to obtain a first feature, and for the image patch set at the third scale, a second type of multimodal large language model is used for feature extraction to obtain a second feature; wherein, the first type of multimodal large language model is a lightweight language model, and the second type of multimodal large language model is a high-precision language model; The local features are determined based on the first feature and the second feature.

[0011] This application provides an information generation apparatus, the apparatus comprising: An acquisition unit is used to acquire a video to be processed and extract a target frame sequence from the video to be processed. The processing unit is configured to determine multi-scale features of the video to be processed based on the target frame sequence; wherein the multi-scale features include at least temporal motion features reflecting the dynamic changes of the video to be processed, global features reflecting the overall content of the video to be processed, and local features reflecting the detailed content of the video to be processed. The generation unit is used to integrate the temporal motion features, the global features, and the local features to obtain the target description information of the video to be processed.

[0012] This application embodiment provides an information generation device, the information generation device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method provided in the embodiments of this application.

[0013] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the method provided in this application.

[0014] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the method provided in this application when executed by a processor.

[0015] The embodiments of this application have the following beneficial effects: First, by extracting the key frame sequence first and then performing feature extraction strategies of different granularities, the huge computational overhead caused by performing full-resolution depth processing on all frames of the original high-definition video is avoided. Second, by specifically extracting and modeling local features, the capture and description of key details in the high-definition video are ensured, significantly improving the richness and accuracy of the generated description. Thus, while ensuring that the temporal information is not lost, the processing efficiency is greatly improved, making real-time or near-real-time description of the video possible. Attached Figure Description

[0016] Figure 1 This is a first flowchart illustrating the information generation method provided in this application embodiment; Figure 2 This is a second flowchart illustrating the information generation method provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the determination of target description information in the information generation method provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of the information generation device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the information generation device provided in the embodiments of this application.

[0017] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] It's important to note that video captioning is a multimedia technology designed to provide a concise textual description or summary of video content. It's typically used to help viewers understand the main points of the video, especially when there is no sound or sound is unavailable. This technology has wide applications in many fields, including filmmaking, virtual reality, and assistive devices for people with disabilities. In e-commerce, video captioning can be used to create product tutorials and guides, helping users better understand how to use the products and improving user satisfaction. It can also convey brand philosophy, corporate culture, and core values ​​through subtitles and video captions, enhancing brand awareness and loyalty. Furthermore, video captioning is an essential step in any written video task.

[0020] The core of video captioning technology lies in automatically converting video content into natural language descriptions. While significant progress has been made in video text description technology, particularly for high-definition (HD) videos such as 1080p, some shortcomings remain, especially in the text description of ultra-HD videos. HD videos often cause problems similar to those encountered with regular-resolution videos: 1. Loss of local information: Reason: Some models, when generating text descriptions for high-resolution videos, may not be able to fully capture certain details in the image, such as subtle textures and details like skin texture and clothing wrinkles; Impact: This lack of detail results in incomplete video descriptions. 2. Increased resolution, especially for 4K and 8K videos, causes a sharp increase in video memory, potentially leading to memory overflow errors.

[0021] Based on this, embodiments of this application provide an information generation method, see [link to relevant documentation]. Figure 1 As shown, when applied to an information generation device, it may specifically include the following steps: Step 101: Obtain the video to be processed and extract the target frame sequence from the video to be processed.

[0022] In this embodiment, the video to be processed refers to the original video for which text descriptions need to be generated. It can be a low-resolution video (such as 480P, 720P, etc.) or a high-resolution video (such as 1080P, 4K, 8K, etc.). The target frame sequence refers to the key frames selected from the original video (i.e., the video to be processed), and is a set of representative frames that can represent the main content, event turning points, and visual changes of the video to be processed. The original video file (i.e., the video to be processed) can be loaded from a storage device or network stream first, and then a video summarization algorithm based on optical flow and visual similarity (such as VideoSummarization via Optical Flow Clustering or Deep Video Summarization) can be used to process the video to be processed, thereby extracting multiple key frames to form the target frame sequence. In this way, by selecting the target frame sequence from the video to be processed for subsequent processing, instead of processing all the original videos as in related technologies, redundant frames can be significantly reduced (by more than 90%) while preserving the key event points in the time sequence and ensuring that the dynamic information of the video is not lost. This significantly reduces the amount of data and computational complexity of subsequent processing.

[0023] Step 102: Based on the target frame sequence, determine the multi-scale features of the video to be processed.

[0024] Among them, multi-scale features include at least temporal motion features that reflect the dynamic changes of the video to be processed, global features that reflect the overall content of the video to be processed, and local features that reflect the detailed content of the video to be processed.

[0025] In this embodiment, the multi-scale features include three types: temporal motion features, global features, and local features. Temporal motion features describe the dynamic behavior and temporal evolution of the video to be processed (e.g., human actions, object movement); global features reflect the overall scene semantics of the video to be processed (e.g., environment, main object, spatial relationships); and local features capture key details in the video to be processed (e.g., facial expressions, text content, object attributes). Temporal motion features, global features, and local features can be extracted in parallel by three feature extraction modules. Specifically, a motion information modeling module is used to extract temporal motion features, a global information modeling module is used to extract global features, and a local information modeling module is used to extract local features, thus obtaining multi-scale features. In this way, while extracting global features, local features are also extracted, which can solve the problem in related technologies where, when generating text descriptions for high-resolution videos, some details in the image, such as subtle textures and details like skin texture and clothing wrinkles, cannot be fully captured, resulting in incomplete video descriptions. This can further improve the accuracy and comprehensiveness of the generated descriptive information.

[0026] Step 103: Integrate temporal motion features, global features, and local features to obtain target description information for the video to be processed.

[0027] In this embodiment of the application, integration refers to fusing feature information (global features, local features, and temporal motion features) from different dimensions into a coherent and complete text description; the target description information can be the text description of the final generated video to be processed, which has temporal sequence, logical structure and rich details. The text fusion module can integrate global features, local features, and temporal motion features. Specifically, the Large Language Model (LLM) within the text fusion module acts as a text editor, receiving global features, local features, and temporal motion features. It then performs deduplication and merging (eliminating duplicate descriptions and merging similar content), temporal sorting (arranging action descriptions according to the order of events), causal inference (supplementing implicit logical relationships (e.g., "He fell because the ground was slippery")), and language polishing (generating fluent, natural, and grammatically correct Chinese descriptions). Based on prompts, it generates target description information for the video to be processed. This results in: 1. Generating semantically complete, logically clear, and detailed video descriptions, taking into account both macro-level scenes and micro-level details; 2. Supporting multi-dimensional information fusion, avoiding description bias from a single perspective; and 3. Improving the naturalness and usability of the generated text, making it suitable for various downstream applications (such as video retrieval, content moderation, and assisted viewing).

[0028] In one feasible implementation, the prompt could be: "You are a professional video analyst. Please generate a high-quality subtitle by combining the following three parts of information: [temporal motion features], [global features], and [local features]. Please ensure that the description includes temporal sequence, spatial relationships, and key details, and that the language is concise and accurate." The output target description could be: "On a city street after the rain, a man in a blue coat is quickly crossing a zebra crossing. He has an anxious expression on his face and is clutching a document tightly in his hand. The background shows the Bank A sign and traffic lights, with the traffic light showing green. Suddenly, he slips and stumbles, but quickly regains his balance and continues walking." The large language model could specifically refer to GPT-4, Claude 3.5, or Tongyi Qianwen, etc.

[0029] The information generation method provided in this application firstly avoids the huge computational overhead caused by performing full-resolution depth processing on all frames of the original high-definition video by first extracting the key frame sequence and then performing feature extraction strategies at different granularities. Secondly, by specifically extracting and modeling local features, it ensures the capture and description of key details in the high-definition video, significantly improving the richness and accuracy of the generated description. Thus, while ensuring that the temporal information is not lost, it greatly improves the processing efficiency, making real-time or near-real-time description of the video possible.

[0030] Based on the foregoing embodiments, this application provides yet another information generation method, see [link to previous document]. Figure 2 As shown, when applied to an information generation device, it may specifically include the following steps: Step 201: Obtain the video to be processed.

[0031] Step 202: Determine the first motion change information between adjacent frames in the video to be processed.

[0032] In this embodiment, adjacent frames refer to two frames played consecutively on the video timeline; the first motion change information refers to quantitative data (such as statistical values ​​of pixel displacement fields, motion energy maps, etc.) that characterize the global or local motion intensity between adjacent frames, calculated by optical flow methods (including sparse optical flow and dense optical flow), i.e., inter-frame differences; specifically, optical flow methods can be used to process the video to be processed, calculate the pixel displacement field between each pair of adjacent frames, and extract indicators such as average motion amplitude and motion region proportion based on the displacement field as the first motion change information to identify the period of intense motion in the video to be processed; in a feasible implementation, the optical flow method can specifically be Farneback, Total Variation-L1 Norm (TV-L1), etc.

[0033] It should be noted that by determining the first motion change information between each pair of adjacent frames, dynamic information can be accurately captured. That is, it is possible to objectively and quantitatively identify the moments when significant actions or changes occur in the video (such as people running or objects moving), providing a core basis for screening key frames containing important dynamic events.

[0034] Step 203: Use a similarity algorithm to determine the content similarity between consecutive frames in the video to be processed.

[0035] In the embodiments of this application, the similarity algorithm refers to an algorithm used to calculate the degree of visual similarity between each consecutive frame in the video to be processed; a consecutive frame refers to a continuous frame sequence in the video to be processed, which is not limited to two frames, but may be a small segment or window; content similarity is a numerical measurement result that represents the degree of visual semantic repetition or similarity between the compared frames or frame segments, and high similarity means that the scene and subject have not changed significantly.

[0036] In this embodiment, each consecutive frame can be processed using image hashing (generating an image "fingerprint" and comparing Hamming distances) or embedding vector similarity calculation (i.e., inputting each frame in the video to be processed into the image encoder of a Contrastive Language-Image Pre-training (CLIP) model, obtaining feature vectors, and then calculating cosine similarity) to obtain the content similarity between each consecutive frame. Then, redundant periods in the video with static scenes, repetitive content, or slow changes can be identified through content similarity. In this way, segments with low information content and repetitive content in the video can be accurately identified, avoiding the selection of too many similar keyframes from these segments, thereby reducing data redundancy in subsequent processing.

[0037] Step 204: Based on the first action change information and content similarity, determine the target frame sequence from the video to be processed.

[0038] In this embodiment, both the first action change information (where the change is occurring) and content similarity (whether the content is novel) are considered. Specifically, when the detected first action change information exceeds a threshold (indicating an event has occurred), frames at or near that moment are preferentially selected. When the content similarity of consecutive frames falls below a certain threshold (indicating a scene or subject change), frames with a sharp drop in similarity are selected as representatives of the new scene. For parts with consistently high content similarity (static or slowly shifting), only a small number or one frame is extracted as a representative. This allows for the selection of key frames with intense movement, scene changes, or events (e.g., when "a person enters the screen," "an event occurs," or "text appears," the frame is preferentially retained). Finally, the selected frames are combined into a significantly simplified target frame sequence arranged chronologically. This significantly reduces the computational load: by removing over 90% of redundant frames, the computational burden on the subsequent graphics processing unit (Graphics Processing) for multi-scale modeling of high-resolution frames is greatly reduced. Unit (GPU) memory pressure and computing time make it possible to process ultra-high-definition long videos; 2. Preserve the integrity of time sequence and events: ensure that the selected keyframes cover the main event turning points of the video, without losing important temporal narrative logic, and lay the foundation for generating a coherent description.

[0039] It should be noted that steps 205, 206 and 207 can be executed in parallel after step 204.

[0040] Step 205: Determine the temporal motion features based on the second action change information between adjacent frames in the target frame sequence.

[0041] In this embodiment, the second motion change information corresponds to the first motion change information extracted from adjacent frames of the video to be processed in step 202. Here, it specifically refers to the motion change data calculated between adjacent frames in the selected target frame sequence. Since the target frame itself is a representative frame with condensed information, the second motion change information here can focus more on the main motion changes that span redundant frames. Temporal motion features refer to the high-level semantic representation or text description used to describe the dynamic behavior and temporal evolution of objects or scenes in the video. It not only includes the direction and amplitude of motion, but also focuses on action descriptions such as "who did what".

[0042] It should be noted that step 205 can be achieved in the following way: Step 205A1: Determine the second action change information between adjacent frames in the target frame sequence.

[0043] In this embodiment of the application, for the extracted target frame sequence, the pixel-level motion between each adjacent frame in the target frame sequence is calculated. Typically, optical flow can be used to calculate the optical flow map, or the frame difference can be calculated. These optical flow maps or difference maps are used as quantified second motion change information, which intuitively reflects the motion trajectory, speed and range of the subject (such as a person or object) between key event points.

[0044] Step 205A2: Use a multimodal large language model and the first cue word to process the change information of each second action to obtain temporal motion features.

[0045] In this embodiment, the Multimodal Large Language Model (MLLM) refers to a multimodal large language model optimized for temporal video modeling, specifically Video-LLaMA, Qwen-VL-Video, or InternVideo2+LLM. It should be noted that this type of model can not only understand images and text, but its architecture (e.g., using a Time-Space Transformer (TimeSformer) or Video Vision Transformer (ViViT) encoder) is specifically designed to process frame sequences and capture temporal dependencies. The first cue word guides the multimodal large language model to focus on and describe motion patterns. The second action change information obtained in step 205A1 can be input into the multimodal large language model along with the first cue word. The encoder inside the multimodal large language model (e.g., TimeSformer) can effectively parse the temporal relationships between frame sequences, while the first cue word guides the decoder to focus on action description generation. In this way, 1. Dynamic semantic understanding is achieved: low-level pixel motion is transformed into high-level, human-readable action descriptions, directly answering the core question of "what actions happened in the video"; 2. The accuracy and coherence of the description are improved: using time-optimized MLLM, the sequence and causal relationships of actions can be better understood (such as "running and then jumping"), generating more accurate and coherent dynamic descriptions than simply analyzing single frames; 3. Focus on core dynamics: since the input is motion information between keyframes, the interference of redundant static frames is avoided, allowing the model to focus more on analyzing and describing the most core dynamic events in the video.

[0046] In one feasible implementation, if the first prompt is: "Please describe the main action in the video based on the following motion trajectory: [Second action change information]", then the output of the multimodal large language model can be: "A man runs from left to right and then jumps over an obstacle."

[0047] Step 206: For each target frame in the target frame sequence, downsample and extract features to obtain global features.

[0048] In this embodiment, downsampling is an image processing operation that reduces the spatial resolution of an image by decreasing the number of pixels. Feature extraction refers to the process of extracting high-level feature vectors that can represent the semantic content of an image. Global features refer to features or descriptions that reflect the overall scene semantics of a video frame. Specifically, these may include environmental categories (such as office, park, home, etc.), the main objects and their quantities (such as a table, five people, etc.), and spatial layout relationships (such as a car on a road), but do not include fine local details. Each target frame in the target frame sequence can be downsampled first, and then feature extraction can be performed to obtain the global features of each target frame. It should be noted that the core idea of ​​determining global features is to sacrifice local details to quickly obtain a macroscopic understanding, thereby avoiding interference from local details. This approach is suitable for scenes with complex backgrounds but no fine requirements (such as urban aerial photography, conference venues, etc.).

[0049] It should be noted that step 206 can be achieved in the following way: Step 206B1: For each target frame, downsample the target frame to obtain the target resolution image of the target frame.

[0050] The target resolution image is an image with a resolution less than the resolution threshold.

[0051] In this embodiment, the target resolution image refers to the image obtained after downsampling, and its resolution (e.g., width × height) is significantly lower than that of the keyframe before downsampling; the resolution threshold is a preset upper limit of size, typically set to the fixed input size of a standard visual encoder (e.g., Vision Transformer-B / 16), such as 224. 224 or 320 320; Specifically, iterate through each target frame in the target frame sequence and perform low-resolution downsampling on each target frame to uniformly reduce its size to a preset target resolution (e.g., 224). 224), thus outputting a series of target resolution images with uniform size and low resolution.

[0052] Step 206B2: Encode each target resolution image using a multimodal large language model and a second cue word to obtain multiple global features.

[0053] In this embodiment of the application, the multimodal large language model specifically refers to a vision-language model, such as Large Language and Vision Assistant (LLaVA) and Qwen-VL. This multimodal large language model has been pre-trained on a large amount of image and text data and is good at understanding the overall scene from images and generating a general description. The second prompt word is a text instruction specifically designed to guide the multimodal large language model to perform a global scene description task. It emphasizes observation and summarization from a macro perspective. The target resolution image obtained in step 206B1, along with a pre-designed second prompt (such as "Please describe the overall scene of this frame: including location, main objects, weather conditions, etc."), can be input into a selected multimodal large language model. The visual encoder in the multimodal large language model then encodes the input target resolution image. Guided by the second prompt, the multimodal large language model generates a natural language description of the overall scene of the frame based on the image features of the target resolution image—that is, the global features of the target frame. By repeating this process for each frame in the target frame sequence, multiple global features corresponding to each key time point are obtained. This approach offers the following advantages: 1. High computational efficiency: The computational and memory overhead for processing low-resolution images is far lower than that for processing original high-resolution frames, enabling the model to quickly traverse all key frames and obtain a stable understanding of the overall macro-level plot of the video; 2. Provides a stable macro-level semantic foundation: The generated global features provide a reliable story background for subsequent text fusion, ensuring the accuracy of the final subtitles in scene description and avoiding interference from local detail noise.

[0054] Step 207: For each target frame, a semantic segmentation model is used to segment the target frame to obtain multiple first regions, and the multiple first regions are processed to obtain local features.

[0055] In this embodiment, the semantic segmentation model is a computer vision model that can assign a semantic category label (such as person, car, sky, background) to each pixel in an image, thereby segmenting the image into regions with different semantic meanings (i.e., semantic region masks). The first region refers to each connected region with a specific semantic label obtained after the initial segmentation of the target frame by the semantic segmentation model, such as a complete human figure region, a car region, a text region, etc. Local features refer to high-level semantic representations or descriptions that reflect key details in the video frame, such as a person's expression, clothing texture, text on an object, fine attributes of a specific object, etc., and it is complementary to global features, focusing on microscopic information.

[0056] In this embodiment, the semantic segmentation model can specifically be a lightweight semantic segmentation model. Therefore, a lightweight semantic segmentation model can be used to perform semantic tiling (i.e., coarse segmentation) on each target frame to obtain multiple first regions. Then, features are extracted from these first regions to obtain global features. It should be noted that this application uses semantic segmentation instead of the fixed grid segmentation (i.e., uniformly dividing a 4K frame into 4...) in related technologies. 4 = 16 times 512 (512 blocks), thus solving the problem in related technologies where a large number of blocks are low-information areas such as sky and walls, wasting computing power.

[0057] In one feasible implementation, the lightweight semantic segmentation model can be either the Mobile Segment Anything Model (MobileSAM) or the Fast Segmentation Convolutional Neural Network (Fast-SCNN).

[0058] It should be noted that the process of processing multiple first regions to obtain local features in step 207 can be achieved in the following way: Step 207C1: Identify and filter multiple first regions to obtain second regions with non-background regions.

[0059] In the embodiments of this application, the second region refers to the set of all non-background regions selected from multiple first regions, and the non-background region refers to the semantic regions other than the background (such as sky, walls, solid-color ground, etc.) in the semantic segmentation result, which usually contain the main body and key information of the video, such as people, vehicles, text, etc.

[0060] In this embodiment, the multiple first regions obtained from the coarse segmentation described above can be parsed. Based on the semantic labels of the regions, regions belonging to the background category are removed, and all non-background regions are retained. These regions are defined as second regions. In this way, a large number of background regions with no or low information (such as large areas of sky or walls) can be directly excluded, avoiding unnecessary and expensive high-resolution analysis of these regions and greatly saving computational resources. It should be noted that the first regions with background areas can be directly downsampled to global features without performing the segmentation in step 207C2 below.

[0061] Step 207C2: For each second region, segment the second region based on its size to obtain at least one initial image block.

[0062] The second region with a size greater than the first threshold uses the first block density, and the second region with a size less than or equal to the first threshold uses the second block density; the second block density is greater than the first block density.

[0063] In this embodiment, size refers to the pixel area or the size of the bounding rectangle occupied by the second region in the frame image; the first threshold is a preset area or size limit used to distinguish between large and small objects; block density refers to the fineness of cutting a region into image blocks, and the first block density is low (e.g., a second region is divided into only 1 block), and the second block density is high (e.g., a second region is divided into multiple small blocks); the initial image block is the basic image unit obtained after dividing each second region according to the block density.

[0064] In this embodiment, each second region is traversed, and its size (e.g., the area of ​​its bounding box) is calculated. If the size of the second region is greater than a first threshold (considered a large object, such as the entire human body), the second region is segmented using a first block density, i.e., using fewer blocks (e.g., one block) to cover the second region, because the overall pose and outline of a large object are more important. If the size of the second region is less than or equal to the first threshold (considered a small object, such as an eye on a face or a piece of text), the second region is segmented using a higher second block density, cutting it into more blocks to ensure that its fine details can be captured. Thus, 1. Intelligent resource allocation: coarse-grained analysis is used for large, structurally simple objects, and fine-grained analysis is used for small objects that require detail, achieving optimized allocation of computing power across regions of different importance; 2. Guaranteed detail coverage: ensuring that small-sized key entities (such as text and logos) can be covered by enough image blocks so that they are not missed in subsequent processing.

[0065] Step 207C3: Determine local features based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches.

[0066] In this embodiment, the hierarchical sampling strategy refers to a pyramid-style multi-scale sampling strategy, which involves sampling and analyzing the same initial image patch (or its corresponding region) at different resolution scales to form hierarchical data from macro to micro. The multimodal large language model here involves two types: a first type of lightweight multimodal large language model (such as MiniCPM-V) and a second type of high-precision multimodal large language model (such as the Qwen-VL-Chat visual language dialogue model), which are used to process features at different scales obtained using the hierarchical sampling strategy. The hierarchical sampling strategy can be used to sample each initial image patch from multiple initial image patches first, and then different types of multimodal large language models can be used to process the results to obtain local features.

[0067] It should be noted that step 207C3 can be achieved in the following way: Step 207c1: For each initial image block, a hierarchical sampling strategy is used to perform multi-scale sampling on the initial image block to obtain an image block set containing at least three different scales.

[0068] The first scale is used to provide the spatial location and range information of the second region in the target frame, the second scale is used to identify the action state of objects in the second region or their interaction with other objects in the scene, and the third scale is used to obtain detailed information in the second region.

[0069] In this embodiment, the image patch set refers to a collection of image samples of different resolutions obtained after multi-scale sampling of the same initial image patch (i.e., a semantic region). Specifically, a hierarchical sampling strategy is adopted to sample each initial image patch at three different scales. The first scale (Level 0, global / original image scale): refers to the original complete keyframe, recording the position coordinates and coverage of the initial image patch (i.e., the second region) in the original high-resolution image, and can be used to locate details (such as the person in the lower left corner of the image) in the final description. The second scale (Level 1, mesoscale): accurately crops the region corresponding to the initial image patch from the original image and then scales it to a medium resolution (e.g., scaled to 768). 768), and can be used to identify the action state or interaction relationship of an object (such as analyzing whether a person's overall posture is standing or running, or determining whether he is holding an object); the third scale (Level 2, local / detail scale): to achieve the preservation of the original high resolution of the area, or to crop it and display it at a high resolution (such as 1024). The input (1024) can be used to obtain the finest details (such as whether a person's expression is smiling or frowning, the text content on clothing, or the texture of an object), thus generating three images at different scales for an initial image patch. In this way, 1. a complete information hierarchy covering "location-relationship-detail" is established for each key region, laying the foundation for generating rich and three-dimensional descriptions; 2. Decoupling analysis objectives: different types of analysis tasks (localization, relationship judgment, detail recognition) are assigned to the most appropriate scale, avoiding the problem of information mixing or loss at a single scale.

[0070] Step 207c2: For the image patch set at the first scale and the image patch set at the second scale, feature extraction is performed using the first type of multimodal large language model to obtain the first feature. For the image patch set at the third scale, feature extraction is performed using the second type of multimodal large language model to obtain the second feature.

[0071] Among them, the first type of multimodal large language model is a lightweight language model, and the second type of multimodal large language model is a high-precision language model.

[0072] In this application embodiment, the first type of multimodal large language model is a lightweight language model, specifically referring to a visual-language model (such as MiniCPM-V) with relatively few parameters, fast inference speed, and low computational resource requirements, and which is sufficient to handle overall and relational understanding tasks with low detail requirements; the second type of multimodal large language model is a high-precision language model, specifically referring to a model with a large number of parameters, strong capabilities, and higher accuracy in complex visual understanding tasks (Qwen-VL-Chat), which is used to handle detail recognition tasks with extremely high accuracy requirements; the first feature is the semantic feature extracted by the lightweight model or the descriptive text generated, and the second feature is the semantic feature extracted by the high-precision model or the descriptive text generated, with the first feature focusing on spatial and relational semantics, and the second feature focusing on fine detail semantics. The first-scale image patch set (location information, usually incorporated into the prompt words in coordinate form) and the second-scale image patch set (meso-resolution image patches) can be used as input to a first-class multimodal large language model (lightweight Vision Language Model, VLM). This first-class VLM then generates descriptions of the location and relationships between objects in the region; these descriptions are aggregated into the first feature. The third-scale image patch set (high-resolution detail image patches) can be used as input to a second-class multimodal large language model (high-precision VLM). This second-class VLM focuses on identifying and describing the subtle details in the image, generating detailed descriptions; these descriptions are aggregated into the second feature. This approach achieves the following: 1. Optimized computational resource allocation: Lightweight models are used for spatial and relational analyses with lower information density and simpler tasks, while high-cost high-precision models are only used for detail analyses requiring the highest precision, achieving optimal overall computational efficiency; 2. Ensured quality of detail descriptions: The strongest model is used for key details, maximizing the accuracy of recognizing subtle content such as facial expressions and text, thereby improving the precision and reliability of the final subtitles.

[0073] Step 207c3: Determine local features based on the first and second features.

[0074] In this embodiment, the first feature (containing information such as region location and object relationships) and the second feature (containing fine details) can be fused. This fusion is not a simple splicing, but rather an integration based on their semantic relationships. For example, the anxious expression identified by the high-precision model (i.e., the second feature) and the person walking quickly (i.e., the first feature) determined by the lightweight model can be associated with the same subject, ultimately forming a unified, structured local feature representation. This representation comprehensively describes the details of all important semantic regions in the keyframe, providing a solid micro-information foundation for the final overall text generation. Thus, 1. a complete local semantic representation is formed: multi-level information such as location, relationships, and details are fused into a coherent local story, allowing the subsequent text generator to easily extract and reference this information; 2. high-quality input is provided for the final fusion: the local information input to the final text fusion module is refined, deredundant, and associated structured information, improving the accuracy and richness of the final generated subtitles.

[0075] It should be noted that for areas identified as text, optical character recognition (OCR) technology can be used alone to extract the text, and this text information can be used as additional input to fuse with the first and second features mentioned above.

[0076] It should be noted that step 208 can be executed after steps 205, 206, and 207.

[0077] Step 208: Integrate temporal motion features, global features, and local features to obtain target description information for the video to be processed.

[0078] It should be noted that, as Figure 3The diagram illustrates the process of determining the target description information for the video to be processed. Specifically, firstly, keyframes are extracted from the video using a video summarization algorithm based on optical flow and visual similarity to remove redundant frames and retain core temporal event points. Subsequently, the extracted target frame sequence is input in parallel into three feature extraction modules: In the motion information modeling module, inter-frame optical flow or differences are calculated and input into a temporally optimized multimodal large language model to generate temporal motion features describing dynamic behavior; in the global information modeling module, keyframes are downsampled, and a visual-language model is used to capture the overall scene semantics to generate global features; in the local information modeling module, semantic segmentation is used to identify non-background regions, and adaptive block division is performed based on region size. Then, a pyramid-style multi-scale sampling strategy is used, combined with multimodal large language models of different capabilities, to extract fine local detail features; finally, a text fusion module (usually a large language model) is used to deduplicate, temporally sort, logically infer, and refine the language of the above three types of features, integrating them to generate a semantically complete, detailed, and temporally coherent video text description (i.e., target description information).

[0079] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0080] The information generation method provided in this application firstly avoids the huge computational overhead caused by performing full-resolution depth processing on all frames of the original high-definition video by first extracting the key frame sequence and then performing feature extraction strategies at different granularities. Secondly, by specifically extracting and modeling local features, it ensures the capture and description of key details in the high-definition video, significantly improving the richness and accuracy of the generated description. Thus, while ensuring that the temporal information is not lost, it greatly improves the processing efficiency, making real-time or near-real-time description of the video possible.

[0081] Based on the foregoing embodiments, this application provides an information generation device that can be applied to... Figure 1 and Figure 2 In the information generation method provided in the corresponding embodiment, refer to Figure 4 As shown, the information generation device 3 may include: an acquisition unit 31, a processing unit 32, and a generation unit 33, wherein: The acquisition unit 31 is used to acquire the video to be processed and extract the target frame sequence from the video to be processed. The processing unit 32 is used to determine the multi-scale features of the video to be processed based on the target frame sequence; wherein the multi-scale features include at least the temporal motion features reflecting the dynamic changes of the video to be processed, the global features reflecting the overall content of the video to be processed, and the local features reflecting the detailed content of the video to be processed. The generation unit 33 is used to integrate temporal motion features, global features and local features to obtain target description information of the video to be processed.

[0082] In other embodiments of this application, the acquisition unit 31 is further configured to perform the following steps: Determine the first motion change information between adjacent frames in the video to be processed; A similarity algorithm is used to determine the content similarity between consecutive frames in the video to be processed; Based on the first action change information and content similarity, the target frame sequence is determined from the video to be processed.

[0083] In other embodiments of this application, the processing unit 32 is further configured to perform the following steps: Based on the second action change information between adjacent frames in the target frame sequence, the temporal motion characteristics are determined. For each target frame in the target frame sequence, downsampling and feature extraction are performed on the target frame to obtain global features; For each target frame, a semantic segmentation model is used to segment the target frame to obtain multiple first regions, and the multiple first regions are processed to obtain local features.

[0084] In other embodiments of this application, the processing unit 32 is further configured to perform the following steps: Determine the second action change information between adjacent frames in the target frame sequence; A multimodal large language model and the first cue word are used to process the change information of each second action to obtain temporal motion features.

[0085] In other embodiments of this application, the processing unit 32 is further configured to perform the following steps: For each target frame, the target frame is downsampled to obtain the target resolution image of the target frame; where the target resolution image is an image with a resolution less than a resolution threshold. A multimodal large language model and a second cue word are used to encode each target resolution image to obtain multiple global features.

[0086] In other embodiments of this application, the processing unit 32 is further configured to perform the following steps: Multiple first regions are identified and filtered to obtain second regions with non-background areas; For each second region, the second region is segmented based on its size to obtain at least one initial image block; wherein, second regions with a size greater than a first threshold are segmented using a first block density, and second regions with a size less than or equal to the first threshold are segmented using a second block density; the second block density is greater than the first block density. Local features are determined based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches.

[0087] In other embodiments of this application, the processing unit 32 is further configured to perform the following steps: For each initial image patch, a hierarchical sampling strategy is used to perform multi-scale sampling on the initial image patch to obtain an image patch set containing at least three different scales; wherein, the first scale is used to provide the spatial location and range information of the second region in the target frame, the second scale is used to identify the action state of the object in the second region or its interaction with other objects in the scene, and the third scale is used to obtain detailed information in the second region. For image patch sets at the first and second scales, a first type of multimodal large language model is used for feature extraction to obtain the first feature. For image patch sets at the third scale, a second type of multimodal large language model is used for feature extraction to obtain the second feature. The first type of multimodal large language model is a lightweight language model, and the second type of multimodal large language model is a high-precision language model. Based on the first and second features, local features are determined.

[0088] It should be noted that a detailed explanation of the steps performed by each unit can be found in [reference needed]. Figure 1 and Figure 2 The information generation method provided in the corresponding embodiments will not be described in detail here.

[0089] The information generation apparatus provided in this application firstly avoids the huge computational overhead caused by performing full-resolution depth processing on all frames of the original high-definition video by first extracting the key frame sequence and then performing feature extraction strategies at different granularities. Secondly, by specifically extracting and modeling local features, it ensures the capture and description of key details in the high-definition video, significantly improving the richness and accuracy of the generated description. Thus, while ensuring that the temporal information is not lost, it greatly improves the processing efficiency, making real-time or near-real-time description of the video possible.

[0090] Based on the foregoing embodiments, embodiments of this application provide an information generation device that can be applied to... Figure 1 and Figure 2 In the information generation method provided in the corresponding embodiment, refer to Figure 5 As shown, the information generation device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein: Communication bus 43 is used to realize the communication connection between processor 41 and memory 42; The processor 41 is used to execute the information generation program in the memory 42 to perform the following steps: Acquire the video to be processed and extract the target frame sequence from the video to be processed; Based on the target frame sequence, determine the multi-scale features of the video to be processed; wherein, the multi-scale features include at least the temporal motion features reflecting the dynamic changes of the video to be processed, the global features reflecting the overall content of the video to be processed, and the local features reflecting the detailed content of the video to be processed. By integrating temporal motion features, global features, and local features, target description information of the video to be processed is obtained.

[0091] In other embodiments of this application, processor 41 is used to execute the information generation program in memory 42 to extract a target frame sequence from the video to be processed, in order to perform the following steps: Determine the first motion change information between adjacent frames in the video to be processed; A similarity algorithm is used to determine the content similarity between consecutive frames in the video to be processed; Based on the first action change information and content similarity, the target frame sequence is determined from the video to be processed.

[0092] In other embodiments of this application, processor 41 is used to execute the information generation program in memory 42 based on the target frame sequence to determine the multi-scale features of the video to be processed, in order to implement the following steps: Based on the second action change information between adjacent frames in the target frame sequence, the temporal motion characteristics are determined. For each target frame in the target frame sequence, downsampling and feature extraction are performed on the target frame to obtain global features; For each target frame, a semantic segmentation model is used to segment the target frame to obtain multiple first regions, and the multiple first regions are processed to obtain local features.

[0093] In other embodiments of this application, the processor 41 is used to execute the information generation program in the memory 42 to determine temporal motion features based on the second action change information between adjacent frames in the target frame sequence, in order to implement the following steps: Determine the second action change information between adjacent frames in the target frame sequence; A multimodal large language model and the first cue word are used to process the change information of each second action to obtain temporal motion features.

[0094] In other embodiments of this application, the processor 41 is used to execute the information generation program in the memory 42 to downsample and extract features from the target frame to obtain global features, in order to implement the following steps: For each target frame, the target frame is downsampled to obtain the target resolution image of the target frame; where the target resolution image is an image with a resolution less than a resolution threshold. A multimodal large language model and a second cue word are used to encode each target resolution image to obtain multiple global features.

[0095] In other embodiments of this application, the processor 41 is used to execute the information generation program in the memory 42 to process multiple first regions to obtain local features, in order to implement the following steps: Multiple first regions are identified and filtered to obtain second regions with non-background areas; For each second region, the second region is segmented based on its size to obtain at least one initial image block; wherein, second regions with a size greater than a first threshold are segmented using a first block density, and second regions with a size less than or equal to the first threshold are segmented using a second block density; the second block density is greater than the first block density. Local features are determined based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches.

[0096] In other embodiments of this application, processor 41 is used to execute the information generation program in memory 42 based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches to determine local features in order to achieve the following steps: For each initial image patch, a hierarchical sampling strategy is used to perform multi-scale sampling on the initial image patch to obtain an image patch set containing at least three different scales; wherein, the first scale is used to provide the spatial location and range information of the second region in the target frame, the second scale is used to identify the action state of the object in the second region or its interaction with other objects in the scene, and the third scale is used to obtain detailed information in the second region. For image patch sets at the first and second scales, a first type of multimodal large language model is used for feature extraction to obtain the first feature. For image patch sets at the third scale, a second type of multimodal large language model is used for feature extraction to obtain the second feature. The first type of multimodal large language model is a lightweight language model, and the second type of multimodal large language model is a high-precision language model. Based on the first and second features, local features are determined.

[0097] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 and Figure 2 The information generation method provided in the corresponding embodiments will not be described in detail here.

[0098] The information generation device provided in this application firstly avoids the huge computational overhead caused by performing full-resolution depth processing on all frames of the original high-definition video by first extracting the key frame sequence and then performing feature extraction strategies at different granularities. Secondly, by specifically extracting and modeling local features, it ensures the capture and description of key details in the high-definition video, significantly improving the richness and accuracy of the generated description. Thus, while ensuring that the temporal information is not lost, it greatly improves the processing efficiency, making real-time or near-real-time description of the video possible.

[0099] Based on the foregoing embodiments, this application provides a computer program product, including a computer program, which, when executed by a processor, implements... Figure 1 and Figure 2 The corresponding embodiments provide the steps of the information generation method.

[0100] Based on the foregoing embodiments, this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to achieve... Figure 1 and Figure 2 The corresponding embodiments provide the steps of the information generation method.

[0101] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An information generation method, characterized in that, The method includes: Acquire the video to be processed, and extract the target frame sequence from the video to be processed; Based on the target frame sequence, multi-scale features of the video to be processed are determined; wherein, the multi-scale features include at least temporal motion features reflecting the dynamic changes of the video to be processed, global features reflecting the overall content of the video to be processed, and local features reflecting the detailed content of the video to be processed. The temporal motion features, global features, and local features are integrated to obtain the target description information of the video to be processed.

2. The method according to claim 1, characterized in that, Extracting the target frame sequence from the video to be processed includes: Determine the first motion change information between adjacent frames in the video to be processed; A similarity algorithm is used to determine the content similarity between consecutive frames in the video to be processed; Based on the first action change information and the content similarity, the target frame sequence is determined from the video to be processed.

3. The method according to claim 1, characterized in that, The step of determining the multi-scale features of the video to be processed based on the target frame sequence includes: The temporal motion features are determined based on the second action change information between adjacent frames in the target frame sequence; For each target frame in the target frame sequence, the target frame is downsampled and its features are extracted to obtain the global features; For each target frame, a semantic segmentation model is used to segment the target frame to obtain multiple first regions, and the multiple first regions are processed to obtain the local features.

4. The method according to claim 3, characterized in that, The step of determining the temporal motion features based on the second action change information between adjacent frames in the target frame sequence includes: Determine the second action change information between adjacent frames in the target frame sequence; The temporal motion features are obtained by processing each second action change information using a multimodal large language model and a first cue word.

5. The method according to claim 3, characterized in that, The process of downsampling and feature extraction on the target frame to obtain the global features includes: For each target frame, the target frame is downsampled to obtain a target resolution image of the target frame; wherein, the target resolution image is an image with a resolution less than a resolution threshold; A multimodal large language model and a second cue word are used to encode each target resolution image to obtain multiple global features.

6. The method according to claim 3, characterized in that, The process of processing the plurality of first regions to obtain the local features includes: The plurality of first regions are identified and filtered to obtain a second region with a non-background region; For each of the second regions, the second region is segmented based on its size to obtain at least one initial image block; wherein, the second regions with a size greater than a first threshold are segmented using a first block density, and the second regions with a size less than or equal to the first threshold are segmented using a second block density; the second block density is greater than the first block density; The local features are determined based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches.

7. The method according to claim 6, characterized in that, The determination of local features based on a hierarchical sampling strategy, a multimodal large language model, and multiple initial image patches includes: For each initial image patch, the hierarchical sampling strategy is used to perform multi-scale sampling on the initial image patch to obtain an image patch set containing at least three different scales; wherein, the first scale is used to provide the spatial location and range information of the second region in the target frame, the second scale is used to identify the action state of the object in the second region or its interaction with other objects in the scene, and the third scale is used to obtain detailed information in the second region. For the image patch set at the first scale and the image patch set at the second scale, a first type of multimodal large language model is used for feature extraction to obtain a first feature, and for the image patch set at the third scale, a second type of multimodal large language model is used for feature extraction to obtain a second feature; wherein, the first type of multimodal large language model is a lightweight language model, and the second type of multimodal large language model is a high-precision language model; The local features are determined based on the first feature and the second feature.

8. An information generation device, characterized in that, The information generation device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the method according to any one of claims 1 to 7.

9. A computer program product, the computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs that can be executed by one or more processors to implement the method as described in any one of claims 1 to 7.

Images