Video understanding method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a video understanding method and device, computer equipment and a storage medium. The method relates to the technology, is applied to the financial field, and comprises the following steps: preprocessing an original video to be understood, and converting multiple frames of images corresponding to the original video into feature vectors; calculating the feature difference of the adjacent frame images, screening out key frame images from the multiple frame images according to the feature difference, and forming a key frame sequence according to the key frame images; dividing the key frame sequence into a plurality of data blocks, inputting the data blocks into a pre-trained video big language model, and outputting a compressed video feature cache; and obtaining question information corresponding to the original video, and generating video understanding information through joint reasoning according to the compressed video feature cache and the question information. The method provides a general solution with both efficiency and precision, and breaks through the application boundary of the prior art especially for the fields of finance, medical treatment and the like which have strict requirements on information integrity and timeliness.

Description

Technical Field

[0001] This application relates to the field of data compression algorithm technology, and in particular to a video understanding method, apparatus, computer device and storage medium. Background Technology

[0002] With the deepening application of artificial intelligence technology in vertical fields, the demand for long-form video understanding is experiencing explosive growth in scenarios such as financial risk control, medical and health care, and elderly care. However, existing technologies face significant bottlenecks: In financial risk control scenarios, real-time abnormal behavior detection is required on massive amounts of surveillance video (such as 24 / 7 recordings from bank counters and ATMs). For example, this involves identifying operational oversights in long-term transactions and tracking suspicious individuals' prolonged, abnormal loitering patterns. However, traditional methods rely solely on temporal redundancy compression (such as discarding adjacent similar frames), failing to distinguish between critical information (such as customer signature actions and cash counting details) and secondary information (such as changes in background lighting), leading to the accidental deletion of crucial risk control clues.

[0003] In healthcare and elderly care settings, remote monitoring systems require long-term analysis of videos taken at home by elderly individuals (such as fall detection and medication behavior monitoring), with video durations typically exceeding 12 hours. Traditional compression methods focus only on visually repetitive frames, neglecting knowledge redundancy: for example, the continuous movements of an elderly person walking daily are temporally redundant and compressible, but subtle hand movements for grasping medication, though visually minor, involve crucial knowledge about medication safety, which traditional methods may misjudge as secondary information.

[0004] Therefore, a method is urgently needed to solve at least one of the above problems. Summary of the Invention

[0005] This application discloses a video understanding method, apparatus, computer device, and storage medium, which aims to address the problem that with the deepening application of artificial intelligence technology in vertical fields, the demand for long video understanding is showing explosive growth in scenarios such as financial risk control, medical health and elderly care, but existing technologies face significant bottlenecks.

[0006] Firstly, this application provides a video understanding method, including: The original video to be understood is preprocessed by converting the multi-frame images corresponding to the original video into feature vectors. Calculate the feature differences between adjacent frames, select keyframe images from multiple frames based on the feature differences, and form a keyframe sequence based on the keyframe images; The keyframe sequence is divided into multiple data blocks and input into a pre-trained video big language model. When the video big language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information. Based on the importance information, the target visual token is selected from the multiple visual tokens, and the compressed video feature cache is output. Obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

[0007] In some embodiments, the preprocessing of the original video to be understood, which converts the multi-frame images corresponding to the original video into feature vectors, includes: processing each frame image of the original video through a preset visual encoder, dividing each frame image into a preset number of patches; extracting features from each patch and converting it into a corresponding mathematical vector to form a vector set containing a three-dimensional data structure including the number of frames, the number of tokens per frame, and the dimension of the token vector.

[0008] In some embodiments, calculating the feature difference between adjacent frame images includes: for each pair of adjacent frame images in the video, calculating the difference value between the corresponding vectors of the same position patches in the adjacent frame images; summing the difference values ​​of all corresponding position patches and taking the average value to obtain the overall difference degree of adjacent frame images, forming a difference degree sequence that characterizes the degree of drastic change in video content as the feature difference.

[0009] In some embodiments, the step of selecting key frame images from multiple frames based on feature differences and forming a key frame sequence based on the key frame images includes: analyzing the difference sequence corresponding to the feature differences and obtaining the corresponding local peak points; marking the frames corresponding to the local peak points as the core frames, while retaining non-peak frames with high difference scores and discarding frames with extremely low difference scores, forming the key frame sequence containing the core frame and other high difference frames, and generating a label list to indicate the core frame in the key frame sequence.

[0010] In some embodiments, determining the importance information of the visual token corresponding to each data block based on the generated attention information, filtering target visual tokens from multiple visual tokens according to the importance information, and outputting a compressed video feature cache includes: using the attention matrix generated when processing the data block using a video large language model, calculating the total attention of each visual token among all attention heads and taking the average to obtain the importance score of the visual token; if the visual token comes from the core frame, setting its corresponding importance score to an extremely high value; sorting the tokens according to the importance score to form a compressed video feature cache.

[0011] In some embodiments, dividing the keyframe sequence into multiple data blocks includes: dividing the keyframe sequence into multiple data blocks sequentially according to a preset fixed number of frames, each data block containing the same number of frames, so as to input the video large language model in sequence.

[0012] In some embodiments, generating video understanding information through joint reasoning based on the compressed video feature cache and the question information includes: receiving the compressed video feature cache and the user's question information using a language decoder in a video large language model; using the question information as reasoning clues to guide the language decoder to locate relevant information in the video feature cache; and generating text-based answer information word by word or phrase by phrase based on the question information and relevant information to form video understanding information.

[0013] Secondly, this application provides a video understanding device, comprising: The video processing unit is used to preprocess the original video to be understood, converting the multi-frame images corresponding to the original video into feature vectors. The difference calculation unit is used to calculate the feature differences between adjacent frame images, select key frame images from multiple frames based on the feature differences, and form a key frame sequence based on the key frame images. The model input unit is used to divide the keyframe sequence into multiple data blocks and input them into the pre-trained video big language model. When the video big language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information, filters the target visual token from the multiple visual tokens according to the importance information, and outputs the compressed video feature cache. The understanding generation unit is used to obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

[0014] Thirdly, this application provides a computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor. The memory stores a strategy model, and the computer program, when executed by the processor, implements the video understanding method provided in any embodiment of this application.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the video understanding method provided in any embodiment of this application.

[0016] This application provides a video understanding method, apparatus, computer device, and storage medium. The provided method is...

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic flowchart illustrating the steps of a video understanding method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a video understanding device provided in an embodiment of this application; Figure 3 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0023] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0024] It should be understood that, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with substantially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0025] It should also be understood that the term "and / or" as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] To facilitate understanding of the embodiments of this application, some terms involved in the embodiments of this application will be briefly explained below.

[0027] With the deepening application of artificial intelligence technology in vertical fields, the demand for long-form video understanding is experiencing explosive growth in scenarios such as financial risk control, medical and health care, and elderly care. However, existing technologies face significant bottlenecks: In financial risk control scenarios, real-time anomaly detection is required on massive amounts of surveillance video (such as 24 / 7 recordings from bank counters and ATMs). For example, identifying operational oversights in long-transaction transactions or tracking suspicious individuals' prolonged, abnormal loitering patterns. However, traditional methods rely solely on temporal redundancy compression (such as discarding adjacent similar frames), failing to distinguish between critical information (such as customer signature actions and cash counting details) and secondary information (such as changes in background lighting), leading to the accidental deletion of crucial risk control clues. Furthermore, length extrapolation techniques are limited by hardware memory, making it difficult to handle continuous surveillance video exceeding 4 hours in a single segment. Financial compliance requirements often necessitate analyzing complete recordings over several days; existing solutions suffer from memory overflows, causing analysis interruptions or requiring segmented processing, severely impacting the timeliness and completeness of risk control.

[0028] In healthcare and elderly care scenarios, remote monitoring systems require long-term analysis of videos taken at home by elderly individuals (such as fall detection and medication behavior monitoring), with video durations typically exceeding 12 hours. Traditional compression methods focus only on visually repetitive frames, neglecting knowledge redundancy: for example, the continuous movements of an elderly person walking daily are temporally redundant and compressible, but subtle hand movements for grasping medication, though visually minor, involve crucial knowledge about medication safety, which traditional methods may misclassify as secondary information. Furthermore, real-time analysis demands low-latency model response, but existing technologies suffer from exponentially increasing computation time due to KV cache expansion, failing to meet the stringent requirements of real-time early warning in medical scenarios (such as accurately capturing the golden rescue time after a fall).

[0029] Whether it's long-term monitoring and analysis in the financial sector or real-time health monitoring in the medical field, the core challenges both point to: The inefficiency of single redundancy compression: It only utilizes temporal redundancy and ignores the differences in the importance of information at the knowledge level, resulting in the loss of key semantic information; Hardware-dependent non-scalability: The length extrapolation does not break through the physical limit of the video memory, and cannot cope with the ultra-long video input that is common in scenarios such as finance and healthcare (such as 24-hour continuous monitoring recording). Insufficient domain adaptability: Compression strategies were not designed for the professional knowledge of vertical domains (such as financial business processes and medical behavioral semantics), and general methods are difficult to meet the industry's accuracy requirements.

[0030] To address the aforementioned problems, this application proposes a video understanding method. Please refer to... Figure 1 , Figure 1 This is a schematic flowchart illustrating the steps of a video understanding method provided in an embodiment of this application.

[0031] It should be noted that the compressed inference method provided in this application can be used in financial systems, as well as in any other scenarios with stringent requirements for completeness and timeliness, such as healthcare, education, and entertainment. This application uses financial and healthcare scenarios as examples for illustration.

[0032] It should be noted that the acquisition of any information involved in the methods and embodiments provided in this application complies with relevant regulations and will not infringe on user privacy.

[0033] like Figure 1 As shown, the proposed video understanding method includes steps S101 to S104.

[0034] S101. Preprocess the original video to be understood by converting the multi-frame images corresponding to the original video into feature vectors.

[0035] Specifically, by converting consecutive frames of the original video into semantic feature vectors, the shortcomings of traditional methods that rely solely on pixel differences are addressed, thus preserving key semantic information.

[0036] Visual encoder: Uses a pre-trained CNN (such as ResNet) or ViT (Visual Transformer) to extract frame-level visual features, with an output dimension of H×W×C (H / W is the feature map size, and C is the number of channels).

[0037] Multi-scale feature fusion extracts features from frames of different resolutions (such as pyramid pooling) to capture detailed information (such as signature handwriting and drug labels) and global information (such as human posture).

[0038] In financial / medical scenarios, domain data is used to fine-tune the visual encoder, enhancing its sensitivity to specialized scene elements such as bank documents and medical equipment.

[0039] Frame sampling includes: Uniform sampling: Extracting raw frames at a fixed frame rate (e.g., 10fps) to avoid missing critical actions. Adaptive sampling: Increasing the sampling rate for dynamically changing areas (e.g., counter operation areas, elderly activity areas) and decreasing the sampling rate for static areas.

[0040] Feature generation involves inputting a single-frame image to a visual encoder and outputting a multi-dimensional feature vector (e.g., 768-dimensional), with each vector corresponding to the semantics of a local region of the image (e.g., "hand gesture", "cash", "medicine bottle").

[0041] In financial scenarios, for bank counter videos, the visual encoder focuses on extracting features from the signature board and cash counting area to generate feature vectors containing semantics such as "signature handwriting trajectory" and "number of cash bills", rather than simply recording pixel values.

[0042] In medical settings, when analyzing videos of elderly people at home, features such as "hand grasping actions," "medicine bottle labels," and "body tilt angles" are extracted, while redundant pixel information such as background lighting or furniture is ignored.

[0043] S102. Calculate the feature differences between adjacent frame images, select key frame images from multiple frames based on the feature differences, and form a key frame sequence based on the key frame images.

[0044] Specifically, by combining feature difference calculation with domain knowledge, frames containing key semantics are dynamically selected to solve the problem of losing key information in traditional temporal redundancy compression.

[0045] Feature difference measurement calculates the cosine similarity or Euclidean distance between the feature vectors of adjacent frames. Frames with a difference value greater than a threshold (such as 0.3) are considered potential keyframes.

[0046] Knowledge-guided filtering, combined with domain rules (such as financial compliance processes and medical behavior knowledge bases), forces the retention of frames with small visual differences but important semantics (such as "mistaken medication" and "abnormal signature order").

[0047] The sliding window optimization employs a strategy of preserving the most different frames within a fixed window (e.g., 50 frames) to avoid the accumulation of consecutive similar frames.

[0048] For consecutive frame feature vectors Ft and Ft+1, the difference value Dt=1 is calculated. cos(Ft,Ft+1).

[0049] Keyframe determination: If Dt > threshold, or if the frame contains key elements in the domain (such as identifying "cash" or "medicine bottle" through object detection), it is marked as a keyframe.

[0050] The sequence is constructed by arranging keyframes in chronological order to form a sparse keyframe sequence (e.g., compressing the original video of 1000 frames into 100 keyframes).

[0051] In financial scenarios, during ATM monitoring, if a frame is detected as "abnormally long-lasting in the camera's blind spot" (with little visual difference but conforming to suspicious behavior rules), the system will forcibly retain the frame as a key frame to avoid accidental deletion of clues due to traditional compression.

[0052] In a medical setting, an elderly person's hand movements when grasping medication are subtle (visual difference < threshold), but the knowledge base determines that "the type of medication does not match the doctor's order." This frame is marked as a keyframe for subsequent medication compliance analysis.

[0053] S103. Divide the keyframe sequence into multiple data blocks and input them into a pre-trained video large language model. When the video large language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information, selects the target visual token from the multiple visual tokens according to the importance information, and outputs the compressed video feature cache.

[0054] Specifically, the keyframe sequence is divided into blocks and input into the Video-LLM (Video Large Language Model). The importance of visual tokens is identified through the attention mechanism, redundant information is compressed and an efficient cache is generated, thus solving the problems of hardware memory limitations and computation time consumption.

[0055] Divide the keyframe sequence into fixed-length data blocks (e.g., 50 frames per block) to avoid excessively long blocks causing memory overflow.

[0056] Each frame's feature map is divided into 16x16 pixel patches using ViT, generating a visual token Vi∈R. N×D (N is the number of tokens, D is the dimension).

[0057] Attention weight filtering utilizes the self-attention mechanism of Video-LLM to calculate the attention weight of each token, retaining target tokens with weights higher than the threshold (such as the top 30% of important tokens), and discarding secondary tokens such as background tokens.

[0058] Incremental caching reduces memory usage (e.g., compression rate of over 50%) by storing only key tokens in a KV cache.

[0059] Blocking and tokenization are performed by dividing the keyframe sequence {F1, F2, ..., FM} into B1, B2, ..., BK blocks in chronological order, with each block containing T frames. Each frame generates L visual tokens using ViT, forming a token sequence Vt∈R. L×D .

[0060] When Video-LLM processes each data block, it outputs an attention matrix A∈R T×LWhere At,l represents the importance weight of the l-th token in frame t. Sort by weight, retain the top K% of tokens (e.g., K=30), generating the compressed token sequence Vt′∈R. K ×D .

[0061] Store the compressed tokens and their attention weights to form a lightweight feature cache Cache={(V1′,A1′),…,(VK′,AK′)}.

[0062] In financial scenarios, when analyzing 4 hours of counter video, the system processes keyframe sequences in blocks. In the "cash overage operation" data block, attention is focused on tokens such as "money counting gestures" and "amount display screen," while compressing secondary tokens such as background personnel movement, reducing the single block memory usage by 60% while retaining 90% of the key semantics.

[0063] In medical scenarios, when processing 12 hours of video footage of elderly people at home, in the "fall detection" data block, the model identifies tokens such as "body center of gravity shift" and "arm protective extension" through attention, discards redundant tokens such as "curtain swaying", and the compressed cache supports real-time calculation (latency <500ms), meeting the early warning requirements of the golden rescue time.

[0064] S104. Obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

[0065] Specifically, by combining compressed feature caching with user questions, semantic understanding results are generated through cross-modal reasoning, thus addressing the problem of insufficient domain adaptability of general models.

[0066] Question encoding transforms the question into a text embedding Q∈R using a text encoder (such as BERT). D .

[0067] Cross-modal attention is achieved by fusing visual cache and text embedding in Video-LLM, calculating cross-attention C=Attn(Q,Cache), and locating question-related visual tokens.

[0068] Introduce financial risk control rules (such as "long-term loans require double review") or medical guidelines (such as "early warnings are required within 10 seconds of a fall") during the reasoning stage to constrain the professionalism of the reasoning results.

[0069] Extract query keywords (such as "abnormal wandering" and "correct medication") and map them to entities in the domain knowledge graph (such as "suspicious behavior" and "drug type").

[0070] Based on keyword retrieval feature cache, locate data blocks containing relevant tokens (such as financial video blocks containing the "person stay area" token, or medical video blocks containing the "medicine bottle contact" token).

[0071] Video-LLM generates natural language responses or warning signals, such as "Between 14:00 and 14:15, a customer lingered in front of the ATM for more than 10 minutes, accompanied by multiple abnormal glances" or "An elderly person mistakenly took antihypertensive medication at 10:05, and needs to be reminded immediately."

[0072] In a financial scenario, for example, if the question is asked, "On the morning of December 25, 2025, was there any overpayment at the counter that was not reviewed?" the system retrieves the keyframe cache for the corresponding time period, locates the "cash counting" data block, discovers the "single-person operation overpayment" token through cross-modal attention, and generates a conclusion based on compliance rules: "An overpayment operation occurred at 10:30, which did not trigger the two-person review process, and there is a compliance risk." In a medical setting, for example, if the question is asked, "Did the elderly person take their heart medication correctly today?" the system analyzes the token in the "drug capture" keyframe, identifies the medicine bottle label as "antihypertensive drug" (not a prescribed medication), and generates an alert based on the medication knowledge base: "The elderly person has mistakenly taken antihypertensive medication and actually needs to take nitroglycerin. Immediate intervention is recommended." In some embodiments, the preprocessing of the original video to be understood, which converts the multi-frame images corresponding to the original video into feature vectors, includes: processing each frame image of the original video through a preset visual encoder, dividing each frame image into a preset number of patches; extracting features from each patch and converting it into a corresponding mathematical vector to form a vector set containing a three-dimensional data structure including the number of frames, the number of tokens per frame, and the dimension of the token vector.

[0073] By preprocessing the raw video to be understood, each frame image is segmented into a fixed number of tiles by a visual encoder and converted into mathematical vectors, forming a three-dimensional data structure (vector set M) containing the number of frames, the number of tokens, and the vector dimension.

[0074] The original video is processed frame by frame using a pre-trained visual encoder (such as ViT or Swin Transformer); each frame is uniformly divided into N patches (e.g., N=256); a feature vector (e.g., 768-dimensional) is extracted from each patch to form M∈R 帧数×N×维度 The three-dimensional structure.

[0075] In financial scenarios, bank counter surveillance video analysis can detect operational oversights in long-term transactions. During preprocessing, frames in the counter area are segmented into tiles, with a focus on extracting tile vectors of tellers' hand operations (such as cash counting and keyboard input); tile vectors of background areas (such as walls and irrelevant personnel) are treated as secondary information and can be discarded preferentially during subsequent compression.

[0076] In medical settings, monitoring elderly people's medication use at home involves recognizing subtle hand movements when grasping medications. During preprocessing, the image of the elderly person's hand area is segmented into independent tokens and converted into vectors; the vectors of environmental background (such as furniture and the floor) serve as redundant information, providing a basis for subsequent compression.

[0077] In some embodiments, calculating the feature difference between adjacent frame images includes: for each pair of adjacent frame images in the video, calculating the difference value between the corresponding vectors of the same position patches in the adjacent frame images; summing the difference values ​​of all corresponding position patches and taking the average value to obtain the overall difference degree of adjacent frame images, forming a difference degree sequence that characterizes the degree of drastic change in video content as the feature difference.

[0078] The feature differences between adjacent frames are calculated, and the differences between tile vectors are measured by cosine similarity to generate a difference sequence [d1,d2,...], which reflects the degree of drastic change in video content.

[0079] For each pair of adjacent frames i and i+1 in the video, calculate the cosine difference of the corresponding position patches: ; di: This is a numerical value representing the overall difference between the i-th frame and the (i+1)-th frame. This application calculates a d value for each pair of adjacent frames in the video.

[0080] M: Represents the vector set obtained after preprocessing the entire video. It is a three-dimensional data structure, which can be understood as [number of frames, number of tokens per frame, vector dimension of each token].

[0081] Mi,j: represents the mathematical vector corresponding to the j-th visual token (i.e. the j-th tile) in the i-th frame of the video.

[0082] Mi+1,j: represents the mathematical vector corresponding to the j-th visual token at the same position in the (i+1)-th frame of the video.

[0083] N: Represents the total number of visual tokens into which a frame of an image is decomposed (e.g., 256).

[0084] cosA,B: This is the cosine similarity function, used to calculate the similarity of two vectors A and B in terms of direction. The result ranges from -1 to 1. If the two vectors are in the same direction, the value is 1 (most similar); if they are in opposite directions, the value is -1; if they are perpendicular, the value is 0.

[0085] 1 cos…: Converts “similarity” to “difference” or “distance”. If two frames are exactly the same, the cos value is 1, and the difference is 0. The greater the difference, the greater the difference in content between the two frames.

[0086] This is the summation symbol, representing a loop calculation. It calculates the token differences between all corresponding positions from the 1st to the Nth token between the i-th and i+1-th frames, and then adds them together.

[0087] 1 / N: The sum of the total differences is divided by the total number of tokens N, which is the average value. This prevents inconsistencies in the units of difference measurement due to different image resolutions (number of tokens).

[0088] Using this formula, this application ultimately obtains a difference sequence [d1, d2, d3, ...], which is like an "electrocardiogram" of the video, recording the drastic changes in the video content.

[0089] Generate a difference sequence; the larger the value, the more drastic the changes between frames (such as actions or scene transitions).

[0090] In financial scenarios, the abnormal loitering patterns of suspicious individuals in front of ATMs are tracked. The tile difference in position and action between adjacent frames is calculated: when a person changes from standing to crouching (e.g., installing a skimming device), the difference di increases significantly; when a person loiters still for a long time, the difference is low, and the corresponding frame can be marked as redundant.

[0091] In medical scenarios, during fall detection and identification of sudden changes in movement, the inter-frame difference between an elderly person walking (low difference) and falling (high difference) is calculated: the peak difference between adjacent frames at the moment of the fall can be quickly captured, triggering an alert.

[0092] In some embodiments, the step of selecting key frame images from multiple frames based on feature differences and forming a key frame sequence based on the key frame images includes: analyzing the difference sequence corresponding to the feature differences and obtaining the corresponding local peak points; marking the frames corresponding to the local peak points as the core frames, while retaining non-peak frames with high difference scores and discarding frames with extremely low difference scores, forming the key frame sequence containing the core frame and other high difference frames, and generating a label list to indicate the core frame in the key frame sequence.

[0093] Keyframes are selected based on the difference sequence, including "core frames" (peak difference frames) and high difference non-peak frames, and a marker list is generated to record the location of the core frames.

[0094] Analyze the difference sequence, identify local peak points (such as the maximum value of di in 3 consecutive frames), and mark them as the core frame; retain non-peak frames with difference higher than a preset threshold (such as mean + standard deviation), and discard redundant frames with difference lower than the threshold (such as static background frames).

[0095] In financial scenarios, key operational nodes are identified by analyzing long-term bank counter transaction videos. The core frames are those corresponding to peak actions such as customer signing and teller counting cash; high-discrepancy frames such as customer handing over card and teller verifying vouchers are retained, while low-discrepancy background frames during the waiting period are discarded.

[0096] In a healthcare setting, the medication process for elderly individuals is monitored, distinguishing between medication dispensing and daily activities. The dominant frame is the peak action frame, such as hand touching the medicine bottle or opening the cap; frames with moderate differences, such as walking to the medicine cabinet or checking the medicine box, are retained, while low-difference frames, such as prolonged sitting while watching television, are discarded.

[0097] In some embodiments, determining the importance information of the visual token corresponding to each data block based on the generated attention information, filtering target visual tokens from multiple visual tokens according to the importance information, and outputting a compressed video feature cache includes: using the attention matrix generated when processing the data block using a video large language model, calculating the total attention of each visual token among all attention heads and taking the average to obtain the importance score of the visual token; if the visual token comes from the core frame, setting its corresponding importance score to an extremely high value; sorting the tokens according to the importance score to form a compressed video feature cache.

[0098] When the model processes data blocks, it uses the attention matrix to calculate token importance, forces the retention of core frame tokens, and compresses non-core frame tokens according to their scores.

[0099] For each data block, calculate the token importance score: ; aj: is a numerical value representing the overall importance score of the j-th visual token in the current data block.

[0100] A is the attention matrix within the model. A_{k,i,j} represents the "attention level" of the i-th token to the j-th token in the k-th "attention head" (a calculation mechanism). The higher this value, the more important the information of the j-th token is to the model when understanding the i-th token.

[0101] h represents the total number of "attention heads" in the model. Modern models typically have multiple heads to capture information from different perspectives.

[0102] lq represents the total number of tokens in the current data block.

[0103] :: For a fixed token j, sum up the "attention" of all other tokens i towards it. This measures the degree to which token j is "collectively concerned" in the current context.

[0104] And 1 / h: Sum the “total attention” that token j receives from all h attention heads, then take the average to get the final importance score aj.

[0105] In financial scenarios, this feature identifies operational oversights in over-the-counter transactions (such as failure to verify customer signatures). Signature tile tokens in the main frame are marked as "non-deletable," and their attention scores are forced to the maximum value. Background tokens in non-main frames (such as posters behind tellers) are deleted due to their low scores, reducing video memory usage.

[0106] In medical settings, the system monitors elderly individuals' medication misuse behaviors (such as picking up the wrong medicine bottle) in real time. Tokens representing hand grasping the medicine bottle in the primary frame are protected to ensure the highest attention score; secondary tokens representing walking movements (such as foot movements) in non-primary frames are filtered by score, retaining tokens representing key joint movements.

[0107] In some embodiments, dividing the keyframe sequence into multiple data blocks includes: dividing the keyframe sequence into multiple data blocks sequentially according to a preset fixed number of frames, each data block containing the same number of frames, so as to input the video large language model in sequence.

[0108] The keyframe sequence is divided into fixed-size data blocks (e.g., 32 frames / block) and input into VideoLLM in sequence to avoid processing a large number of tokens at once, which could cause memory overflow.

[0109] The filtered keyframe sequence is divided into continuous blocks according to a fixed frame length C (e.g., C=32): block 1=[1,C], block 2=[C+1,2C],…; each block is input into the model for processing, and the KV buffer is compressed immediately after each block is processed.

[0110] In financial scenarios, such as analyzing 4 hours of continuous surveillance footage, it can detect suspicious individuals exhibiting abnormal behavior over extended periods. The original video (approximately 14,000 frames) is filtered to obtain about 2,000 keyframes, which are then divided into 63 blocks of 32 frames each. Processing is done block by block, and each block is compressed, reducing video memory usage by 70%, thus avoiding memory overflow issues caused by loading the entire frame at once in traditional methods.

[0111] In medical scenarios, such as real-time analysis of 12 hours of home video footage to monitor emergencies like falls, the keyframe sequence is divided into 375 blocks of 32 frames each (assuming 12,000 frames remain after filtering); each block takes approximately 2 seconds to process, with an overall latency of less than 10 minutes, meeting the real-time alert requirements of medical scenarios.

[0112] In some embodiments, generating video understanding information through joint reasoning based on the compressed video feature cache and the question information includes: receiving the compressed video feature cache and the user's question information using a language decoder in a video large language model; using the question information as reasoning clues to guide the language decoder to locate relevant information in the video feature cache; and generating text-based answer information word by word or phrase by phrase based on the question information and relevant information to form video understanding information.

[0113] By combining the compressed video feature cache with user questions, answers are generated through joint reasoning using a language decoder, and relevant information is located using question keywords.

[0114] The language decoder receives a key-value cache (video “memory summary”) and user questions (such as a sequence of text tokens); it uses keywords in the questions (such as “finally”, “signature”, “fall”) as attention-guided cues to retrieve tokens with corresponding timestamps or semantics from the cache; and it generates a word-for-word answer, for example: “The customer paused for 3 seconds while signing and then changed the amount.” In a financial scenario, for example, if the question is asked, "During the period from 14:20 to 14:25, did the teller miss scanning any customer's documents?" the decoder locates the cache of the corresponding data block based on "14:20-14:25" and checks whether the core frame contains a token for document scanning actions. If there is no relevant high attention score token in the cache, the answer is: "No document scanning operation record was detected during this period." In a medical setting, for example, if the question is asked, "Did the elderly person take their blood pressure medication today?" the decoder retrieves high-attention tokens related to the medication bottle label and hand gestures from the cache based on the word "blood pressure medication". If a token for grasping the blood pressure medication bottle is found in the master frame, the answer is: "Yes, the elderly person took out their blood pressure medication and took it at 10:15". In some embodiments, the present invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for processing and understanding long video content, particularly suitable for enhancing the ability of existing video large language models to process ultra-long video sequences.

[0115] With the widespread adoption of high-definition video and streaming media, video data in our lives is growing at an unprecedented rate, and its duration is also increasing, ranging from short videos of a few minutes to movies or live recordings of several hours. To enable computers to "understand" these long videos like humans do, the Video Large Language Model (VideoLLM) has emerged.

[0116] VideoLarge Language Model (VideoLLM) is an advanced artificial intelligence model that not only possesses powerful language understanding capabilities similar to ChatGPT, but can also process visual information. Simply put, you can show it a video and then ask it a question in text; it can then provide an answer based on the video content.

[0117] A token is the basic unit of information that an AI model processes. For text, a word or character can be a token. For video, a single frame is broken down into many "visual tokens," representing different regions and features of the image. The longer the video, the more tokens are needed.

[0118] Key-value (KV) caching is a short-term memory mechanism within large models. To efficiently generate answers, the model stores previously computed information (i.e., the "keys" and "values" of tokens) in the KV cache. The longer the video, the more tokens need to be processed, and the space occupied by this short-term memory (i.e., video memory) expands rapidly.

[0119] Currently, VideoLLM faces a significant challenge in processing long videos: memory limitations. Just as the human brain cannot remember all the details of an entire movie at once, a computer's video memory cannot accommodate the massive number of tokens and huge key-value caches generated by long videos. To address this issue, traditional techniques primarily fall into two categories: 1. Length Extrapolation: This technique uses special training methods to try to get the model "accustomed" to longer inputs. However, this is only a temporary solution. It's like putting a slightly larger lid on a small bucket; when the video gets longer, the bucket (GPU memory) will still overflow and quickly reach its physical limits.

[0120] 2. Visual Token Compression: This method attempts to reduce the amount of information in a video before it enters the model. It mainly utilizes the "temporal redundancy" in the video.

[0121] Temporal redundancy refers to the phenomenon where adjacent frames in a video are highly similar in content. For example, in an interview shot from a fixed camera position, the background remains motionless for several seconds in a row; this is temporal redundancy.

[0122] Traditional methods discard or merge similar frames to reduce data volume. However, this compression method is rudimentary because it only considers visual repetition and ignores a more advanced form of redundancy—knowledge redundancy.

[0123] Knowledge redundancy refers to certain information in a video that, while visually changing, is not equally important for understanding the overall content. Models can use their learned "world knowledge" to determine which information is less important. For example, in a basketball video, the players' actions are key information, while a spectator's waving gesture might be considered knowledge redundancy.

[0124] In summary, existing technologies are either limited by hardware limitations or have low compression efficiency, resulting in poor performance when processing truly long videos. Therefore, there is an urgent need in this field for a smarter and more efficient video compression method to unleash the capabilities of VideoLLM for processing long videos.

[0125] The core objective of this invention is to provide an innovative, plug-and-play method for understanding long videos that requires no additional training. This method aims to address the memory bottleneck and low compression efficiency issues mentioned in the background section. By simultaneously eliminating "temporal redundancy" and "knowledge redundancy" in the video, it enables existing models to process ultra-long videos several times their original capacity limit without increasing hardware costs.

[0126] The core innovations of this invention are as follows: 1. Dual Redundancy Joint Compression Framework: Unlike traditional methods that only focus on temporal redundancy, this invention establishes a collaborative framework that intelligently "simplifies" video information in two steps. The first step is to remove visually repetitive frames before the video enters the model; the second step is to further compress secondary information in the model's internal "short-term memory" during model processing.

[0127] 2. Keyframe Selection Mechanism Based on Motion Peaks: This is the key to eliminating "temporal redundancy" in this invention. It mimics the human perceptual system; the brain naturally focuses on moments of "most dramatic motion change" when watching videos. Similarly, this invention identifies these frames as "pivot frames" by calculating the differences between adjacent frames and precisely capturing the peak points of these differences. These "pivot frames" are considered the most critical nodes in the video and must be preserved intact.

[0128] 3. Short-Term Memory Compression Mechanism Based on Model's Own Knowledge: This is the key to eliminating "knowledge redundancy" in this invention. It cleverly utilizes the attention scores generated by the model when processing videos. These scores can be seen as a reflection of how important the model considers the current information to be. This invention stipulates that the information in the "core frame" is sacred and inviolable in the model's "short-term memory" (KV cache) and must be 100% retained. For other "non-core frames," they are filtered based on their attention scores, retaining only the high-scoring, important information and discarding the low-scoring, less important information.

[0129] Through the above innovations, this invention provides a new, efficient, and universal solution for the field of long video understanding, greatly expanding the upper limit of the video processing time and the scope of application of existing AI models.

[0130] For example, the complete workflow of this invention will be illustrated through a specific example and an in-depth analysis of the details and formulas of each step.

[0131] Scenario: Suppose a VideoLLM is asked to watch a 5-minute animated film and answer, "What did the main character say at the end?". The model's original capability can only handle about 1 minute of video; directly inputting a 5-minute video will cause it to crash due to memory overflow.

[0132] The process after applying this invention is as follows: Step 1: Intelligent Filtering, Focusing on Key Elements (Eliminating Temporal Redundancy). The goal of this step is to perform a "coarse screening" before the video enters the large model, removing frames with duplicate content and identifying the key frames with the most information.

[0133] Input: Raw video file (5-minute animation). Processing procedure: 1. Video Preprocessing: The system first uses a vision encoder to convert each frame of the video into a mathematical form that the computer can understand. Specifically, each frame is cut into multiple small patches (e.g., 256), and each patch is then converted into a mathematical vector. Thus, a single frame becomes a set of 256 vectors. The entire video becomes a huge vector set, which this application refers to as M.

[0134] 2. Calculate inter-frame differences: Next, the system starts the keyframe selection mechanism to calculate the visual difference between each pair of adjacent frames.

[0135] ; di: This is a numerical value representing the overall difference between the i-th frame and the (i+1)-th frame. This application calculates a d value for each pair of adjacent frames in the video.

[0136] M: Represents the vector set obtained after preprocessing the entire video. It is a three-dimensional data structure, which can be understood as [number of frames, number of tokens per frame, vector dimension of each token].

[0137] Mi,j: represents the mathematical vector corresponding to the j-th visual token (i.e. the j-th tile) in the i-th frame of the video.

[0138] Mi+1,j: represents the mathematical vector corresponding to the j-th visual token at the same position in the (i+1)-th frame of the video.

[0139] N: Represents the total number of visual tokens into which a frame of an image is decomposed (e.g., 256).

[0140] cosA,B: This is the cosine similarity function, used to calculate the similarity of two vectors A and B in terms of direction. The result ranges from -1 to 1. If the two vectors are in the same direction, the value is 1 (most similar); if they are in opposite directions, the value is -1; if they are perpendicular, the value is 0.

[0141] 1 cos…: Converts “similarity” to “difference” or “distance”. If two frames are exactly the same, the cos value is 1, and the difference is 0. The greater the difference, the greater the difference in content between the two frames.

[0142] This is the summation symbol, representing a loop calculation. It calculates the token differences between all corresponding positions from the 1st to the Nth token between the i-th and i+1-th frames, and then adds them together.

[0143] 1 / N: The sum of the total differences is divided by the total number of tokens N, which is the average value. This prevents inconsistencies in the units of difference measurement due to different image resolutions (number of tokens).

[0144] Using this formula, this application ultimately obtains a difference sequence [d1, d2, d3, ...], which is like an "electrocardiogram" of the video, recording the drastic changes in the video content.

[0145] 3. Peak Location and Filtering: The system analyzes this difference sequence ("electrocardiogram") to find local peaks on the curve. The frames corresponding to these peaks indicate significant scene changes or object movement, and are therefore marked as "pivot frames." In addition to the "pivot frames," the system retains other frames with high difference scores and discards a large number of frames with extremely low difference scores and almost unchanged content.

[0146] Output: 1. A filtered, shorter sequence of keyframes. 2. A list of tags indicating which frames in the new sequence are the "keyframes".

[0147] Step 2: Watch and forget simultaneously to reduce the burden (eliminate knowledge redundancy): The goal of this step is to compress the "short-term memory" of the model in real time during the video processing, discarding secondary information and retaining only the core content.

[0148] Input: The keyframe sequence and the list of “core frame” markers generated in the first step.

[0149] Processing procedure: 1. Chunk input: The system divides the keyframe sequence into fixed-size data chunks, such as 32 frames at a time, and then inputs them sequentially into VideoLLM for processing.

[0150] 2. Processing and compressing memory: After processing each data block, the model creates a "short-term memory" in its internal key-value cache. At this point, the short-term memory compression mechanism immediately processes this newly added memory.

[0151] Token importance calculation: The system uses the attention matrix A inside the model to determine the importance of each visual token.

[0152] ; aj: is a numerical value representing the overall importance score of the j-th visual token in the current data block.

[0153] A is the attention matrix within the model. A_{k,i,j} represents the "attention level" of the i-th token to the j-th token in the k-th "attention head" (a calculation mechanism). The higher this value, the more important the information of the j-th token is to the model when understanding the i-th token.

[0154] h represents the total number of "attention heads" in the model. Modern models typically have multiple heads to capture information from different perspectives.

[0155] lq represents the total number of tokens in the current data block.

[0156] :: For a fixed token j, sum up the "attention" of all other tokens i towards it. This measures the degree to which token j is "collectively concerned" in the current context.

[0157] And 1 / h: Sum the “total attention” that token j receives from all h attention heads, then take the average to get the final importance score aj.

[0158] Protecting the core frame: The system checks if the token in the current data block comes from the "core frame" marked in the first step. If so, the system artificially sets its importance score aj to a maximum value (e.g., infinity). This ensures that no information from the "core frame" is discarded in the next step.

[0159] Perform "forget": The system sorts all tokens according to their importance scores, and then only retains the records of the tokens with the highest scores in a predetermined percentage (e.g., 90%) in the KV cache, while the records of the lower 10% of tokens are directly deleted from the KV cache.

[0160] 3. Repeatedly: Next, the model processes the second data block, performs the above memory compression again, and then the third, the fourth... until all data blocks are processed.

[0161] Output: A highly compressed final key-value cache that retains all key information.

[0162] Step 3: Forming a final understanding and answering: This step is the end of the entire process. The model will use the carefully prepared information from the first two steps to generate a final answer to the user's question.

[0163] Input: The video's "memory summary," which is the highly compressed key video information finally stored in the KV cache after the second step of processing. This represents the model's comprehensive understanding of the entire video's visual content. User's text question: The specific question input by the user that the model needs to answer, such as: "What did the main character say at the end?"

[0164] At this point, the language decoder part of VideoLLM, which is specifically responsible for generating text, begins to work. It receives the two core inputs mentioned above.

[0165] The language decoder performs a joint reasoning process. It doesn't view the video summary or user question in isolation. Instead, it treats the user's question as an "instruction" or "clue," guiding it to "refer to" the condensed video "memory summary." For example, the word "last" in the question will guide the model to focus on the later timestamps in the "memory summary." Based on a comprehensive understanding of the question and related video content, the decoder begins generating text tokens (i.e., constructing the answer word by word). It organizes the language into a fluent, coherent, and accurate sentence that directly answers the user's question. Output: A text-based answer: A direct response to the user's question generated by the model.

[0166] Throughout the process, this invention uses two-step intelligent compression to transform an originally unprocessable ultra-long video task into a controllable task within the capabilities of existing hardware. It doesn't brute-force discard any information; instead, like an efficient learner, it understands how to "grasp the key points" and "selectively memorize," thus achieving efficient and accurate long video comprehension.

[0167] like Figure 2 As shown, Figure 2 This is a schematic diagram of a video understanding device provided in an embodiment of this application. The video understanding device is used to execute the aforementioned video understanding method. The video understanding device can be configured on a terminal or a server.

[0168] like Figure 2 As shown, the video understanding device 200 includes...

[0169] The video processing unit 201 is used to preprocess the original video to be understood, converting the multi-frame images corresponding to the original video into feature vectors. The difference calculation unit 202 is used to calculate the feature differences between adjacent frame images, select key frame images from multiple frame images based on the feature differences, and form a key frame sequence based on the key frame images. The model input unit 203 is used to divide the keyframe sequence into multiple data blocks and input them into the pre-trained video big language model. When the video big language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information, filters the target visual token from the multiple visual tokens according to the importance information, and outputs the compressed video feature cache. The understanding generation unit 204 is used to obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

[0170] In some embodiments, the preprocessing of the original video to be understood, which converts the multi-frame images corresponding to the original video into feature vectors, includes: processing each frame image of the original video through a preset visual encoder, dividing each frame image into a preset number of patches; extracting features from each patch and converting it into a corresponding mathematical vector to form a vector set containing a three-dimensional data structure including the number of frames, the number of tokens per frame, and the dimension of the token vector.

[0171] In some embodiments, calculating the feature difference between adjacent frame images includes: for each pair of adjacent frame images in the video, calculating the difference value between the corresponding vectors of the same position patches in the adjacent frame images; summing the difference values ​​of all corresponding position patches and taking the average value to obtain the overall difference degree of adjacent frame images, forming a difference degree sequence that characterizes the degree of drastic change in video content as the feature difference.

[0172] In some embodiments, the step of selecting key frame images from multiple frames based on feature differences and forming a key frame sequence based on the key frame images includes: analyzing the difference sequence corresponding to the feature differences and obtaining the corresponding local peak points; marking the frames corresponding to the local peak points as the core frames, while retaining non-peak frames with high difference scores and discarding frames with extremely low difference scores, forming the key frame sequence containing the core frame and other high difference frames, and generating a label list to indicate the core frame in the key frame sequence.

[0173] In some embodiments, determining the importance information of the visual token corresponding to each data block based on the generated attention information, filtering target visual tokens from multiple visual tokens according to the importance information, and outputting a compressed video feature cache includes: using the attention matrix generated when processing the data block using a video large language model, calculating the total attention of each visual token among all attention heads and taking the average to obtain the importance score of the visual token; if the visual token comes from the core frame, setting its corresponding importance score to an extremely high value; sorting the tokens according to the importance score to form a compressed video feature cache.

[0174] In some embodiments, dividing the keyframe sequence into multiple data blocks includes: dividing the keyframe sequence into multiple data blocks sequentially according to a preset fixed number of frames, each data block containing the same number of frames, so as to input the video large language model in sequence.

[0175] In some embodiments, generating video understanding information through joint reasoning based on the compressed video feature cache and the question information includes: receiving the compressed video feature cache and the user's question information using a language decoder in a video large language model; using the question information as reasoning clues to guide the language decoder to locate relevant information in the video feature cache; and generating text-based answer information word by word or phrase by phrase based on the question information and relevant information to form video understanding information.

[0176] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0177] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 3 It runs on the computer device shown.

[0178] Please see Figure 3 , Figure 3 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server. See also... Figure 3 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0179] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any video understanding method.

[0180] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0181] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any video understanding method.

[0182] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0184] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: The original video to be understood is preprocessed by converting the multi-frame images corresponding to the original video into feature vectors. Calculate the feature differences between adjacent frames, select keyframe images from multiple frames based on the feature differences, and form a keyframe sequence based on the keyframe images; The keyframe sequence is divided into multiple data blocks and input into a pre-trained video big language model. When the video big language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information. Based on the importance information, the target visual token is selected from the multiple visual tokens, and the compressed video feature cache is output. Obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

[0185] In some embodiments, the preprocessing of the original video to be understood, which converts the multi-frame images corresponding to the original video into feature vectors, includes: processing each frame image of the original video through a preset visual encoder, dividing each frame image into a preset number of patches; extracting features from each patch and converting it into a corresponding mathematical vector to form a vector set containing a three-dimensional data structure including the number of frames, the number of tokens per frame, and the dimension of the token vector.

[0186] In some embodiments, calculating the feature difference between adjacent frame images includes: for each pair of adjacent frame images in the video, calculating the difference value between the corresponding vectors of the same position patches in the adjacent frame images; summing the difference values ​​of all corresponding position patches and taking the average value to obtain the overall difference degree of adjacent frame images, forming a difference degree sequence that characterizes the degree of drastic change in video content as the feature difference.

[0187] In some embodiments, the step of selecting key frame images from multiple frames based on feature differences and forming a key frame sequence based on the key frame images includes: analyzing the difference sequence corresponding to the feature differences and obtaining the corresponding local peak points; marking the frames corresponding to the local peak points as the core frames, while retaining non-peak frames with high difference scores and discarding frames with extremely low difference scores, forming the key frame sequence containing the core frame and other high difference frames, and generating a label list to indicate the core frame in the key frame sequence.

[0188] In some embodiments, determining the importance information of the visual token corresponding to each data block based on the generated attention information, filtering target visual tokens from multiple visual tokens according to the importance information, and outputting a compressed video feature cache includes: using the attention matrix generated when processing the data block using a video large language model, calculating the total attention of each visual token among all attention heads and taking the average to obtain the importance score of the visual token; if the visual token comes from the core frame, setting its corresponding importance score to an extremely high value; sorting the tokens according to the importance score to form a compressed video feature cache.

[0189] In some embodiments, dividing the keyframe sequence into multiple data blocks includes: dividing the keyframe sequence into multiple data blocks sequentially according to a preset fixed number of frames, each data block containing the same number of frames, so as to input the video large language model in sequence.

[0190] In some embodiments, generating video understanding information through joint reasoning based on the compressed video feature cache and the question information includes: receiving the compressed video feature cache and the user's question information using a language decoder in a video large language model; using the question information as reasoning clues to guide the language decoder to locate relevant information in the video feature cache; and generating text-based answer information word by word or phrase by phrase based on the question information and relevant information to form video understanding information.

[0191] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the video understanding methods provided in the embodiments of this application.

[0192] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0193] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video understanding method, characterized in that, include: The original video to be understood is preprocessed by converting the multi-frame images corresponding to the original video into feature vectors. Calculate the feature differences between adjacent frames, select keyframe images from multiple frames based on the feature differences, and form a keyframe sequence based on the keyframe images; The keyframe sequence is divided into multiple data blocks and input into a pre-trained video big language model. When the video big language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information. Based on the importance information, the target visual token is selected from the multiple visual tokens, and the compressed video feature cache is output. Obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

2. The method according to claim 1, characterized in that, The preprocessing of the original video to be understood, which converts the multi-frame images corresponding to the original video into feature vectors, includes: Each frame of the original video is processed by a preset visual encoder, and each frame is divided into a preset number of tiles. For each tile, features are extracted and converted into corresponding mathematical vectors, forming a vector set with a three-dimensional data structure containing the number of frames, the number of tokens per frame, and the dimension of the token vector.

3. The method according to claim 2, characterized in that, The calculation of feature differences between adjacent frames includes: For each pair of adjacent frame images in the video, calculate the difference between the vectors corresponding to the same position patches in the adjacent frame images; The summation of the difference values ​​of all corresponding positional blocks is averaged to obtain the overall difference degree of adjacent frame images, forming a difference degree sequence that characterizes the degree of drastic change in video content as the feature difference.

4. The method according to claim 1, characterized in that, The step of selecting keyframe images from multiple frames based on feature differences and forming a keyframe sequence based on the keyframe images includes: Analyze the difference sequence corresponding to the feature differences and obtain the corresponding local peak points; The frames corresponding to local peak points are marked as the core frames, while non-peak frames with high difference scores are retained and frames with extremely low difference scores are discarded, forming the key frame sequence containing the core frames and other high difference frames, and generating a tag list to indicate the core frames in the key frame sequence.

5. The method according to claim 4, characterized in that, The process of determining the importance information of the visual token corresponding to each data block based on the generated attention information, filtering the target visual token from multiple visual tokens according to the importance information, and outputting the compressed video feature cache includes: The attention matrix generated when processing data blocks using the video large language model is used to calculate the total attention of each visual token among all attention heads and take the average to obtain the importance score of the visual token. If the visual token comes from the core frame, its corresponding importance score is set to an extremely high value; The tokens are sorted according to their importance scores to form a compressed video feature cache.

6. The method according to claim 1, characterized in that, The process of dividing the keyframe sequence into multiple data blocks includes: The keyframe sequence is divided into multiple data blocks according to a preset fixed number of frames. Each data block contains the same number of frames, which are then input into the video large language model in sequence.

7. The method according to claim 1, characterized in that, The step of generating video understanding information through joint reasoning based on the compressed video feature cache and the question information includes: The language decoder in the video large language model is used to receive the compressed video feature cache and the user's question information. The question information is used as a clue for reasoning, guiding the language decoder to locate relevant information in the video feature cache; Based on the question and related information, textual responses are generated word by word or phrase by phrase to form video comprehension information.

8. A video understanding device, characterized in that, The method applied to any one of claims 1-7 includes: The video processing unit is used to preprocess the original video to be understood, converting the multi-frame images corresponding to the original video into feature vectors. The difference calculation unit is used to calculate the feature differences between adjacent frame images, select key frame images from multiple frames based on the feature differences, and form a key frame sequence based on the key frame images. The model input unit is used to divide the keyframe sequence into multiple data blocks and input them into the pre-trained video big language model. When the video big language model processes each data block, it determines the importance information of the visual token corresponding to each data block based on the generated attention information, filters the target visual token from the multiple visual tokens according to the importance information, and outputs the compressed video feature cache. The understanding generation unit is used to obtain the question information corresponding to the original video, and generate video understanding information through joint reasoning based on the compressed video feature cache and the question information.

9. A computer device, characterized in that, The server includes a processor, a memory, and a computer program stored in the memory and executable by the processor, the memory storing a policy model, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1-7.

Images