Visual token compression method and device, computer equipment and storage medium

Abstract

The embodiment of the invention belongs to the technical field of artificial intelligence, and relates to a visual token compression method and device, computer equipment and a storage medium, and the method comprises the steps: receiving a visual token compression request which is sent by a user terminal and carries an original video frame; performing key frame screening operation on the original video frame to obtain a key video frame; inputting the key video frame into a visual language model to generate a visual token; performing visual token screening operation on the visual token according to the attention of the text instruction to the visual area and the content importance of the visual area to obtain a target visual token; and outputting the target visual token to the user terminal. The method can be used for carrying out related video processing in service systems such as medical treatment, health and pension, and can improve the processing efficiency, accurately extract key information and enhance the interactivity.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a visual token compression method, apparatus, computer device, and storage medium. Background Technology

[0002] In the medical field, with the development of digital technology, a large amount of medical video data has been generated, such as surgical videos and medical examination videos (e.g., ultrasound and endoscopic videos). This video data contains rich medical information, but directly processing and analyzing this raw video data presents many challenges. On the one hand, the raw video data volume is enormous, resulting in high transmission and storage costs; on the other hand, the videos contain a large amount of redundant information, and direct analysis would reduce processing efficiency and make it difficult to quickly locate key medical information.

[0003] Currently, although there are some video processing technologies, most lack targeted optimization for medical scenarios and cannot adequately meet the accuracy and efficiency requirements of medical services for extracting key information. Summary of the Invention

[0004] The purpose of this application is to provide a visual token compression method, apparatus, computer device, and storage medium to solve the problem that existing video processing technologies cannot meet the requirements for accuracy and efficiency in extracting key information.

[0005] To address the aforementioned technical problems, this application provides a visual token compression method, employing the following technical solution:

[0006] Accept visual token compression requests sent by user terminals, which carry the original video frames;

[0007] Perform a keyframe filtering operation on the original video frames to obtain key video frames;

[0008] The key video frames are input into a visual language model to generate visual tokens;

[0009] The visual tokens are filtered based on the attention given to the visual region by the text instructions and the importance of the content of the visual region itself, to obtain the target visual tokens;

[0010] The target visual token is output to the user terminal.

[0011] Furthermore, the step of performing keyframe filtering on the original video frames to obtain key video frames specifically includes the following steps:

[0012] Calculate the pixel information entropy index and inter-frame pixel difference index of the original video frame;

[0013] Based on the pixel information entropy index and the inter-frame pixel difference index, the original video frames are subjected to an original frame filtering operation to obtain candidate video frames.

[0014] The candidate video frames are input into a visual encoder for semantic feature transformation to obtain semantic feature vectors;

[0015] The candidate video frames are filtered based on the semantic feature vectors to obtain the key video frames.

[0016] Furthermore, the step of performing original frame filtering on the original video frames based on the pixel information entropy index and the inter-frame pixel difference index to obtain candidate video frames specifically includes the following steps:

[0017] The pixel information entropy index and the inter-frame pixel difference index are fused to obtain a preliminary information score for each original video frame.

[0018] The original video frames are divided into subsets according to a preset time period.

[0019] Based on the preliminary information scores, preliminary screening operations are performed on the original video frame subsets to obtain the candidate video frames.

[0020] Furthermore, the step of performing candidate frame filtering on the candidate video frames based on the semantic feature vector to obtain the key video frames specifically includes the following steps:

[0021] The semantic novelty score of each candidate video frame relative to the previous frame is calculated based on the semantic feature vector.

[0022] The semantic motion intensity score of each candidate video frame relative to the previous frame is calculated based on the semantic feature vector.

[0023] Calculate the image-text relevance score of each candidate video frame relative to the previous frame;

[0024] A first score fusion operation is performed on the semantic novelty score, the semantic motion intensity score, and the image-text relevance score to obtain the first fusion score for each candidate video frame;

[0025] The candidate video frames are then subjected to a final screening operation based on the first fusion score to obtain the key video frames.

[0026] Furthermore, the step of filtering the visual tokens based on the attention given to the visual region by the text instruction and the importance of the content of the visual region itself to obtain the target visual token specifically includes the following steps:

[0027] The key video frames are input into the visual language model to obtain the initial visual token;

[0028] Calculate the attention score for each of the initial visual tokens;

[0029] The initial visual token is filtered based on the attention score to obtain the target visual token.

[0030] Furthermore, the step of performing a visual token filtering operation on the initial visual token based on the attention score to obtain the target visual token specifically includes the following steps:

[0031] The initial visual tokens are classified according to the attention scores to obtain key visual tokens and non-key visual tokens.

[0032] The semantic novelty score corresponding to each of the initial visual tokens is used as the source frame importance score;

[0033] A second score fusion operation is performed on the attention score and the source frame importance score to obtain a second fused score;

[0034] Based on the second fusion score, a non-critical visual token filtering operation is performed on the non-critical visual tokens to obtain the optimal visual tokens;

[0035] The key visual token and the preferred visual token are used as the target visual token.

[0036] To address the aforementioned technical problems, this application also provides a visual token compression device, which employs the following technical solution:

[0037] The request receiving module is used to accept visual token compression requests sent by user terminals, which carry the original video frames.

[0038] The keyframe filtering module is used to perform keyframe filtering operations on the original video frames to obtain key video frames.

[0039] The visual token generation module is used to input the key video frames into the visual language model to generate visual tokens;

[0040] The visual token filtering module is used to perform visual token filtering operations on the visual tokens based on the attention given to the visual area by the text command and the importance of the content of the visual area itself, so as to obtain the target visual token.

[0041] The visual token output module is used to output the target visual token to the user terminal.

[0042] Furthermore, the keyframe filtering module includes:

[0043] The index calculation submodule is used to calculate the pixel information entropy index and inter-frame pixel difference index of the original video frame.

[0044] The original frame filtering submodule is used to perform original frame filtering operation on the original video frames according to the pixel information entropy index and the inter-frame pixel difference index to obtain candidate video frames.

[0045] The semantic feature transformation submodule is used to input the candidate video frame into the visual encoder to perform semantic feature transformation operations to obtain a semantic feature vector.

[0046] The candidate frame filtering submodule is used to perform candidate frame filtering operations on the candidate video frames based on the semantic feature vector to obtain the key video frames.

[0047] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0048] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the visual token compression method as described above.

[0049] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0050] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the visual token compression method described above.

[0051] This application provides a visual token compression method, comprising: receiving a visual token compression request carrying original video frames sent by a user terminal; performing a keyframe filtering operation on the original video frames to obtain key video frames; inputting the key video frames into a visual language model to generate visual tokens; performing a visual token filtering operation on the visual tokens based on the attention given to visual regions by text instructions and the importance of the content of the visual regions themselves to obtain target visual tokens; and outputting the target visual tokens to the user terminal. Compared with the prior art, this application significantly reduces the amount of data to be processed by keyframe filtering and visual token filtering, improves the processing efficiency of medical video data, and saves computing resources and time costs; combining medical features and text instructions for keyframe and visual token filtering can more accurately extract key medical information in medical videos, providing more valuable support for medical diagnosis and teaching; and guiding the visual token filtering process by inputting text instructions realizes human-computer interaction, improving the flexibility and practicality of the system. Attached Figure Description

[0052] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0054] Figure 2 This is a flowchart illustrating the implementation of the visual token compression method provided in this application embodiment;

[0055] Figure 3 This is a schematic diagram of the structure of the visual token compression device provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0061] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0062] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, and desktop computer, etc.

[0063] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0064] It should be noted that the visual token compression method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the visual token compression device is generally set in the server / terminal device.

[0065] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0066] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the visual token compression method according to this application. The visual token compression method includes steps S201, S202, S203, S204, and S205.

[0067] In step S201, a visual token compression request carrying the original video frame is accepted from the user terminal.

[0068] In the embodiments of this application, the user terminal refers to a terminal device used to execute the visual token compression method provided in this application. The user terminal may be a mobile terminal such as a mobile phone, smartphone, laptop, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), navigation device, etc., as well as a fixed terminal such as a digital TV, desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not intended to limit this application.

[0069] In this embodiment, the user inputs a raw video frame through their terminal device (such as a mobile phone, computer, etc.), and this input is received by the system. This raw video frame is the specific content that the user wants the system to perform visual token compression. Specifically, the raw video frame can be medical image data related to a medical scenario. For example, it could be data such as endoscopic examination or minimally invasive surgery recordings. It should be understood that the examples of raw video frames here are for convenience of understanding only and are not intended to limit this application.

[0070] In step S202, a keyframe filtering operation is performed on the original video frames to obtain key video frames.

[0071] In this embodiment, the keyframe filtering module in the server employs a keyframe filtering algorithm based on the movement trajectory of surgical instruments and changes in the morphology of heart tissue. By analyzing the position and movement trajectory of surgical instruments in video frames, as well as changes in the morphology of heart tissue (such as the opening and closing of heart valves and the contraction of myocardium), video frames corresponding to key surgical procedures are filtered out, such as key moments in heart valve repair and key frames in vascular anastomosis, thus obtaining key video frames.

[0072] In step S203, key video frames are input into the visual language model to generate visual tokens.

[0073] In this embodiment, the selected key video frames are input into a pre-trained visual language model. This model, trained on a large number of cardiac surgery images and related surgical text descriptions, is capable of converting key video frames into visual tokens with semantic information. For example, key frames related to heart valve repair can be converted into visual tokens representing semantics such as "heart valve repair operation" and "normal valve morphology."

[0074] In step S204, a visual token filtering operation is performed on the visual token based on the attention given to the visual region by the text instruction and the importance of the content of the visual region itself, to obtain the target visual token.

[0075] In this embodiment, the doctor inputs a text command on the workstation, such as "Pay attention to valve morphological changes during the heart valve repair process." The visual token filtering module on the server analyzes the correlation between the visual token and the text command, as well as the importance of the content in the region represented by the visual token (the heart valve region). Visual tokens that are relevant to and of high importance in relation to heart valve morphological changes are selected to obtain the target visual token.

[0076] In step S205, the target visual token is output to the user terminal.

[0077] In this embodiment, the server outputs the target visual tokens to the doctor's workstation. The workstation can then display these target visual tokens to the doctor in an intuitive way, such as displaying the key visual token sequence of valve morphology changes in chart form, helping the doctor quickly review and analyze key information during the surgery.

[0078] This application provides a visual token compression method, comprising: receiving a visual token compression request carrying original video frames sent by a user terminal; performing keyframe filtering on the original video frames to obtain key video frames; inputting the key video frames into a visual language model to generate visual tokens; performing visual token filtering on the visual tokens based on the attention given to the visual region by the text instruction and the importance of the content of the visual region itself to obtain target visual tokens; and outputting the target visual tokens to the user terminal. Compared with the prior art, this application significantly reduces the amount of data to be processed by keyframe filtering and visual token filtering, improves the processing efficiency of medical video data, and saves computing resources and time costs; combining medical features and text instructions for keyframe and visual token filtering can more accurately extract key medical information in medical videos, providing more valuable support for medical diagnosis and teaching; and guiding the visual token filtering process by inputting text instructions realizes human-computer interaction, improving the flexibility and practicality of the system.

[0079] In some optional implementations of the embodiments of this application, the step of performing keyframe filtering on the original video frames to obtain key video frames specifically includes the following steps:

[0080] Calculate the pixel information entropy index and inter-frame pixel difference index of the original video frames;

[0081] The original video frames are filtered based on the pixel information entropy index and the inter-frame pixel difference index to obtain candidate video frames.

[0082] The candidate video frames are input into the visual encoder for semantic feature transformation to obtain semantic feature vectors;

[0083] Candidate video frames are filtered based on semantic feature vectors to obtain key video frames.

[0084] In this embodiment of the application, the application processes all input video frames (denoted as F). initial The quantity is N large ), calculate two low-cost metrics that do not require a depth visual encoder:

[0085] (1) Pixel information entropy H p (f iInformation entropy measures the richness of texture and detail in a single frame of an image. High information entropy usually indicates a more complex scene. The information entropy of an image can be calculated using the following formula:

[0086]

[0087] Where p(k) is the frame f i The probability that a pixel value is k.

[0088] (2) Inter-frame pixel difference D p (f i ,f i-1 The following formula can be used to measure the content changes and motion amplitude between adjacent frames:

[0089] D p (f i ,f i-1 )=‖vec(g i )-vec(g i-1 )‖2

[0090] Among them, vec(g i ) is the flattened pixel vector after grayscale conversion of the i-th frame.

[0091] In this embodiment, the application performs a raw frame filtering operation on the raw video frames to select a medium-sized candidate set F with higher scores. candidate (Quantity is N) medium (e.g., 128 frames).

[0092] In this embodiment of the application, the application considers candidate video frames F candidate The frames in the image are fed into the model's visual encoder to extract their high-dimensional semantic feature vectors v. i This reduces computational and memory overhead.

[0093] In this embodiment, the application performs a candidate frame filtering operation on candidate video frames to ultimately select a set F of key frames that possess both content importance and temporal representativeness. final (e.g., 64 frames).

[0094] Compared with existing technologies, this application aims to quickly filter out a large number of obviously redundant frames through key frame filtering operations, thereby reducing the burden of subsequent high-cost calculations.

[0095] In some optional implementations of the embodiments of this application, the step of performing original frame filtering operation on the original video frames based on the pixel information entropy index and the inter-frame pixel difference index to obtain candidate video frames specifically includes the following steps:

[0096] A fusion operation is performed on the pixel information entropy index and the inter-frame pixel difference index to obtain the preliminary information score of each original video frame.

[0097] The original video frames are divided into subsets according to a preset time period.

[0098] Based on the preliminary information scores, preliminary screening operations are performed on the original video frame subsets to obtain candidate video frames.

[0099] In this embodiment of the application, after calculating the above-mentioned element information entropy index and inter-frame pixel difference index, the above-mentioned indexes are weighted and fused to obtain f for each frame. i Calculate a preliminary information score. pre (f i Subsequently, the video was divided into multiple macro-level time periods, each allocated a K-frame budget, and the highest-scoring K-frame was selected from each period.

[0100] Compared with existing technologies, this application can screen out a medium-sized candidate set F with higher scores. candidate (Quantity is N) medium (e.g., 128 frames).

[0101] In some optional implementations of the embodiments of this application, the step of performing candidate frame filtering operation on candidate video frames based on semantic feature vectors to obtain key video frames specifically includes the following steps:

[0102] The semantic novelty score of each candidate video frame relative to the previous frame is calculated based on the semantic feature vector.

[0103] The semantic motion intensity score of each candidate video frame relative to the previous frame is calculated based on the semantic feature vector.

[0104] Calculate the image-text relevance score of each candidate video frame relative to the previous frame;

[0105] A first score fusion operation is performed on the semantic novelty score, semantic motion intensity score, and image-text relevance score to obtain the first fusion score for each candidate video frame.

[0106] The candidate video frames are then filtered based on the first fusion score to obtain the key video frames.

[0107] In this embodiment, the semantic novelty score is calculated based on semantic feature vectors to determine the degree of semantic change in each frame relative to the previous frame. This score more accurately captures the actual changes in content. The calculation formula is as follows:

[0108]

[0109] Subsequently, the semantic novelty score is weighted and fused with the semantic motion intensity and image-text relevance scores to form the final retention score. final (f i ).

[0110] Then, the candidate set F was tested again. candidate Based on its final retained score final Perform the screening.

[0111] Compared with existing technologies, this application can filter out a set of keyframes (e.g., 64 frames) that are both important in terms of content and representative in terms of time.

[0112] In some optional implementations of the embodiments of this application, the step of performing a visual token filtering operation on the visual token based on the attention given to the visual region by the text instruction and the importance of the content of the visual region itself to obtain the target visual token specifically includes the following steps:

[0113] Key video frames are input into the visual language model to obtain the initial visual token;

[0114] Calculate the attention score for each initial visual token;

[0115] The initial visual token is filtered based on the attention score to obtain the target visual token.

[0116] In this embodiment, for a visual language model, each frame typically generates hundreds of visual tokens. Retaining all of them would severely impact the efficiency of the generation stage and consume a large amount of GPU memory. Therefore, this application designs a token compression strategy. The core idea of ​​this strategy is to calculate a comprehensive score for each visual token based on the attention given to the visual region by the text instruction and the importance of the content of the visual region itself, and then filter them accordingly.

[0117] In this embodiment of the application, the attention score (A) of this application score This aims to measure the degree of attention a text instruction gives to each visual token. Attention scores are accumulated across multiple key attention layers (l∈L) of the model, representing the attention given to a single visual token t by all text token queries. j The key (Key,k) j The attention weight generated by the visual token. The higher the score, the more relevant the image region represented by the visual token is to the question. The calculation formula is as follows:

[0118]

[0119] Among them, w lThese are preset hierarchical weights used to emphasize that attention to certain layers (such as the middle and later layers) is more focused on higher-level semantic information.

[0120] Compared with existing technologies, this application can ensure that the core visual details most directly related to text instructions are not lost.

[0121] In some optional implementations of the embodiments of this application, the step of performing visual token filtering operation on the initial visual token based on the attention score to obtain the target visual token specifically includes the following steps:

[0122] The initial visual tokens are classified based on the attention scores to obtain key visual tokens and non-key visual tokens.

[0123] The semantic novelty score corresponding to each initial visual token is used as the source frame importance score;

[0124] A second score fusion operation is performed on the attention score and the source frame importance score to obtain the second fused score;

[0125] Based on the second fusion score, non-critical visual tokens are filtered to obtain the optimal visual tokens.

[0126] Use the key visual token and the preferred visual token as the target visual token.

[0127] In this embodiment of the application, in order to ensure that the core visual details most directly related to the text instructions are not lost, the system will unconditionally retain a portion of the attention score A. score The highest-scoring tokens. For example, directly retain the visual tokens that rank in the top 20% by score.

[0128] In this embodiment, after filtering out the core visual details, the system calculates a fusion score for the remaining visual tokens. hybrid (t j This score incorporates the attention score (A). score ) and source frame importance score (F score This allows for a more comprehensive assessment of the token's value.

[0129] In this embodiment of the application, the importance of the source frame (F) score ) represents the visual token t j The original frame f(t) j The score is directly inherited from the semantic novelty score calculated in Phase 1 (multi-level video frame selection). semantic(f(t j A token from a keyframe that is novel and informative is inherently more important.

[0130] In this embodiment of the application, the above-mentioned fusion score is... hybrid (t j The calculation formula is as follows:

[0131] Score hybrid (t j )=α·A score (t j )+(1-α)·F score (t j )

[0132] The hyperparameter α is used to balance the weights between immediate attention and inherent importance. The system sorts the remaining tokens based on this fusion score and selects the best subset (e.g., 30%) to supplement the final retention quota.

[0133] Compared to existing technologies, the number of visual tokens can be efficiently compressed to about 30% to 40% of the original number. This not only significantly reduces the GPU memory and computational requirements of the subsequent generation stage, improving inference speed, but also ensures the accuracy of model inference by retaining the tokens that are most relevant to the task and have the richest information.

[0134] In summary, the visual token compression method provided in this application has the following advantages:

[0135] 1. Intelligent frame selection balancing efficiency and effectiveness: Phase 1 addresses the dilemma of "blindness" and "high cost" through a two-step strategy of "low-cost feature initial screening" and "high-cost semantic selection." It first quickly eliminates a massive number of obviously irrelevant frames at extremely low cost, then precisely selects a small number of high-potential candidate frames. This ensures that the final set of frames fed into the model covers the key information of the video with the highest probability while keeping the overhead of video frame processing within a reasonable range.

[0136] 2. In medical video scenarios such as endoscopy or X-ray imaging, since there is often a lot of redundant information in video frames, by dynamically filtering visual tokens and retaining only the visual information that is crucial to diagnosis and analysis, the computational resources and video memory consumption during subsequent processing can be greatly reduced.

[0137] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0138] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0139] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0140] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0141] Further reference Figure 3 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a visual token compression device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0142] like Figure 3 As shown, the visual token compression device 200 of this application embodiment includes:

[0143] The request receiving module 210 is used to accept a visual token compression request sent by the user terminal, which carries the original video frame.

[0144] The keyframe filtering module 220 is used to perform keyframe filtering operations on the original video frames to obtain key video frames.

[0145] The visual token generation module 230 is used to input key video frames into the visual language model to generate visual tokens;

[0146] The visual token filtering module 240 is used to perform visual token filtering operations based on the attention given to the visual area by the text command and the importance of the content of the visual area itself, so as to obtain the target visual token.

[0147] The visual token output module 250 is used to output the target visual token to the user terminal.

[0148] In this embodiment of the application, a visual token compression device 200 is provided, comprising: a request receiving module 210, configured to receive a visual token compression request carrying original video frames sent by a user terminal; a key frame filtering module 220, configured to perform key frame filtering operations on the original video frames to obtain key video frames; a visual token generation module 230, configured to input the key video frames into a visual language model to generate visual tokens; a visual token filtering module 240, configured to perform visual token filtering operations on visual tokens based on the attention of the visual region by the text instruction and the importance of the content of the visual region itself to obtain target visual tokens; and a visual token output module 250, configured to output the target visual tokens to the user terminal. Compared with existing technologies, this application significantly reduces the amount of data to be processed and improves the processing efficiency of medical video data by using keyframe filtering and visual token filtering, saving computing resources and time costs. Combining medical features and text commands for keyframe and visual token filtering can more accurately extract key medical information from medical videos, providing more valuable support for medical diagnosis and teaching. By guiding the visual token filtering process with input text commands, human-computer interaction is realized, improving the system's flexibility and practicality.

[0149] In some optional implementations of the embodiments of this application, the keyframe filtering module includes:

[0150] The index calculation submodule is used to calculate the pixel information entropy index and inter-frame pixel difference index of the original video frame.

[0151] The raw frame filtering submodule is used to perform raw frame filtering operations on the raw video frames based on the pixel information entropy index and the inter-frame pixel difference index to obtain candidate video frames.

[0152] The semantic feature transformation submodule is used to input candidate video frames into the visual encoder to perform semantic feature transformation operations and obtain semantic feature vectors.

[0153] The candidate frame filtering submodule is used to perform candidate frame filtering operations on candidate video frames based on semantic feature vectors to obtain key video frames.

[0154] In some optional implementations of the embodiments of this application, the above-mentioned original frame filtering submodule includes:

[0155] The index fusion unit is used to perform index fusion operations on the pixel information entropy index and the inter-frame pixel difference index to obtain the preliminary information score of each original video frame.

[0156] The video frame division unit is used to divide the original video frames according to a preset time period to obtain a subset of the original video frames;

[0157] The preliminary screening unit is used to perform preliminary screening operations on the original video frame subsets based on the preliminary information scores to obtain candidate video frames.

[0158] In some optional implementations of the embodiments of this application, the above-mentioned candidate frame filtering submodule includes:

[0159] The novelty score calculation unit is used to calculate the semantic novelty score of each candidate video frame relative to the previous frame based on the semantic feature vector.

[0160] The motion intensity score calculation unit is used to calculate the semantic motion intensity score of each candidate video frame relative to the previous frame based on the semantic feature vector.

[0161] The correlation score calculation unit is used to calculate the image-text correlation score of each candidate video frame relative to the previous frame.

[0162] The first score fusion unit is used to perform a first score fusion operation on the semantic novelty score, semantic motion intensity score and image-text relevance score to obtain the first fusion score of each candidate video frame.

[0163] The final filtering unit is used to perform a final filtering operation on the candidate video frames based on the first fusion score to obtain the key video frames.

[0164] In some optional implementations of the embodiments of this application, the above-mentioned visual token filtering module includes:

[0165] The visual token generation submodule is used to input key video frames into the visual language model to obtain the initial visual token;

[0166] The attention score calculation submodule is used to calculate the attention score for each initial visual token.

[0167] The visual token filtering submodule is used to perform visual token filtering operations on the initial visual tokens based on the attention score to obtain the target visual tokens.

[0168] In some optional implementations of the embodiments of this application, the above-mentioned visual screening submodule includes:

[0169] The visual token classification unit is used to classify the initial visual tokens according to the attention score to obtain key visual tokens and non-key visual tokens;

[0170] The importance score acquisition unit is used to take the semantic novelty score corresponding to each initial visual token as the importance score of the source frame;

[0171] The second score fusion unit is used to perform a second score fusion operation on the attention score and the source frame importance score to obtain the second fused score;

[0172] The non-critical visual token filtering unit is used to perform non-critical visual token filtering operation on non-critical visual tokens according to the second fusion score to obtain the optimal visual tokens.

[0173] The target visual token acquisition unit is used to obtain the key visual token and the preferred visual token as the target visual token.

[0174] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of this application.

[0175] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only the computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0176] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0177] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 310 may also include both internal storage units and external storage devices of the computer device 300. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for the visual token compression method. Furthermore, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0178] In some embodiments, the processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 320 is typically used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions stored in the memory 310 or to process data, for example, to execute computer-readable instructions of the visual token compression method.

[0179] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 300 and other electronic devices.

[0180] The computer equipment provided in this application significantly reduces the amount of data that needs to be processed by keyframe filtering and visual token filtering, improves the processing efficiency of medical video data, and saves computing resources and time costs. By combining medical features and text commands for keyframe and visual token filtering, key medical information in medical videos can be extracted more accurately, providing more valuable support for medical diagnosis and teaching. By guiding the visual token filtering process with input text commands, human-computer interaction is realized, improving the system's flexibility and practicality.

[0181] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the visual token compression method described above.

[0182] The computer-readable storage medium provided in this application significantly reduces the amount of data that needs to be processed by keyframe filtering and visual token filtering, thereby improving the processing efficiency of medical video data and saving computing resources and time costs. By combining medical features and text commands for keyframe and visual token filtering, key medical information in medical videos can be extracted more accurately, providing more valuable support for medical diagnosis and teaching. The process of visual token filtering is guided by inputting text commands, realizing human-computer interaction and improving the system's flexibility and practicality.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0184] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method of visual token compression, characterized by, The method comprises the following steps: receiving a visual token compression request sent by a user terminal and carrying an original video frame; performing key frame screening on the original video frame to obtain a key video frame; inputting the key video frame into a visual language model to generate a visual token; performing visual token screening on the visual token according to the attention of a visual area to a text instruction and the content importance of the visual area itself to obtain a target visual token; and outputting the target visual token to the user terminal.

2. The visual token compression method of claim 1, wherein, The step of performing key frame screening on the original video frame to obtain a key video frame comprises the following steps: calculating a pixel information entropy index and a frame pixel difference index of the original video frame; performing original frame screening on the original video frame according to the pixel information entropy index and the frame pixel difference index to obtain a candidate video frame; inputting the candidate video frame into a visual encoder to perform semantic feature conversion to obtain a semantic feature vector; performing candidate frame screening on the candidate video frame according to the semantic feature vector to obtain the key video frame.

3. The visual token compression method of claim 2, wherein, The step of performing original frame screening on the original video frame according to the pixel information entropy index and the frame pixel difference index to obtain a candidate video frame comprises the following steps: performing index fusion on the pixel information entropy index and the frame pixel difference index to obtain a preliminary information score of each original video frame; dividing the original video frame into original video frame subsets according to a preset time period; performing preliminary screening on the original video frame subsets according to the preliminary information score to obtain the candidate video frame.

4. The visual token compression method of claim 2, wherein, The step of performing candidate frame screening on the candidate video frame according to the semantic feature vector to obtain the key video frame comprises the following steps: calculating a semantic novelty score of each frame of the candidate video frame relative to a previous frame according to the semantic feature vector; calculating a semantic motion intensity score of each frame of the candidate video frame relative to a previous frame according to the semantic feature vector; calculating a text-image correlation score of each frame of the candidate video frame relative to a previous frame; performing first score fusion on the semantic novelty score, the semantic motion intensity score and the text-image correlation score to obtain a first fusion score of each candidate video frame; performing final screening on the candidate video frame according to the first fusion score to obtain the key video frame.

5. The visual token compression method of claim 1, wherein, The step of performing visual token screening on the visual token according to the attention of a visual area to a text instruction and the content importance of the visual area itself to obtain a target visual token comprises the following steps: inputting the key video frame into a visual language model to obtain an initial visual token; calculating an attention score of each initial visual token; performing visual token screening on the initial visual token according to the attention score to obtain the target visual token.

6. The visual token compression method of claim 5, wherein, The step of performing visual token screening operation on the initial visual token according to the attention score to obtain the target visual token specifically comprises the following steps: performing classification operation on the initial visual token according to the attention score to obtain key visual tokens and non-key visual tokens; taking the semantic novelty score corresponding to each initial visual token as a source frame importance score; performing second score fusion operation on the attention score and the source frame importance score to obtain a second fusion score; performing non-key visual token screening operation on the non-key visual token according to the second fusion score to obtain an optimal visual token; taking the key visual token and the optimal visual token as the target visual token.

7. A visual token compression apparatus, characterized by, Comprise: The request receiving module is used for accepting the visual token compression request carrying the original video frame sent by the user terminal; The key frame screening module is used for performing key frame screening operation on the original video frame to obtain key video frames; The visual token generation module is used for inputting the key video frames into a visual language model to generate visual tokens; The visual token screening module is used for performing visual token screening operation on the visual tokens according to the attention degree of the visual area and the content importance of the visual area according to the text instruction to obtain target visual tokens; The visual token output module is used for outputting the target visual tokens to the user terminal.

8. The visual token compression apparatus of claim 7, wherein, The key frame screening module comprises: The index calculation submodule is used for calculating the pixel information entropy index and the interframe pixel difference index of the original video frame; The original frame screening submodule is used for performing original frame screening operation on the original video frame according to the pixel information entropy index and the interframe pixel difference index to obtain candidate video frames; The semantic feature conversion submodule is used for inputting the candidate video frames into a visual encoder to perform semantic feature conversion operation to obtain semantic feature vectors; The candidate frame screening submodule is used for performing candidate frame screening operation on the candidate video frames according to the semantic feature vectors to obtain the key video frames. 9.A computer device, comprising a memory and a processor, and characterized in that, The memory stores computer readable instructions, and the processor executes the computer readable instructions to realize the steps of the visual token compression method in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer readable instructions, and the computer readable instructions are executed by the processor to realize the steps of the visual token compression method in any one of claims 1 to 6.

Images