Real-time Keyframe Extraction System and Method for Multimodal Large Models

Abstract

This invention proposes a keyframe extraction method for multimodal large-scale models, which improves the efficiency and generation quality of multimodal large-scale model video data processing through collaborative computing between the user end and the cloud. The technical solution includes: 1) After the user application submits data, the interface service performs preliminary processing on the data, dividing and merging video and audio frames into data slices according to time sequence; 2) Within each data slice on the user end, n key video frames are dynamically extracted and compressed and packaged with the corresponding audio frames for transmission to the cloud; 3) The cloud encodes and fuses the multimodal data before inputting it into the large-scale model generation result; 4) The cloud evaluates the generation quality, and the user end adaptively adjusts the data preprocessing and keyframe extraction parameter configurations based on the generation quality and response latency. This invention significantly improves the response speed and output quality of multimodal large-scale model cloud services by reducing redundant data transmission and dynamically optimizing computation and transmission resources.

Description

Technical Field

[0001] This invention belongs to the field of edge-cloud collaborative optimization technology in the Internet application layer, specifically a real-time keyframe extraction system and method for multimodal large models. Background Technology

[0002] With the deepening development of artificial intelligence technology, multimodal large models, by integrating multi-dimensional data such as video, audio, and text, have significantly improved perception and decision-making capabilities in complex scenarios. This has significantly enhanced the model's understanding and processing of complex real-world problems, expanding its commercial boundaries. It has demonstrated a significant driving force in multiple industries, including education, healthcare, finance, and manufacturing. With increasing market demand and policy support, the Chinese multimodal large model market continues to expand. Statistics show that in the first half of 2024, the domestic multimodal large model industry market size was approximately 3.3 billion yuan. Numerous companies, universities, and research institutions have increased investment, releasing numerous general-purpose large models. As the technology matures and costs decrease, the application of multimodal large models in more fields becomes possible, further driving market growth. Therefore, research on multimodal large models is of great significance.

[0003] The delivery of services for multimodal large-scale models heavily relies on the computing power and resource scheduling efficiency of cloud platforms. The computational intensity and real-time requirements pose a dual challenge to infrastructure. Core tasks such as video stream parsing and cross-modal feature fusion involve large-scale parallel computing, with each inference requiring significant GPU resources, placing higher demands on the dynamic scheduling capabilities of computing clusters. At the real-time level, scenarios such as industrial quality inspection and autonomous driving require millisecond-level precision in end-to-end responses, while applications like medical diagnosis need to ensure strict temporal synchronization of multimodal data. This contradiction between high real-time requirements and resource-intensive computing becomes a key bottleneck. Regarding network transmission, the burst bandwidth of video streams and the multi-level traffic differences formed by voice and sensor data can easily cause network congestion, while traditional static resource allocation mechanisms can lead to contention for computing units during peak concurrency periods, significantly increasing service response fluctuations. Therefore, designing a resource-efficient cloud service architecture has become a core technical direction for ensuring service stability and response efficiency.

[0004] Currently, most cloud services for multimodal large models use raw video streams as direct input, leading to high redundancy in end-to-end data transmission and excessively long context windows. Statistics show that unprocessed 1080P video streams can consume 4-8 Mbps of bandwidth, with over 60% of consecutive frames containing semantically repetitive content. This not only exacerbates network load but may also interfere with the model's attention mechanism due to redundant information, reducing output accuracy. Summary of the Invention

[0005] To address the issues of high load and redundant information in existing multimodal large models during video streaming, this invention proposes a keyframe extraction system and method for multimodal large models. By dynamically selecting high-information-density keyframes to compress video data traffic, the system optimizes model output quality while reducing transmission overhead.

[0006] The keyframe extraction method for large multimodal models is implemented according to the following steps:

[0007] Step 1: Build a multimodal large model cloud service architecture. Deploy service interfaces on the end devices for preprocessing and extracting keyframes, and deploy multimodal large models on the cloud devices for performing data encoding, modality fusion and inference tasks.

[0008] Step 2: The service interface of the terminal device receives the multimodal data stream and performs preprocessing to obtain text data and data slices containing audio and video data.

[0009] Multimodal data streams include text data, video data, and audio data. The preprocessing process for each is as follows:

[0010] Text data: Text data received directly by the service interface is directly uploaded to the cloud device. Text data contained in video or audio data is extracted through speech recognition and OCR and uploaded to the cloud device as well.

[0011] Video and audio data: The preset data segment length is used to divide the video and audio data into video frames and audio frames of equal duration according to the preset time length, and then package them into data segments;

[0012] The amount of data in all data was initially reduced through parameter adjustment, sampling, and compression.

[0013] Step 3: Extract keyframes from each video data segment using a video keyframe extraction algorithm, and package the data segment containing only keyframes with the audio data segment and transmit them to the cloud device.

[0014] The keyframe selection algorithm based on the difference between adjacent frames is as follows:

[0015] Step 301: For a certain data slice, initialize the number of keyframes and the data slice boundary index list, recursively divide the data slice and allocate the number of keyframes until the first sub-data slice only needs to select one keyframe.

[0016] The specific method for recursively partitioning data slices is as follows:

[0017] For a given data slice t, initially determine the total number of video frames n contained in that data slice. t and all video frames within the current data slice. The number of keyframes k to be selected within the data slice is obtained through a configuration algorithm. t .

[0018] The data slice was then divided into two parts, and half of the keyframes were selected within each part. One keyframe; continue to divide the first half of the data slice equally, and evenly distribute the number of keyframes to be selected in each newly divided data slice, until only one keyframe needs to be selected in each sub-data slice.

[0019] In this process, a list is used. key and list bd This is used to record the number of keyframes to be selected within each sub-data slice and the boundary index of that sub-data slice.

[0020] Step 302: Select a key frame from the sub-data slices that only require one key frame based on the frame difference between adjacent frames using the secretary algorithm, and determine the initial threshold.

[0021] The process of selecting keyframes using the secretary algorithm is as follows:

[0022] First, observe before For each frame, the maximum frame difference between adjacent frames is recorded as an initial threshold. In subsequent frames, the first frame whose frame difference with the previous frame exceeds this threshold is selected as the keyframe. If no subsequent frame meets this condition, the last frame is selected as the keyframe.

[0023] Step 303, according to the list key The remaining sub-data slices are processed sequentially in ascending order of the number of keyframes to be selected recorded in the data. After keyframe selection is completed for each sub-data slice, the threshold is updated until all sub-data slices have been processed.

[0024] For each sub-data slice, the frame difference between adjacent frames is calculated sequentially. The mechanism for triggering keyframe selection is as follows: 1. If the frame difference of the current frame exceeds the threshold and the number of keyframes required for the interval has not been reached, then the current frame is selected as a keyframe. Once the required number of keyframes is reached, the remaining frames are discarded. 2. If the number of remaining frames is equal to the number of keyframes that have not yet been selected, then not selecting any frames would result in an insufficient number of keyframes. In this case, all remaining frames are selected as keyframes.

[0025] The threshold update mechanism is as follows: when the current sub-data slice completes the key frame selection, if the next sub-data slice needs to select k′ key frames, then the k′ largest value is selected from the frame difference of the processed frames as the new threshold.

[0026] Step 304: Repeat steps 301 to 303 until all data slices have completed the selection of keyframes;

[0027] Step 305: After extracting keyframes from each data slice, replace the video frame portion of the original data slice with the keyframes to obtain a data slice with semantically duplicated video frames eliminated.

[0028] Step 4: The terminal device uploads each data slice containing only keyframes to the cloud device in sequence, and simultaneously inputs it into the multimodal big model for fusion with audio data and text data, and transmits the generated content back to the terminal device; at the same time, the multimodal big model evaluates the generated content and transmits the evaluation results to the optimizer of the terminal device.

[0029] Step 5: The user-side optimizer considers both response latency and generation quality, and adjusts the preprocessing configuration and the number of keyframes specified in the keyframe extraction algorithm.

[0030] Step 6: Based on the adjusted preprocessing configuration and the number of keyframes, return to Step 2 and repeat Steps 2 through 5 until data processing is complete.

[0031] A real-time keyframe extraction system for multimodal large models includes a user terminal and a cloud device. The user terminal is equipped with a data acquisition module, a preprocessing module, a keyframe extraction module, and an optimizer module. First, the data acquisition module collects multimodal data streams and sends them to the preprocessing module for preprocessing, dividing the multimodal data streams into text data, audio data, and video data. Then, the keyframe extraction module extracts keyframes from the video data and replaces the original video data with keyframes. Finally, the keyframe video data is packaged with the text and audio data and submitted to the cloud device. The cloud device is equipped with a modality fusion module, a multimodal large model, and an evaluation module. The modality fusion module encodes and fuses the keyframe video data with the text and audio data, transmits the fused data to the multimodal large model, and outputs the generated content. The generated content is simultaneously transmitted to the user terminal and the evaluation module. The evaluation module assesses the quality of the generated content and feeds the evaluation results back to the optimizer module on the user terminal. The optimizer module optimizes the parameters of the preprocessing module and the keyframe extraction module.

[0032] The advantages of this invention are:

[0033] (1) This invention constructs an efficient cloud service processing framework for multimodal large models through a collaborative architecture between the user end and the cloud. By combining cloud quality feedback and dynamic adjustment of end-side parameters, it balances transmission efficiency and generation quality. This framework effectively solves the problems of high data transmission redundancy and low resource utilization in multimodal large model services, and provides an efficient and reliable end-cloud collaborative solution for multimodal large model cloud service scenarios.

[0034] (2) After receiving the original multimodal data at the user end, the present invention uses compression coding technology to reduce transmission redundancy, accurately captures semantic key frames through the principle of maximizing frame difference, and filters key video frames online in real time based on the improved secretary algorithm, which significantly reduces the amount of transmitted data and model context interference.

[0035] (3) After completing cross-modal fusion and large model inference in the cloud, the present invention drives the user end to adaptively optimize the preprocessing parameters and key frame extraction strategy through quality assessment and real-time feedback, thereby optimizing the overall system efficiency and user experience. Attached Figure Description

[0036] Figure 1 This invention proposes a multimodal large-scale model cloud service framework.

[0037] Figure 2 This is a flowchart of the keyframe selection algorithm proposed in this invention. Detailed Implementation Plan

[0038] This invention proposes a real-time keyframe extraction system and method for multimodal large-scale models. It selects keyframes based on the difference between adjacent frames of a video modality, thereby eliminating redundant semantic information in the original data. Through collaborative computing between the user end and the cloud, it improves the efficiency and generation quality of multimodal large-scale model video data processing, effectively saving network resources while reducing the context length of the input large-scale model and improving model accuracy. The technical solution includes: 1) After the user application submits data, the interface service performs preliminary processing, dividing and merging video and audio frames into data slices according to time sequence; 2) Within each data slice on the user end, n key video frames are dynamically extracted and compressed and packaged with the corresponding audio frames for transmission to the cloud; 3) The cloud encodes and fuses the multimodal data before inputting it into the large-scale model generation result; 4) The cloud evaluates the generation quality, and the user end adaptively adjusts the data preprocessing and keyframe extraction parameter configurations based on the generation quality and response latency. This invention significantly improves the response speed and output quality of multimodal large-scale model cloud services by reducing redundant data transmission and dynamically optimizing computation and transmission resources.

[0039] The overall system structure of the present invention is as follows: Figure 1 As shown, the client-side and cloud devices jointly process different parts of the multimodal large model. The client-side is responsible for data acquisition and preprocessing, where a keyframe extraction algorithm is used to segment and extract keyframes from the video, and submits the processed data to the cloud devices. The cloud devices are responsible for multimodal information encoding and fusion, as well as inference and generation quality assessment of the multimodal large model, and responding to user requests. The client-side ultimately adjusts its data preprocessing strategy based on the generation results and evaluation metrics returned by the cloud, such as adjusting the resolution or the number of keyframes, thereby optimizing the overall system efficiency and user experience.

[0040] The keyframe extraction method for multimodal large models is implemented according to the following steps:

[0041] Step (1) Build a multimodal large model cloud service architecture, in which the middle device deploys service interfaces for preprocessing and extracting key frames, and the cloud device deploys a multimodal large model for performing data encoding, modality fusion and inference tasks;

[0042] Step (2) The terminal device receives the multimodal data stream submitted by the application and performs preprocessing. It extracts text data through speech recognition and OCR, and reduces the amount of data by compression and sampling. It packages video frames and audio frames of equal duration into data slices.

[0043] To reduce the amount of metadata and process it into a format suitable for subsequent frameworks, the service interface of the user terminal device receives video streams from the upper-layer application and preprocesses them. Specifically, the service interface directly uploads text data to the cloud, while other received video and audio data are segmented into video frames and audio frames of equal duration. Text is extracted from the video or video through speech recognition and OCR, and the amount of all data is initially reduced through parameter adjustment, sampling, compression and other methods.

[0044] Due to the synchronization requirements of multimodal large models for video and audio data, user devices will package video frames and audio frames within the same time period into data segments and upload them. Therefore, at this stage, it is necessary to perform preliminary segmentation and merging of data frames and audio frames within the corresponding time period according to the preset time length of the data segments.

[0045] Step (3) calls the keyframe extraction algorithm based on the secretary algorithm for each data slice sequence to extract the keyframes in each data slice, specifically including:

[0046] (3a) Initialize the number of keyframes and the list of data slice boundary indices, recursively divide the data slices and allocate the number of keyframes until the first sub-data slice only needs to select one keyframe.

[0047] (3b) Calculate the differences in pixel, edge and region features between adjacent frames using the frame difference method, and select key frames by combining dynamic threshold: In the first stage, the initial threshold is determined by the secret algorithm, and in the subsequent stages, the threshold is updated according to the difference value of the selected key frames, and frames with difference values ​​exceeding the threshold are selected first.

[0048] Traditional video keyframe selection methods typically select keyframes at uniform intervals within data slices based on frame rate configuration, lacking a fine-grained perception of video content. This approach easily overlooks important events or key changes, affecting the accuracy of video analysis. Furthermore, multimodal large models focus on the semantic content of video rather than fluency in their understanding of visual information, which does not align with the actual needs of multimodal large models for video streams. To address this issue, this invention proposes a novel keyframe selection algorithm based on the difference between adjacent frames. This algorithm extracts quantitative features from video frames to calculate the differences between adjacent frames, thereby detecting significant changes in video content. Three main methods for quantitatively calculating video frame features with relatively low computational complexity exist: pixel features compare pixel differences between adjacent video frames, edge features capture differences in object contours, and region features measure the degree of change in the area segmented by edges. Within each video interval, the algorithm selects several frames with the greatest changes as keyframes. By prioritizing these key changes, the algorithm can more comprehensively understand the video content, thereby improving the accuracy of the analysis results.

[0049] Since keyframe selection is an online problem, frames need to be processed sequentially, and future frames cannot be predicted, it's impossible to simultaneously calculate and evaluate the differences of all frames. Therefore, the basic design idea of ​​the algorithm is to calculate the difference of the current frame, combine it with the difference information of previously processed frames, and immediately decide whether to select the current frame as the keyframe. To this end, the algorithm introduces the secretary problem and optimizes it. The original secretary algorithm aims to select the largest video frame from n video frames online. The specific approach of the secretary algorithm is to first observe the previous... The method records the maximum frame difference within a frame as a threshold, and selects the first frame in subsequent frames that exceeds this threshold as the keyframe. If no frame meets the condition, the last frame is selected. This method can achieve near-optimal online keyframe selection under limited information.

[0050] In the scenario described in this invention, it is necessary to select a specific number of k keyframes, while traditional secretary algorithms can only be used to select a single keyframe. To solve this problem, this invention proposes a keyframe selection algorithm. At the beginning of each data slice t, the configuration algorithm obtains the number of keyframes k to be selected within that data slice. t Subsequently, during the initialization phase, the algorithm divides the data slice into two equal parts, and selects half of the keyframes within each part; that is, it selects keyframes from each part. The algorithm first divides the data slice into several keyframes, then further divides the first half of the data slice into equal parts, evenly distributing the number of keyframes to be selected within each new slice, until only one keyframe needs to be selected for each slice. During this process, the algorithm uses a list... key and list bd This is used to record the number of keyframes to be selected within each sub-data slice and the boundary index of that sub-data slice.

[0051] After initialization, the algorithm will begin processing video frames. In the first stage, the algorithm will process the first sub-slice of data. Within this slice, each video frame will be scored using its frame difference from the previous frame, and a keyframe will be selected within this interval using a traditional online secretariat algorithm. After the first stage is completed, the maximum frame difference value in the first slice will be recorded as a threshold for processing subsequent slices.

[0052] In the second stage, the algorithm will process the remaining data slices sequentially. Within each sub-data interval, there are two situations that will trigger the selection of keyframes: 1. If the frame difference of the current frame exceeds the threshold and the number of keyframes required for the interval is not reached, then the current frame is selected as a keyframe; 2. If the number of remaining frames is equal to the number of unselected keyframes, then if no frames are selected, the number of keyframes will be insufficient, so all remaining frames need to be selected as keyframes.

[0053] The threshold is updated after each sub-data slice is processed. Since the length of the next data slice and the number of keyframes to be selected are the same as the sum of the lengths and keyframes of all previously processed data slices, the new data slice can be selected using the frame difference information of the processed frames. If the new sub-data slice needs to select k′ keyframes, then the k′-th largest value from the frame differences of the processed frames is selected as the new threshold. The algorithm continues to run until all frames have been processed.

[0054] After extracting keyframes within each data slice, replacing the video frame portion of the original data slice with the keyframes yields a data slice with semantically duplicated video frames eliminated.

[0055] Specifically, such as Figure 2 As shown, the steps of the keyframe selection algorithm proposed in this invention are as follows;

[0056] 1. Confirm the parameters and data required for this round of algorithm execution, including: the number of keyframes k to be extracted within each data slice. t The total number of video frames for each data slice, n t and all video frames within the current data slice.

[0057] 2. Create and maintain two lists: a list of the number of keyframes to be selected for each sub-data slice. key List of boundary indices for sub-data slices bd .

[0058] 3. Add to list key , will n t Add to list bd .

[0059] 4. When k t When >1, let And repeat step 3 until k t =1.

[0060] 5. Using the secretary algorithm based on the frame difference between adjacent frames... arrive Select a keyframe and calculate the maximum frame difference (dif). max .

[0061] 6. Add keyframes to l t and will dif max This serves as the subsequent frame difference threshold.

[0062] 7. Set a temporary variable r = 1, and use it as a list. bd The circular index is used to represent the currently processed sub-data slice; the temporary variable c = 0 records the number of keyframes extracted for each data slice. The algorithm processes data online. and The content of the sub-data slice.

[0063] 8. Within each loop, calculate the frame. and frame difference dif i If the frame difference is greater than the selection threshold (dif) i >threshold) and the data slice also needs to select keyframes (list) key [r]>c), or the remaining frames equal to the number of keyframes still to be selected (list) key [r]-c==list bd [r]-i+1), then Select as a keyframe and add to l t And let c = c + 1.

[0064] 9.If i <list bd [r] indicates that the current sub-data slice has been processed. Let r = r + 1, c = 0, and update the threshold to the th element in the processed video frame list. key [r] large frame differences. If there are still unprocessed frames,

[0065] Repeat step 8.

[0066] Step (4) The terminal device uploads the data slice containing key frames to the cloud. The cloud device integrates multimodal data and inputs it into the large model for inference, generates output results, and uses technical indicators such as perplexity to make a preliminary judgment on the quality of model generation. The generation quality judgment evaluation indicators and generated content are then transmitted back to the user terminal.

[0067] Step (5) The cloud feeds back the output results and quality indicators to the end device. The end device dynamically adjusts the preprocessing parameters and the number of key frames according to the latency and quality requirements, and optimizes the subsequent data slice processing strategy.

[0068] The client-side optimizer considers both response latency and generation quality, adjusting preprocessing configurations and the number of keyframes specified in the keyframe extraction algorithm to optimize user experience. Specifically, preprocessing configurations include audio sampling rate, compression coding method, and video resolution. Higher configurations result in lower data distortion, allowing the cloud-based multimodal large model to more clearly capture semantic information in the data stream. However, this also increases transmission and computation latency due to the increased data volume. The number of keyframes affects the amount of data to be transmitted, context length, and video semantic completeness. Too many keyframes lead to excessive data volume and context length, negatively impacting response latency and model generation quality. Too few keyframes result in significant semantic gaps in the video, further affecting model generation quality.

[0069] Step (6) Repeat steps (2) to (5) until all data processing is complete.

[0070] In summary, this invention proposes a keyframe extraction method for multimodal large model cloud services. By using dynamic frame difference thresholding and a recursive secretary algorithm, it achieves online screening of high information density keyframes, significantly reducing video data transmission redundancy and model context length, and improving end-to-cloud collaboration efficiency while ensuring the quality of multimodal large model generation.

Claims

1. A keyframe extraction method for multimodal large models, characterized in that, Includes the following steps: Step 1: Build a multimodal large model cloud service architecture. Deploy service interfaces on the end devices for preprocessing and extracting keyframes, and deploy multimodal large models on the cloud devices for performing data encoding, modality fusion and inference tasks. Step 2: The service interface of the terminal device receives the multimodal data stream and performs preprocessing to obtain text data and data slices containing audio and video data. Step 3: Extract keyframes from each video data segment using a video keyframe extraction algorithm, and package the data segment containing only keyframes with the audio data segment and transmit them to the cloud device. The keyframe selection algorithm based on the difference between adjacent frames is as follows: Step 301: For a given data slice, initialize the number of keyframes and the data slice boundary index list. Recursively divide the data slice and allocate the number of keyframes until the first sub-data slice only needs to select one keyframe and use the list. and To record the number of keyframes to be selected within each sub-data slice and the boundary index of that sub-data slice; Step 302: Select a key frame from the sub-data slices that only require one key frame based on the frame difference between adjacent frames using the secretary algorithm, and determine the initial threshold. Step 303, according to The remaining sub-data slices are processed sequentially in ascending order of the number of keyframes to be selected recorded in the data. After keyframe selection is completed for each sub-data slice, the threshold is updated until all sub-data slices have been processed. For each sub-data slice, the frame difference between adjacent frames is calculated sequentially. The mechanism for triggering key frame selection is as follows:

1. If the frame difference of the current frame exceeds the threshold and the number of key frames required for the sub-data slice has not been reached, then the current frame is selected as the key frame. Once the required number of key frames is reached, the remaining frames are discarded.

2. If the number of remaining frames equals the number of unselected keyframes, then not selecting any frames will result in an insufficient number of keyframes. In this case, all remaining frames will be selected as keyframes. The threshold update mechanism is as follows: After the current sub-data slice completes keyframe selection, the next sub-data slice needs to select... If the first keyframe is selected, then the second keyframe is chosen from the frame differences of the already processed frames. The larger value is used as the new threshold; Step 304: Repeat steps 301 to 303 until all data slices have completed the selection of keyframes; Step 305: After extracting keyframes from each data slice, replace the video frame portion of the original data slice with the keyframes to obtain a data slice with semantically duplicated video frames eliminated. Step 4: The terminal device uploads each data slice containing only keyframes to the cloud device in sequence, and simultaneously inputs it into the multimodal big model for fusion with audio data and text data, and transmits the generated content back to the terminal device; at the same time, the multimodal big model evaluates the generated content and transmits the evaluation results to the optimizer of the terminal device. Step 5: The end device optimizer considers both response latency and generation quality, and adjusts the preprocessing configuration and the number of keyframes specified in the keyframe extraction algorithm. Step 6: Based on the adjusted preprocessing configuration and the number of keyframes, return to Step 2 and repeat Steps 2 through 5 until data processing is complete.

2. The keyframe extraction method for multimodal large models according to claim 1, characterized in that, The multimodal data stream includes text data, video data, and audio data, and the preprocessing process for each is as follows: Text data: Text data received directly by the service interface is directly uploaded to the cloud device. Text data contained in video or audio data is extracted through speech recognition and OCR and uploaded to the cloud device as well. Video and audio data: The preset data slice duration is used to divide the video and audio data into video frames and audio frames of equal duration, and then package them into data slices.

3. The keyframe extraction method for multimodal large models according to claim 1, characterized in that, The specific method for recursively partitioning data slices is as follows: For a specific data slice At the beginning, confirm the total number of video frames contained in the data slice. and all video frames within the current data slice. The number of keyframes to be selected within the data slice is obtained through a configuration algorithm. ; The data slice was then divided into two parts, and half of the keyframes were selected within each part. One keyframe; continue to divide the first half of the data slice equally, and evenly distribute the number of keyframes to be selected in each newly divided data slice, until the first sub-data slice only needs to select one keyframe.

4. The keyframe extraction method for multimodal large models according to claim 1, characterized in that, The process of selecting a keyframe from a sub-data slice where only one keyframe needs to be selected using the secretary algorithm is as follows: First, observe before For each frame, the maximum frame difference between adjacent frames is recorded as an initial threshold. In subsequent frames, the first frame whose frame difference with the previous frame exceeds the threshold is selected as the key frame. If no subsequent frame meets the condition, the last frame is selected as the key frame.

5. A keyframe extraction system for multimodal large models, used to implement the keyframe extraction method according to any one of claims 1 to 4, characterized in that, It includes terminal devices and cloud devices. The terminal devices are equipped with a data acquisition module, a preprocessing module, a keyframe extraction module, and an optimizer module. First, the data acquisition module acquires multimodal data streams and sends them to the preprocessing module for preprocessing, which divides the multimodal data streams into text data, audio data, and video data. Then, the keyframe extraction module extracts keyframes from the video data and replaces the original video data with keyframes. Finally, the keyframe video data is packaged with text and audio data and submitted to the cloud device. The cloud device is equipped with a modal fusion module, a multimodal large model, and an evaluation module. The modal fusion module encodes and fuses the keyframe video data with text and audio data, transmits the fused data to the multimodal large model, and outputs the generated content. The generated content is simultaneously transmitted to the end device and the evaluation module. The evaluation module evaluates the quality of the generated content and feeds the evaluation results back to the optimizer module of the end device. The optimizer module optimizes the parameters of the preprocessing module and the keyframe extraction module.

Images