Video processing method and device, electronic equipment, storage medium and program product

Abstract

The application discloses a video processing method, device, electronic equipment, storage medium and program product. The video processing method comprises: in the decoding stage, reading video frames in a to-be-processed video frame by frame, decoding a video frame to obtain a frame data in an original format after reading each video frame, and transmitting the frame data in the original format to a conversion stage; in the conversion stage, converting the frame data in the original format into frame data in a target format after each time, transmitting the frame data in the target format to an encoding stage; in the encoding stage, encoding the frame data in the target format to obtain semantic representation data, and transmitting at least one frame of semantic representation data to an inference stage; and in the inference stage, inputting at least one frame of semantic representation data to a multi-modal large model for inference to obtain an inference result after obtaining at least one frame of semantic representation data. The application can reduce the occupation of a storage area, reduce processing delay, and improve resource utilization.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, multimodal large models, represented by VLLM (Vision-Language Large Models), have received widespread attention and application in fields such as computer vision and natural language processing. These multimodal large models, due to their ability to understand and generate across modalities, have demonstrated enormous potential in video analysis, content generation, and intelligent interaction, attracting significant attention from both academia and industry. However, existing multimodal large model video processing frameworks are often limited by performance bottlenecks during specific inference processes, leading to numerous challenges in their widespread application in real-time multimodal tasks.

[0003] Current video processing workflows generally employ a serial processing model, where the entire video is decoded and converted before being encoded and inferred uniformly. However, this approach results in high memory consumption, significant processing latency, and low resource utilization. Summary of the Invention

[0004] The purpose of this invention is to provide a video processing method, apparatus, device, storage medium, and product to solve the problems of high memory consumption, large processing latency, and low resource utilization caused by current video processing methods. The specific technical solution is as follows: To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a video processing method, the method comprising: During the decoding stage, video frames in the video to be processed are read frame by frame. Each time a video frame is read, it is decoded to obtain a frame of original format data. Each time a frame of original format data is obtained, it is transmitted to the conversion stage. During the conversion phase, each time a frame of original format data is obtained, the frame of original format data is converted into a frame of target format data required by the multimodal large model. Each time a frame of target format data is obtained, the frame of target format data is transmitted to the encoding phase. During the encoding stage, each time a frame of target format data is obtained, the frame of target format data is encoded to obtain a frame of semantic representation data. Each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is transmitted to the inference stage. During the inference phase, each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is input into the multimodal large model for inference, and the inference result output by the multimodal large model is obtained.

[0005] Optionally, transmitting the frame data in the original format to the conversion stage includes: transmitting the frame data in the original format to the conversion stage using a first storage area; the method further includes: in the conversion stage, after obtaining the frame data in the original format each time, notifying the decoding stage to delete the frame data in the original format from the first storage area; The step of transmitting the frame data of the target format to the encoding stage includes: transmitting the frame data of the target format to the encoding stage using a second storage area; the method further includes: in the encoding stage, after obtaining the frame data of the target format, notifying the conversion stage to delete the frame data of the target format from the second storage area; The step of transmitting the at least one frame of semantic representation data to the inference stage includes: transmitting the at least one frame of semantic representation data to the inference stage using a third storage area; the method further includes: during the inference stage, after obtaining each at least one frame of semantic representation data, notifying the encoding stage to delete the at least one frame of semantic representation data from the third storage area.

[0006] Optionally, inputting the at least one frame of semantic representation data into the multimodal large model for inference includes: The at least one frame of semantic representation data is input into the multimodal large model for inference. During the inference process, a linear transformation is performed on the at least one frame of semantic representation data to calculate the newly added key vector and the newly added value vector, and the stored key vector and the stored value vector are read from the key-value cache. Based on the stored key vectors and value vectors, as well as the newly added key vectors and value vectors, inference is performed to obtain the inference result, and the newly added key vectors and value vectors are stored in the key-value cache.

[0007] Optionally, storing the newly added key vector and the newly added value vector in the key-value cache includes: if the key-value cache reaches a set length, clearing the oldest stored key vector and value vector, storing the newly added key vector and the newly added value vector in the key-value cache, and recording the storage time of the newly added key vector and the newly added value vector.

[0008] Optionally, the method further includes: recording processing progress information during video processing; wherein the video frame corresponds to a frame number, and the processing progress information includes the frame number of the video frame that has just completed inference; after video processing is interrupted and resumed, acquiring the processing progress information, determining the remaining video frames in the video to be processed based on the processing progress information, and continuing to execute the decoding stage, the conversion stage, the encoding stage, and the inference stage on the remaining video frames.

[0009] Secondly, embodiments of this application provide a video processing apparatus, the apparatus comprising: The decoding module is used to read video frames from the video to be processed frame by frame during the decoding stage. For each video frame read, the video frame is decoded to obtain a frame of original format frame data. For each frame of original format frame data obtained, the frame of original format frame data is transmitted to the conversion stage. The conversion module is used to convert each frame of original format data into a frame of target format data required by the multimodal large model during the conversion stage, and to transmit the frame of target format data to the encoding stage after each frame of target format data is obtained. The encoding module is used to encode each frame of target format data obtained in the encoding stage to obtain a frame of semantic representation data, and to transmit the at least one frame of semantic representation data to the inference stage after obtaining at least one frame of semantic representation data. The inference module is used to, during the inference phase, input the at least one frame of semantic representation data into the multimodal large model for inference, and obtain the inference result output by the multimodal large model.

[0010] Optionally, the decoding module is specifically configured to transmit the frame data of the original format to the conversion stage using the first storage area; the conversion module is further configured to, in the conversion stage, notify the decoding stage to delete the frame data of the original format from the first storage area after obtaining the frame data of the original format each time. The conversion module is specifically used to transmit the frame data of the target format to the encoding stage using the second storage area; the encoding module is also used to notify the conversion stage to delete the frame data of the target format from the second storage area after obtaining the frame data of the target format in the encoding stage. The encoding module is specifically used to transmit the at least one frame of semantic representation data to the inference stage using the third storage area; the inference module is also used to notify the encoding stage to delete the at least one frame of semantic representation data from the third storage area after obtaining the at least one frame of semantic representation data in the inference stage.

[0011] Optionally, the inference module includes: The cache acquisition unit is used to input the at least one frame of semantic representation data into the multimodal large model for inference. During the inference process, the at least one frame of semantic representation data is subjected to linear transformation to calculate the newly added key vector and the newly added value vector, and the stored key vector and the stored value vector are read from the key-value cache. The inference storage unit is used to perform inference based on the stored key vectors and stored value vectors, as well as the newly added key vectors and newly added value vectors, to obtain the inference result, and to store the newly added key vectors and newly added value vectors in the key-value cache.

[0012] Optionally, the inference storage unit is specifically used to, if the key-value cache reaches a set length, clear the oldest stored key vector and value vector, store the newly added key vector and value vector in the key-value cache, and record the storage time of the newly added key vector and value vector.

[0013] Optionally, the device further includes: A recording module is used to record processing progress information during video processing; wherein, each video frame corresponds to a frame number, and the processing progress information includes the frame number of the video frame that has just completed inference; The recovery module is used to obtain the processing progress information after the video processing is interrupted and resumed, determine the remaining video frames in the video to be processed based on the processing progress information, and continue to execute the decoding stage, the conversion stage, the encoding stage and the inference stage on the remaining video frames.

[0014] Thirdly, embodiments of this application provide an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to execute the computer programs stored in the memory to implement the video processing method as described in any of the preceding claims.

[0015] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the video processing method as described in any of the preceding claims.

[0016] Fifthly, embodiments of this application provide a computer program product storing a computer program that, when executed by a processor, implements the video processing method as described in any of the preceding claims.

[0017] In this embodiment, the decoding, conversion, encoding, and inference stages are executed in parallel. Data processed in each stage can proceed to the next stage immediately, without waiting for the entire video to be decoded and converted before encoding and inference of the entire sequence. This eliminates the need to store the entire video data, reducing storage space usage. The absence of waiting time also reduces processing latency. Parallel execution fully utilizes various resources, eliminates resource idleness, and improves resource utilization.

[0018] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some drawings of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a video processing method according to an embodiment of this application; Figure 2 This is a schematic diagram of a video processing procedure according to an embodiment of this application; Figure 3 This is a schematic diagram of another video processing procedure according to an embodiment of this application; Figure 4 This is a structural block diagram of a video processing apparatus according to an embodiment of this application; Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Traditional video processing workflows employ a serial computing model, where decoding, conversion, encoding, and inference are executed sequentially. After each frame of video is decoded and converted, the next frame is decoded and converted. After all frames are decoded and converted, all frames are encoded. Finally, all frames are inferred in a unified manner.

[0023] However, the above method will lead to the following problems: High storage space usage: To decode and process long videos, the original video frames need to be decoded and converted into a high-resolution format and cached in the storage area. Simultaneously, the encoded data also needs to be stored. This dual storage mechanism leads to excessive storage space usage, especially with high video resolution or a large number of frames, easily causing memory overflow issues. In this embodiment, a pipelined parallel processing of the decoding, conversion, encoding, and inference stages is used to store data frame by frame, and the stored data is promptly deleted after being read, thereby reducing the system's storage space pressure.

[0024] Significant processing latency: Relying on a serial processing chain, the inference of a multimodal large model can only begin after the entire video has been decoded, converted, and encoded. This approach inevitably leads to time delays, increasing latency in video stream processing and making it difficult to meet real-time video processing requirements. In this embodiment, by using a pipelined parallel processing approach for the decoding, conversion, encoding, and inference stages, inference of at least one frame can be achieved without waiting for all video frames to be decoded, converted, and encoded. This reduces the overall latency from video input to inference output, meeting the needs of real-time application scenarios.

[0025] Low resource utilization: Video decoding and multimodal large model inference typically rely on different types of hardware resources. In serial processing mode, there is a lack of effective coordinated scheduling between video decoding and model inference. Often, while one stage is being computed, resources in another stage are idle, resulting in resource utilization troughs and hindering efficient use of hardware resources. In this embodiment, a pipeline mechanism is used to enable highly parallel execution of video decoding, conversion, encoding, and inference tasks, avoiding resource idleness, fully activating the computing power of various hardware resources in the system, eliminating resource utilization troughs, and achieving efficient use of hardware resources.

[0026] To address the aforementioned issues, this application proposes a new, efficient pipelined parallel processing architecture that changes the serial processing approach. Each stage processes different subtasks, thereby effectively reducing storage space usage, improving resource utilization, and lowering overall processing latency.

[0027] Reference Figure 1The diagram shows a flowchart of a video processing method according to an embodiment of this application.

[0028] like Figure 1 As shown, the video processing method may include the following steps: Step 101: In the decoding stage, video frames in the video to be processed are read frame by frame. For each video frame read, the video frame is decoded to obtain a frame of original format data. For each frame of original format data obtained, the frame of original format data is transmitted to the conversion stage.

[0029] In this embodiment, the video processing task of the multimodal large model is subdivided into four sub-tasks: decoding, conversion, encoding, and inference. These stages are executed in parallel. For example, after decoding the first frame, the decoding stage can continue decoding the second frame, while the conversion stage can convert the decoded first frame. After converting the first frame, the conversion stage can convert the decoded second frame, while the encoding stage can encode the converted first frame. After encoding the first frame, the inference stage can infer from the encoded first frame, while the encoding stage can encode the converted second frame, and so on.

[0030] In this embodiment, the video to be processed (such as a local file or online video stream) is obtained from a video source. During the decoding stage, video frames in the video to be processed can be read frame by frame. For each video frame read, the system's video decoding hardware decodes the frame to obtain a frame of raw format data. After obtaining each frame of raw format data, it can be transferred to the conversion stage for conversion processing, without waiting for all decoding to be completed before entering the conversion stage.

[0031] For example, transmitting the frame data in its original format to the conversion stage includes: transmitting the frame data in its original format to the conversion stage using a first storage area. Specifically, the frame data in its original format is stored in the first storage area during the decoding stage, and the frame data in its original format is read from the first storage area during the conversion stage. The first storage area can be a memory area or a video memory area. After obtaining each frame data in its original format, the decoding stage stores the frame data in its original format in the first storage area. After storing the frame data in its original format, the decoding stage can notify the conversion stage to read it, and upon receiving the notification, the conversion stage reads the frame data in its original format from the first storage area.

[0032] For example, during the conversion phase, after each frame of original format data is obtained (i.e., after reading the frame of original format data), the decoding phase is notified to delete the frame of original format data from the first storage area. Upon receiving this notification, the decoding phase deletes the frame of original format data from the first storage area. Therefore, the original format frame data obtained by the decoding phase is not stored in the first storage area for a long period, but only for necessary intermediate transmission, avoiding the occupation of a large amount of storage space.

[0033] For example, the original format frame data can be frame data in YUV (luminance-blue difference-red difference) format.

[0034] For example, the decoding stage can be executed using CPU (Central Processing Unit) hardware resources or GPU (Graphics Processing Unit) hardware resources.

[0035] Step 102: In the conversion stage, each time a frame of original format data is obtained, the frame of original format data is converted into a frame of target format data required by the multimodal large model. Each time a frame of target format data is obtained, the frame of target format data is transmitted to the encoding stage.

[0036] In this embodiment, after obtaining each frame of raw format data in the decoding stage, the frame of raw format data is transmitted to the conversion stage for conversion processing. In the conversion stage, after obtaining each frame of raw format data, the frame of raw format data is converted into a frame of target format data required by the multimodal large model. After obtaining each frame of target format data, the frame of target format data can be transmitted to the encoding stage for encoding processing, without waiting for all conversions to be completed before entering the encoding stage.

[0037] Since the original format frame data after decoding is stored in the YUV color space and cannot be directly used for inference of multimodal large models, the system automatically converts the original format frame data into the target format frame data required by the multimodal large model. The target format refers to the format required by the multimodal large model to facilitate compatibility with it. Therefore, during the conversion stage, for each frame of original format frame data obtained, any applicable format conversion tool can be used to convert the original format frame data to obtain a frame of target format frame data. For example, if the original format is YUV and the target format is RGB (Red-Green-Blue), a YUV to RGB format conversion tool can be used to convert the YUV format to RGB format to obtain the frame data in the target format.

[0038] For example, transmitting a frame data of the target format to the encoding stage includes: transmitting the frame data of the target format to the encoding stage using a second storage area. Specifically, the frame data of the target format is stored in the second storage area during the conversion stage, and the frame data of the target format is read from the second storage area during the encoding stage. The second storage area can be a memory area or a video memory area. After obtaining each frame data of the target format, the conversion stage stores the frame data of the target format in the second storage area. After completing the storage of the frame data of the target format, the conversion stage can notify the encoding stage to read it, and upon receiving the notification, the encoding stage reads the frame data of the target format from the second storage area.

[0039] For example, during the encoding stage, after each frame of target format data is obtained (i.e., after reading the target format frame data), the encoding stage is notified to delete the target format frame data from the second storage area. Upon receiving this notification, the encoding stage deletes the target format frame data from the second storage area. Therefore, the target format frame data obtained during the conversion stage is not stored in the second storage area for an extended period; it is only used for necessary intermediate transmission, thus avoiding the occupation of a large amount of storage space.

[0040] For example, the target format frame data can be RGB format frame data.

[0041] For example, the conversion stage can be executed using either CPU hardware resources or GPU hardware resources. Accordingly, the frame data in the target format can be in the form of NumPy arrays (based on CPU and memory) or GPU tensors (based on GPU and video memory), which facilitates compatibility with large multimodal models corresponding to different types of input data.

[0042] Step 103: In the encoding stage, each time a frame of target format data is obtained, the frame of target format data is encoded to obtain a frame of semantic representation data. Each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is transmitted to the inference stage.

[0043] In this embodiment, after obtaining each frame of target format data in the conversion stage, the frame of target format data is transmitted to the encoding stage for encoding processing. In the encoding stage, each frame of target format data is encoded to obtain a frame of semantic representation data. After obtaining at least one frame of semantic representation data, the at least one frame of semantic representation data is transmitted to the inference stage, without waiting for all encoding to be completed before entering the inference stage.

[0044] For example, during the encoding stage, the frame data in the target format is input into a dedicated encoder of the multimodal large model. The encoder encodes the frame data in the target format to generate a frame of semantic representation data related to temporal semantics that conforms to the input format of the multimodal large model.

[0045] For example, transmitting the at least one frame of semantic representation data to the inference stage includes: transmitting the at least one frame of semantic representation data to the inference stage using a third storage area. Specifically, each time a frame of semantic representation data is obtained during the encoding stage, the frame of semantic representation data is stored in the third storage area, and at least one frame of semantic representation data is read from the third storage area during the inference stage. The third storage area can be a memory area or a video memory area. After obtaining each frame of semantic representation data, the encoding stage stores the frame of semantic representation data in the third storage area. After storing at least one frame of semantic representation data, the encoding stage can notify the inference stage to read it, and upon receiving the notification, the inference stage reads the at least one frame of semantic representation data from the third storage area.

[0046] For example, during the inference phase, after obtaining each frame of semantic representation data (i.e., after reading each frame of semantic representation data), the encoding phase is notified to delete the at least one frame of semantic representation data from the third storage area. Upon receiving this notification, the encoding phase deletes the at least one frame of semantic representation data from the third storage area. Therefore, the encoded semantic representation data is not stored in the third storage area for an extended period; it is only used for necessary intermediate transmission, thus avoiding the occupation of a large amount of storage space.

[0047] For example, the encoder can be a tokenizer, and the semantic representation data can be token data.

[0048] For example, the coding phase can be executed using either CPU hardware resources or GPU hardware resources.

[0049] Step 104: In the inference phase, for each frame of semantic representation data obtained, the at least one frame of semantic representation data is input into the multimodal large model for inference, and the inference result output by the multimodal large model is obtained.

[0050] In this embodiment of the application, after obtaining at least one frame of semantic representation data during the encoding stage, the at least one frame of semantic representation data is transmitted to the inference stage for inference. During the inference stage, the at least one frame of semantic representation data is input into the multimodal large model for inference, and the inference result output by the multimodal large model is obtained.

[0051] In this embodiment of the application, the multimodal large model can perform inference on at least one frame of semantic representation data, without having to wait for all the semantic representation data before inputting it into the multimodal large model for processing.

[0052] For example, the inference phase can be performed using GPU hardware resources.

[0053] In summary, by optimizing the pipelined parallel architecture, the efficiency and resource utilization of multimodal large-scale model video processing are improved, providing a more efficient and scalable technical solution for the deployment of multimodal large-scale models in video processing scenarios. This optimization method is particularly suitable for scenarios requiring the processing of real-time or long-duration video, enabling multimodal large-scale models to have stronger real-time processing capabilities and lower resource usage costs, which will help promote the further development and application of related technologies.

[0054] In one optional implementation, the process of inputting the at least one frame of semantic representation data into the multimodal large model for inference may include: inputting the at least one frame of semantic representation data into the multimodal large model for inference; during the inference process, performing a linear transformation on the at least one frame of semantic representation data to calculate newly added key vectors and newly added value vectors, and reading the stored key vectors and stored value vectors from the key-value cache; performing inference based on the stored key vectors and stored value vectors, as well as the newly added key vectors and newly added value vectors, to obtain an inference result, and storing the newly added key vectors and newly added value vectors in the key-value cache.

[0055] For example, the process of performing a linear transformation on the at least one frame of semantic representation data to calculate the newly added key vector and the newly added value vector may include: multiplying the at least one frame of semantic representation data with the first weight matrix corresponding to the key vector to obtain a first result, and adding the first result with the first bias term corresponding to the key vector to obtain the newly added key vector; multiplying the at least one frame of semantic representation data with the second weight matrix corresponding to the value vector to obtain a second result, and adding the second result with the second bias term corresponding to the value vector to obtain the newly added value vector. Wherein, the first weight matrix, the first bias term, the second weight matrix, and the second bias term are the model parameters of the multimodal large model.

[0056] For example, the key vector may include a semantic feature identifier corresponding to the semantic representation data. The semantic feature identifier may include, but is not limited to: the object category (e.g., people, vehicles, trees, etc.) and scene attributes (e.g., indoor, outdoor, etc.) in the video frame corresponding to the semantic representation data, and the frame number of the video frame corresponding to the semantic representation data, etc.

[0057] For example, the value vector may include the semantic content carrier corresponding to the semantic representation data. The semantic content carrier may include, but is not limited to, the deep semantic representation corresponding to the semantic representation data, such as specific semantic conclusions in the video frame corresponding to the semantic representation data, such as a person walking or a vehicle turning left, etc.

[0058] For example, in the process of reasoning based on the stored key vector and the stored value vector, as well as the newly added key vector and the newly added value vector, to obtain the reasoning result, the stored key vector and the newly added key vector can be concatenated to obtain a concatenated key vector, and the stored value vector and the newly added value vector can be concatenated to obtain a concatenated value vector. Attention calculation is performed based on the concatenated key vector and the concatenated value vector to obtain the attention calculation result. Then, the attention calculation result is processed using a feedforward neural network, an output layer, etc., to obtain the reasoning result.

[0059] In this approach, an incremental key-value cache (KV Cache) mechanism is adopted to address the need for frequent access to long-term context in large model inference. A key-value cache is established and maintained on a time series basis. After each inference of semantic representation data is completed, the key vector and value vector obtained in this calculation are stored in the key-value cache. In the next inference process of semantic representation data, the key vector and value vector stored in the key-value cache can be directly reused as context information, thereby realizing context-aware continuous inference and enabling the complex context dependencies between frames to be effectively supported under the key-value cache mechanism.

[0060] In one optional implementation, the process of storing the newly added key vector and the newly added value vector in the key-value cache may include: if the key-value cache reaches a set length, clearing the oldest stored key vector and value vector, storing the newly added key vector and the newly added value vector in the key-value cache, and recording the storage time of the newly added key vector and the newly added value vector.

[0061] In this approach, considering that the semantics of a video frame are most affected by its nearest preceding frame, discarding older information has a relatively small impact on the inference accuracy of the current video frame. Therefore, each time a key vector and value vector are stored, the storage time of the key vector and value vector is recorded. A set length is set for the key-value cache. If the key-value cache reaches the set length, the oldest stored key vector and value vector are cleared, and the newly added key vector and value vector are stored in the key-value cache. The storage time of the newly added key vector and value vector is also recorded, thereby avoiding unnecessary occupation of the storage area.

[0062] In one optional implementation, considering that video processing is a computationally intensive and time-consuming task, it is easily interrupted during operation due to external interference (such as network jitter, terminal resource contention, etc.) or internal needs (such as system maintenance, etc.). If the entire video is reprocessed after the interruption is restored, it will result in a waste of computing resources and time. Therefore, the method further includes: recording processing progress information during video processing; after the video processing is restored from an interruption, obtaining the processing progress information, determining the remaining video frames in the video to be processed based on the processing progress information, and continuing to execute the decoding stage, the conversion stage, the encoding stage, and the inference stage on the remaining video frames. In this approach, it is not necessary to reprocess the entire video after the interruption is restored, which improves video processing efficiency and enhances the flexibility and reliability of the system.

[0063] For example, during video processing, the system can automatically generate and store log records, which can record processing progress information. Accordingly, after video processing is interrupted and resumed, the processing progress information is obtained from the log records, and the remaining video frames in the video to be processed are determined based on the processing progress information. The decoding stage, the conversion stage, the encoding stage, and the inference stage are then performed on the remaining video frames.

[0064] For example, during video processing, the multimodal large model can use the key-value caching method described above for inference, and processing progress information can be recorded in the key-value cache. Accordingly, after video processing is interrupted and resumed, the processing progress information is retrieved from the key-value cache, and the remaining video frames in the video to be processed are determined based on the processing progress information. The decoding stage, the conversion stage, the encoding stage, and the inference stage are then performed on the remaining video frames.

[0065] For example, a frame number can be assigned to each video frame in the video to be processed, and the processing progress information may include the frame number of the video frame that has most recently completed inference. Accordingly, in the process of determining the remaining video frames in the video to be processed based on the processing progress information, the video segment in the video to be processed from the frame following the frame with the most recently completed inference frame number to the last frame is determined as the remaining video frames in the video to be processed.

[0066] In this embodiment, after each inference is completed, the inference result can be dynamically output to the application interface to ensure real-time feedback of the inference result, which is suitable for tasks that require real-time response (such as autonomous driving, real-time video analysis, live subtitle generation, etc.).

[0067] In this embodiment, a pipelined parallel processing architecture is designed to allow each stage to operate synchronously without blocking each other. Data flow is achieved through a buffer, enabling time-series linkage between stages and preventing processing stagnation. Task separation allows each stage to utilize hardware computing resources simultaneously, flexibly allocating resources across multiple stages to maximize efficiency and reduce single-point pressure during peak computations. An incremental key-value caching mechanism dynamically updates the key-value cache within the multimodal model, ensuring contextual relevance and enabling efficient cross-frame multimodal information correlation calculations. An interrupt recovery mechanism supports resuming computation from breakpoints, enhancing system flexibility.

[0068] In this embodiment, the decoding stage, the conversion stage, the encoding stage, and the inference stage can be processed using different hardware resources according to actual needs.

[0069] Reference Figure 2 The diagram illustrates a video processing procedure according to an embodiment of this application.

[0070] like Figure 2As shown, the video processing process may include: loading the video file stored on the hard drive into memory to form a video stream; executing a decoding stage through CPU process 1, reading the video stream from memory, decoding it into frame data in the original format, and writing the frame data in the original format back to memory (first storage area); executing a conversion stage through CPU process 2, reading the frame data in the original format from memory, converting it into NumPy data (RGB format), and writing the NumPy data (numerical array data) back to memory (second storage area); executing an encoding stage through CPU process 3, reading the NumPy data from memory, encoding it into token data through a tokenizer, and writing the token data back to memory (third storage area); the token data in memory is transferred to GPU memory, converted into a token tensor, and the GPU executes an inference stage, reading the token tensor and inputting it into a multimodal large model to complete inference.

[0071] Reference Figure 3 The diagram illustrates another video processing procedure according to an embodiment of this application.

[0072] like Figure 3 As shown, the video processing process may include: loading the video file stored on the hard drive into memory to form a video stream, and then transferring the video stream to the video memory; executing the decoding stage through GPU process 1, reading the video stream from the video memory, decoding it into frame data in the original format, and writing the frame data in the original format back to the video memory (first storage area); executing the conversion stage through GPU process 2, reading the frame data in the original format from the video memory, converting it into an RGB Tensor, and writing the RGB Tensor back to the video memory (second storage area); executing the encoding stage through GPU process 3, reading the RGB Tensor from the video memory, encoding it into a Token Tensor through a Tokenizer, and writing the Token Tensor back to the video memory (third storage area); executing the inference stage through GPU process 4, reading the Token Tensor and inputting it into a multimodal large model to complete inference.

[0073] Reference Figure 4 The diagram shows a structural block diagram of a video processing apparatus according to an embodiment of this application.

[0074] like Figure 4 As shown, the video processing device may include the following modules: The decoding module 401 is used to read video frames in the video to be processed frame by frame during the decoding stage. For each video frame read, the video frame is decoded to obtain a frame of original format frame data. For each frame of original format frame data obtained, the frame of original format frame data is transmitted to the conversion stage. The conversion module 402 is used to convert each frame of original format data into a frame of target format data required by the multimodal large model during the conversion stage, and to transmit the frame of target format data to the encoding stage each time a frame of original format data is obtained. The encoding module 403 is used to encode the frame data of the target format each time a frame data of the target format is obtained in the encoding stage to obtain a frame of semantic representation data, and to transmit the at least one frame of semantic representation data to the inference stage each time at least one frame of semantic representation data is obtained. The reasoning module 404 is used to, during the reasoning phase, input the at least one frame of semantic representation data into the multimodal large model for reasoning after obtaining the at least one frame of semantic representation data, and obtain the reasoning result output by the multimodal large model.

[0075] Optionally, the decoding module 401 is specifically used to transmit the frame data of the original format to the conversion stage using the first storage area; the conversion module 402 is further used to notify the decoding stage to delete the frame data of the original format from the first storage area after obtaining the frame data of the original format in the conversion stage. The conversion module 402 is specifically used to transmit the frame data of the target format to the encoding stage using the second storage area; the encoding module 403 is also used to notify the conversion stage to delete the frame data of the target format from the second storage area after obtaining the frame data of the target format in the encoding stage. The encoding module 403 is specifically used to transmit the at least one frame of semantic representation data to the inference stage using the third storage area; the inference module 404 is further used to notify the encoding stage to delete the at least one frame of semantic representation data from the third storage area after obtaining the at least one frame of semantic representation data in the inference stage.

[0076] Optionally, the inference module 404 includes: The cache acquisition unit is used to input the at least one frame of semantic representation data into the multimodal large model for inference. During the inference process, the at least one frame of semantic representation data is subjected to linear transformation to calculate the newly added key vector and the newly added value vector, and the stored key vector and the stored value vector are read from the key-value cache. The inference storage unit is used to perform inference based on the stored key vectors and stored value vectors, as well as the newly added key vectors and newly added value vectors, to obtain the inference result, and to store the newly added key vectors and newly added value vectors in the key-value cache.

[0077] Optionally, the inference storage unit is specifically used to, if the key-value cache reaches a set length, clear the oldest stored key vector and value vector, store the newly added key vector and value vector in the key-value cache, and record the storage time of the newly added key vector and value vector.

[0078] Optionally, the device further includes: A recording module is used to record processing progress information during video processing; wherein, each video frame corresponds to a frame number, and the processing progress information includes the frame number of the video frame that has just completed inference; The recovery module is used to obtain the processing progress information after the video processing is interrupted and resumed, determine the remaining video frames in the video to be processed based on the processing progress information, and continue to execute the decoding stage, the conversion stage, the encoding stage and the inference stage on the remaining video frames.

[0079] In this embodiment, the decoding, conversion, encoding, and inference stages are executed in parallel. Data processed in each stage can proceed to the next stage immediately, without waiting for the entire video to be decoded and converted before encoding and inference of the entire sequence. This eliminates the need to store the entire video data, reducing storage space usage. The absence of waiting time also reduces processing latency. Parallel execution fully utilizes various resources, eliminates idle resource waiting states, and improves resource utilization.

[0080] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0081] The video processing device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0082] The video processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0083] The video processing apparatus provided in this application embodiment can implement the various processes implemented in the method embodiment of any of the above claims. To avoid repetition, it will not be described again here.

[0084] Optionally, embodiments of this application also provide an electronic device.

[0085] Figure 5 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0086] Memory 503 is used to store computer programs; When processor 501 executes a computer program stored in memory 503, it performs the following steps: During the decoding stage, video frames in the video to be processed are read frame by frame. Each time a video frame is read, it is decoded to obtain a frame of original format data. Each time a frame of original format data is obtained, it is transmitted to the conversion stage. During the conversion phase, each time a frame of original format data is obtained, the frame of original format data is converted into a frame of target format data required by the multimodal large model. Each time a frame of target format data is obtained, the frame of target format data is transmitted to the encoding phase. During the encoding stage, each time a frame of target format data is obtained, the frame of target format data is encoded to obtain a frame of semantic representation data. Each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is transmitted to the inference stage. During the inference phase, each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is input into the multimodal large model for inference, and the inference result output by the multimodal large model is obtained.

[0087] Optionally, transmitting the frame data in the original format to the conversion stage includes: transmitting the frame data in the original format to the conversion stage using a first storage area; the method further includes: in the conversion stage, after obtaining the frame data in the original format each time, notifying the decoding stage to delete the frame data in the original format from the first storage area; The step of transmitting the frame data of the target format to the encoding stage includes: transmitting the frame data of the target format to the encoding stage using a second storage area; the method further includes: in the encoding stage, after obtaining the frame data of the target format, notifying the conversion stage to delete the frame data of the target format from the second storage area; The step of transmitting the at least one frame of semantic representation data to the inference stage includes: transmitting the at least one frame of semantic representation data to the inference stage using a third storage area; the method further includes: during the inference stage, after obtaining each at least one frame of semantic representation data, notifying the encoding stage to delete the at least one frame of semantic representation data from the third storage area.

[0088] Optionally, inputting the at least one frame of semantic representation data into the multimodal large model for inference includes: The at least one frame of semantic representation data is input into the multimodal large model for inference. During the inference process, a linear transformation is performed on the at least one frame of semantic representation data to calculate the newly added key vector and the newly added value vector, and the stored key vector and the stored value vector are read from the key-value cache. Based on the stored key vectors and value vectors, as well as the newly added key vectors and value vectors, inference is performed to obtain the inference result, and the newly added key vectors and value vectors are stored in the key-value cache.

[0089] Optionally, storing the newly added key vector and the newly added value vector in the key-value cache includes: if the key-value cache reaches a set length, clearing the oldest stored key vector and value vector, storing the newly added key vector and the newly added value vector in the key-value cache, and recording the storage time of the newly added key vector and the newly added value vector.

[0090] Optionally, the method further includes: recording processing progress information during video processing; wherein the video frame corresponds to a frame number, and the processing progress information includes the frame number of the video frame that has just completed inference; after video processing is interrupted and resumed, acquiring the processing progress information, determining the remaining video frames in the video to be processed based on the processing progress information, and continuing to execute the decoding stage, the conversion stage, the encoding stage, and the inference stage on the remaining video frames.

[0091] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0092] The communication interface is used for communication between the aforementioned terminal and other devices.

[0093] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0094] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0095] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0096] In another embodiment of the present invention, a readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements any of the video processing methods described in the above embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0097] In another embodiment of the present invention, a computer program product is also provided, which stores a computer program. When the computer program is executed by a processor, it implements any of the video processing methods described in the above embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0098] The various embodiments in this specification are related to each other and are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0099] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0100] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0101] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0102] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A video processing method, characterized in that, The method includes: During the decoding stage, video frames in the video to be processed are read frame by frame. Each time a video frame is read, it is decoded to obtain a frame of original format data. Each time a frame of original format data is obtained, it is transmitted to the conversion stage. During the conversion phase, each time a frame of original format data is obtained, the frame of original format data is converted into a frame of target format data required by the multimodal large model. Each time a frame of target format data is obtained, the frame of target format data is transmitted to the encoding phase. During the encoding stage, each time a frame of target format data is obtained, the frame of target format data is encoded to obtain a frame of semantic representation data. Each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is transmitted to the inference stage. During the inference phase, each time at least one frame of semantic representation data is obtained, the at least one frame of semantic representation data is input into the multimodal large model for inference, and the inference result output by the multimodal large model is obtained.

2. The method according to claim 1, characterized in that: The step of transmitting the frame data in the original format to the conversion stage includes: transmitting the frame data in the original format to the conversion stage using a first storage area; the method further includes: in the conversion stage, after obtaining the frame data in the original format each time, notifying the decoding stage to delete the frame data in the original format from the first storage area; The step of transmitting the frame data of the target format to the encoding stage includes: transmitting the frame data of the target format to the encoding stage using a second storage area; the method further includes: in the encoding stage, after obtaining the frame data of the target format, notifying the conversion stage to delete the frame data of the target format from the second storage area; The step of transmitting the at least one frame of semantic representation data to the inference stage includes: transmitting the at least one frame of semantic representation data to the inference stage using a third storage area; the method further includes: during the inference stage, after obtaining each at least one frame of semantic representation data, notifying the encoding stage to delete the at least one frame of semantic representation data from the third storage area.

3. The method according to claim 1, characterized in that, The step of inputting the at least one frame of semantic representation data into the multimodal large model for inference includes: The at least one frame of semantic representation data is input into the multimodal large model for inference. During the inference process, a linear transformation is performed on the at least one frame of semantic representation data to calculate the newly added key vector and the newly added value vector, and the stored key vector and the stored value vector are read from the key-value cache. Based on the stored key vectors and value vectors, as well as the newly added key vectors and value vectors, inference is performed to obtain the inference result, and the newly added key vectors and value vectors are stored in the key-value cache.

4. The method according to claim 3, characterized in that, The step of storing the newly added key vector and the newly added value vector into the key-value cache includes: If the key-value cache reaches a set length, the oldest stored key vector and value vector are cleared, and the newly added key vector and value vector are stored in the key-value cache, and the storage time of the newly added key vector and value vector is recorded.

5. The method according to claim 1, characterized in that, The method further includes: During video processing, processing progress information is recorded; wherein, each video frame corresponds to a frame number, and the processing progress information includes the frame number of the video frame that has just completed inference; After the video processing is resumed from interruption, the processing progress information is obtained, the remaining video frames in the video to be processed are determined based on the processing progress information, and the decoding stage, the conversion stage, the encoding stage and the inference stage are continued to be executed on the remaining video frames.

6. A video processing apparatus, characterized in that, The device includes: The decoding module is used to read video frames from the video to be processed frame by frame during the decoding stage. For each video frame read, the video frame is decoded to obtain a frame of original format frame data. For each frame of original format frame data obtained, the frame of original format frame data is transmitted to the conversion stage. The conversion module is used to convert each frame of original format data into a frame of target format data required by the multimodal large model during the conversion stage, and to transmit the frame of target format data to the encoding stage after each frame of target format data is obtained. The encoding module is used to encode each frame of target format data obtained in the encoding stage to obtain a frame of semantic representation data, and to transmit the at least one frame of semantic representation data to the inference stage after obtaining at least one frame of semantic representation data. The inference module is used to, during the inference phase, input the at least one frame of semantic representation data into the multimodal large model for inference, and obtain the inference result output by the multimodal large model.

7. The apparatus according to claim 6, characterized in that, The reasoning module includes: The cache acquisition unit is used to input the at least one frame of semantic representation data into the multimodal large model for inference. During the inference process, the at least one frame of semantic representation data is subjected to linear transformation to calculate the newly added key vector and the newly added value vector, and the stored key vector and the stored value vector are read from the key-value cache. The inference storage unit is used to perform inference based on the stored key vectors and stored value vectors, as well as the newly added key vectors and newly added value vectors, to obtain the inference result, and to store the newly added key vectors and newly added value vectors in the key-value cache.

8. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a computer program stored in memory, implements the method as described in any one of claims 1-5.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

10. A computer program product, characterized in that, The computer program product stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-5.

Images