Video processing method and device based on multi-modal large model, equipment and medium

Abstract

The invention relates to the technical field of computer vision, in particular to a video processing method and device based on a multi-modal large model, equipment and a medium. When the multi-modal large model is used for processing the video, not only is the feature of each image frame in the video input into the large language model in the multi-modal large model, but also the feature corresponding to the key sub-image in the to-be-promoted image frame is input into the large language model. The information included in the key sub-image can be clear when the large language model is processed, so that the video processing accuracy of the multi-mode large model is improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a video processing method, apparatus, device and medium based on a multimodal large model. Background Art

[0002] With the development of multimodal large-scale model technology, more and more industries are using these models for video processing. However, due to the limitations of the model architecture, the number of input tokens is limited. Therefore, the resolution of each video frame input to the multimodal large-scale model cannot be too high. Generally, each image frame is compressed to a resolution of 448*448. However, low-resolution image frames cannot well preserve image details, which affects the processing results of the multimodal large-scale model.

[0003] Therefore, how to improve the accuracy of video processing by large multimodal models has become an urgent problem to be solved. Summary of the Invention

[0004] The embodiments of the present application provide a video processing method, apparatus, device and medium based on a multimodal large model to solve the problem of low accuracy of video processing by multimodal large models in the prior art.

[0005] The present application provides a video processing method based on a multimodal large model, wherein the multimodal large model includes an input layer, a visual encoder, a feature processing layer, a multilayer perceptron, and a large language model. The method includes:

[0006] The input layer selects at least one image frame from the received video as the image frame to be enhanced, splits the image frame to be enhanced into multiple sub-images; adjusts each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution, and then inputs them into the visual encoder to obtain a first feature matrix corresponding to each image;

[0007] The feature processing layer selects a preset number of key sub-images from the plurality of sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; inputs the first feature matrix corresponding to each key sub-image, the image frame to be enhanced, and each of the other image frames into the multilayer perceptron for feature mapping to obtain a third feature matrix;

[0008] The third feature matrix and the second feature matrix are input into the large language model to obtain a processing result corresponding to the video.

[0009] In the embodiment of the present application, the input layer of the multimodal large model selects at least one image frame in the received video as the image frame to be enhanced, and splits the image frame to be enhanced into multiple sub-images; each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a preset resolution and input into the visual encoder to obtain a first feature matrix corresponding to each image; the feature processing layer then selects a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; each key sub-image, the image frame to be enhanced, and the first feature matrix corresponding to each other image frame are input into the multilayer perceptron for feature mapping to obtain a third feature matrix; the third feature matrix and the second feature matrix are input into the large language model to obtain the processing result corresponding to the video. When the multimodal large model is used to process the video, not only the features of each image frame in the video are input into the large language model in the multimodal large model, but also the features corresponding to the key sub-image in the image frame to be enhanced are input into the large language model, so that the large language model can clearly understand the information included in the key sub-image during processing, thereby improving the accuracy of the multimodal large model in processing the video.

[0010] Furthermore, the input layer selects at least one image frame from the received video as the image frame to be enhanced, including:

[0011] The input layer selects a preset number of candidate image frames in the video; for each candidate image frame, determines a first similarity between the candidate image frame and other candidate image frames; based on each first similarity, determines a second similarity corresponding to the candidate image frame; and determines the candidate image frame corresponding to the highest second similarity as the image frame to be improved.

[0012] The above technical solution selects a candidate image frame with the highest similarity to other candidate image frames from multiple candidate image frames as the image frame to be improved, which reduces resource usage and improves the efficiency of video processing by the multimodal large model.

[0013] Furthermore, splitting the image frame to be enhanced into a plurality of sub-images includes:

[0014] Determining an adjusted first resolution according to the first split ratio and the preset resolution;

[0015] Adjusting the resolution of the image frame to be improved based on the first resolution;

[0016] The image frame to be enhanced after the resolution adjustment is split into a plurality of sub-images according to the first splitting ratio.

[0017] The above technical solution adjusts the resolution of the image frame to be enhanced based on the first resolution, which can ensure that the resolution of each sub-image obtained by splitting based on the first splitting ratio is consistent, and can further ensure the accuracy of video processing.

[0018] Furthermore, the process of determining the first split ratio includes:

[0019] Determining a pixel ratio between the number of pixels in width and the number of pixels in height of the image frame to be enhanced;

[0020] determining the deviation between the pixel ratio and each candidate split ratio respectively;

[0021] The candidate split ratio corresponding to the minimum deviation is determined as the first split ratio.

[0022] In the above technical solution, when determining the first split ratio, the candidate split ratio that is closest to the pixel ratio of the image frame to be improved is selected from the candidate split ratios as the first split ratio, and adjustments are made based on the first split ratio. This can ensure that the resolution of the candidate split image frame is adjusted as little as possible, thereby ensuring to the greatest extent that the image content in the candidate split image frame is not affected, thereby improving the accuracy of video processing.

[0023] Furthermore, the process of determining each candidate split ratio includes:

[0024] Each candidate split ratio is constructed according to a preset minimum number of blocks and a preset maximum number of blocks, wherein the product of the two values ​​in any candidate split ratio is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks.

[0025] The above technical solution constructs each candidate split ratio according to the preset minimum number of blocks and the preset maximum number of blocks, which can ensure that the number of sub-images obtained when the image frame to be enhanced is subsequently split is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks, thereby reducing the number of tokens processed by the multimodal large model.

[0026] Furthermore, after splitting the image frame to be enhanced into a plurality of sub-images, and before adjusting each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and inputting them into the visual encoder, the method further includes:

[0027] The input layer selects a second split ratio from each pre-saved candidate split ratio, and determines an adjusted second resolution according to the second split ratio and a preset resolution;

[0028] Adjusting the resolution of the image frame to be improved based on the second resolution;

[0029] The image frame to be enhanced after the resolution adjustment is split into multiple sub-images according to the second splitting ratio; wherein the splitting traces of the image frame to be enhanced after the resolution adjustment is split according to the second splitting ratio do not overlap with the splitting traces corresponding to the sub-images that have been obtained.

[0030] The above technical solution ensures that the splitting traces of the image frame to be improved after the resolution is adjusted according to the second splitting ratio do not overlap with the splitting traces corresponding to the sub-images that have been obtained. In this way, the places separated by the first splitting can be supplemented in the second splitting process, alleviating the boundary jagged effect.

[0031] Furthermore, the feature processing layer selects a preset number of key sub-images from the plurality of sub-images according to the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image, including:

[0032] The feature processing layer determines the weight corresponding to each sub-image based on the second feature matrix and position code corresponding to the received prompt text, the first feature matrix and position code corresponding to each image, the model dimension, and the weight calculation formula;

[0033] According to the weight corresponding to each sub-image, a preset number of key sub-images are selected from each sub-image.

[0034] The above technical solution determines the weight corresponding to each sub-image based on the weight calculation formula, and then selects a preset number of key sub-images in each sub-image according to the weight corresponding to each sub-image. Subsequently, the large language model can pay more attention to the information in the key sub-images during the video processing process, reducing the computing resources of the large language model and thus improving the processing efficiency of the video.

[0035] The present application also provides a video processing device based on a multimodal large model, wherein the multimodal large model includes an input layer, a visual encoder, a feature processing layer, a multilayer perceptron, and a large language model, and the device includes:

[0036] a splitting module, configured for the input layer to select at least one image frame from the received video as an image frame to be enhanced, and split the image frame to be enhanced into a plurality of sub-images;

[0037] a feature extraction module, configured to adjust each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and input the resolutions into the visual encoder to obtain a first feature matrix corresponding to each image;

[0038] a screening module configured to select a preset number of key sub-images from the plurality of sub-images according to the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image in the feature processing layer;

[0039] a feature mapping module, configured to input the first feature matrix corresponding to each of the key sub-images, the image frame to be enhanced, and each of the other image frames into the multilayer perceptron for feature mapping to obtain a third feature matrix;

[0040] An inference module is used to input the third feature matrix and the second feature matrix into the large language model to obtain a processing result corresponding to the video.

[0041] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned video processing methods based on a multimodal large model when executing a computer program stored in a memory.

[0042] The present application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned video processing methods based on a multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 A schematic diagram of a process flow of a video processing process based on a multimodal large model provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of splitting an image frame to be enhanced provided in an embodiment of the present application;

[0046] Figure 3a An image splitting diagram provided in the embodiment of the present application Figure 1 ;

[0047] Figure 3b An image splitting diagram provided in the embodiment of the present application Figure 2 ;

[0048] Figure 4 A schematic diagram of a video processing process of a multimodal large model provided in an embodiment of the present application;

[0049] Figure 5A schematic diagram of the structure of a video processing device based on a multimodal large model provided in an embodiment of the present application;

[0050] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the technical solutions of the embodiments of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0052] An embodiment of the present application provides a video processing method, apparatus, device, and medium based on a multimodal large model. In the method, the multimodal large model includes an input layer, a visual encoder, a feature processing layer, a multilayer perceptron, and a large language model. The method includes: the input layer selects at least one image frame in the received video as an image frame to be enhanced, and splits the image frame to be enhanced into multiple sub-images; each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a preset resolution and input into the visual encoder to obtain a first feature matrix corresponding to each image; the feature processing layer selects a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; each key sub-image, the image frame to be enhanced, and the first feature matrix corresponding to each other image frame are input into the multilayer perceptron for feature mapping to obtain a third feature matrix; the third feature matrix and the second feature matrix are input into the large language model to obtain a processing result corresponding to the video.

[0053] Figure 1 A flowchart of a video processing process based on a multimodal large model is provided in an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0054] S101: The input layer selects at least one image frame in the received video as the image frame to be enhanced, and splits the image frame to be enhanced into multiple sub-images; each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a preset resolution and input into the visual encoder to obtain a first feature matrix corresponding to each image.

[0055] The video processing method based on a multimodal large model provided in this application is applied to an electronic device, which may be a server, a PC, a mobile terminal, etc.

[0056] In order to accurately understand the information in the video, in an embodiment of the present application, a multimodal large model can be used to process the video. When processing the video, the multimodal large model can know the content included in the video, so as to perform corresponding analysis on the video according to the user's requirements and output the results expected by the user. Among them, the multimodal large model may include an input layer, a visual encoder, a feature processing layer, a multi-layer perceptron and a large language model. Among them, the video can be input into the electronic device by the user and then sent to the multimodal large model by the electronic device, or it can be sent to the electronic device by other electronic devices and then sent to the multimodal large model by the electronic device. The content included in the video can be arbitrary, and technicians in this field can configure the method of obtaining the video as needed. After the electronic device obtains the video, it can input the video into the multimodal large model so that the multimodal large model processes the video.

[0057] Due to the redundancy of video information, there is a great similarity between adjacent image frames. If all image frames in the video are processed by resolution enhancement, resource usage will be increased and reasoning time will be prolonged. Therefore, in the embodiment of the present application, it is proposed to use a combination of high resolution of some image frames + low resolution of some image frames to enhance the ability of the multimodal large model to process high-resolution videos. After receiving the video, the input layer of the multimodal large model can select at least one image frame in the video as the image frame to be enhanced. In other words, the image frame to be enhanced can be one or more. When selecting the image frame to be enhanced, any one or more frames can be randomly selected in the video as the image frame to be enhanced, or one image frame can be selected as the image frame to be enhanced at every preset number of frames in the video.

[0058] After obtaining the image frame to be enhanced, the image frame to be enhanced can be split into multiple sub-images. When splitting the image frame to be enhanced, the image frame to be enhanced can be split into a preset number of sub-images or according to a preset ratio.

[0059] After obtaining each sub-image, features of each image can be extracted. In an embodiment of the present application, each sub-image, the image frame to be improved, and other image frames in the video can be adjusted to a preset resolution and then input into a visual encoder to obtain a first feature matrix corresponding to each image. The first feature matrix can be a row with multiple columns, a column with multiple rows, or multiple rows and multiple columns. How the visual encoder extracts features from an image is a prior art, and the embodiments of the present application will not be described in detail. Among them, the preset resolution can be any resolution, for example, 448*448, and those skilled in the art can configure it according to the performance requirements of the multimodal large model. Compared with the prior art that compresses each image frame to a preset resolution, the present application also adjusts the resolution of each sub-image to the preset resolution. In most cases, the resolution of the sub-image is improved. Even if the resolution of the sub-image is compressed during the resolution adjustment process, the compression is small, and more image details can be retained compared to the prior art.

[0060] S102: The feature processing layer selects a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; and inputs the first feature matrix corresponding to each key sub-image, the image frame to be enhanced, and each of the other image frames into the multilayer perceptron for feature mapping to obtain a third feature matrix.

[0061] Since the resolution of some image frames in the video is improved, the number of tokens processed by the multimodal large model is also expanded. In addition, there is a large amount of redundant information in different image frames of the video. In the process of video processing, more attention needs to be paid to the dynamic changes between frames. Therefore, in an embodiment of the present application, some important sub-images can be selected from the multiple sub-images that are split as key sub-images, and then processed by the large language model in the multimodal large model. In an embodiment of the present application, after obtaining the first feature matrix corresponding to each image, the feature processing layer of the multimodal large model can select a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code. Among them, the prompt text is a text used to prompt the multimodal large model what kind of processing to perform on the received video. The prompt text can be received by the multimodal large model together with the video. After obtaining the prompt text, the multimodal large model can use the text encoder to extract features from the prompt text, obtain the second feature matrix corresponding to the prompt text, and input it into the feature processing layer. The feature processing layer can analyze the features and positions of each sub-image according to the requirements in the second feature matrix, and select a preset number of key sub-images that the multimodal large model considers important from the multiple sub-images. For example, in general, the content recorded in the image area located in the center of the picture is generally more important, so the feature processing layer can obtain the sub-image corresponding to the preset key area according to the position encoding of each sub-image, and determine whether the content included in the sub-image is related to the content described in the prompt text based on the first feature matrix of the sub-image corresponding to the key area. If so, the sub-image can be determined as a key sub-image. If not, other images adjacent to the image corresponding to the key area can be obtained, and it can be continued to be determined whether the content included in the other images is related to the content described in the prompt text. If so, the corresponding sub-image can be determined as a key sub-image. If not, other adjacent images can be obtained, and the same process can be repeated to determine whether the content included in the other images is related to the content described in the prompt text. If so, the corresponding sub-image can be determined as a key sub-image. If not, other adjacent images can be obtained, and so on.

[0062] After determining the key sub-images, the first feature matrix corresponding to each key sub-image, the image frame to be enhanced, and each other image frame in the video can be input into a multi-layer perceptron for feature mapping to obtain a third feature matrix. In other words, feature mapping is performed on the first feature matrix corresponding to each key sub-image and all image frames in the video to obtain a third feature matrix. In other words, the multi-layer perceptron performs feature mapping on the first feature matrix according to the input requirements of the large language model to obtain a third feature matrix. Before performing feature mapping, all feature matrices can be spliced ​​in a preset order, and feature mapping can be performed on the spliced ​​feature matrices.

[0063] S103: Input the third feature matrix and the second feature matrix into the large language model to obtain a processing result corresponding to the video.

[0064] After obtaining the third feature matrix, the third feature matrix and the second feature matrix can be input into the large language model, and the large language model of the multimodal large model performs corresponding processing on the third feature matrix based on the information included in the second feature matrix corresponding to the prompt text to obtain the processing result corresponding to the video. In other words, the feature information included in the video is analyzed and processed based on the requirements of the prompt text. Since the third feature matrix not only includes the features corresponding to each image frame in the video, but also includes the features corresponding to the key sub-image, and the features corresponding to the key sub-image are extracted based on the image after resolution enhancement or a smaller degree of resolution compression, it will contain more details, which can improve the accuracy of video processing.

[0065] In the embodiment of the present application, the input layer of the multimodal large model selects at least one image frame in the received video as the image frame to be enhanced, and splits the image frame to be enhanced into multiple sub-images; each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a preset resolution and input into the visual encoder to obtain a first feature matrix corresponding to each image; the feature processing layer then selects a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; each key sub-image, the image frame to be enhanced, and the first feature matrix corresponding to each other image frame are input into the multilayer perceptron for feature mapping to obtain a third feature matrix; the third feature matrix and the second feature matrix are input into the large language model to obtain the processing result corresponding to the video. When the multimodal large model is used to process the video, not only the features of each image frame in the video are input into the large language model in the multimodal large model, but also the features corresponding to the key sub-image in the image frame to be enhanced are input into the large language model, so that the large language model can clearly understand the information included in the key sub-image during processing, thereby improving the accuracy of the multimodal large model in processing the video.

[0066] In order to further improve the accuracy of video processing, based on the above embodiment, in the embodiment of the present application, the input layer selects at least one image frame in the received video as the image frame to be improved, including:

[0067] The input layer selects a preset number of candidate image frames in the video; for each candidate image frame, determines a first similarity between the candidate image frame and other candidate image frames; based on each first similarity, determines a second similarity corresponding to the candidate image frame; and determines the candidate image frame corresponding to the highest second similarity as the image frame to be improved.

[0068] In an embodiment of the present application, when the input layer selects image frames to be improved in a video, it may select a preset number of candidate image frames from the video. The preset number is any integer, such as 5, 15, 50, 100, etc. Specifically, a preset number of key image frames may be selected from the video as candidate image frames based on a key frame extraction model, or one image frame may be selected as a candidate image frame every predetermined number of frames in the video.

[0069] After obtaining each candidate image frame, a first similarity between the candidate image frame and other candidate image frames can be determined for each candidate image frame. For example, the first similarity can be calculated based on the following cosine similarity calculation formula:

[0070]

[0071] Among them, similarity represents the first similarity, A represents the feature matrix corresponding to the candidate image frame, B represents the feature matrix corresponding to any other candidate image frame, and n represents the number of items included in the feature matrix. Then, A i It represents the value of the i-th item in the feature matrix corresponding to the candidate image frame, B i It represents the value of the i-th item in the feature matrix corresponding to any other candidate image frame.

[0072] After obtaining each first similarity, a second similarity corresponding to the candidate image frame can be determined based on each first similarity. For example, the second similarity can be determined as the sum of all first similarities corresponding to the candidate image frame, or as the average of all first similarities corresponding to the candidate image frame.

[0073] After obtaining the second similarity corresponding to each candidate image frame, the candidate image frame corresponding to the highest similarity may be determined as the image frame to be improved.

[0074] In an embodiment of the present application, one image frame in the video is used as the image frame to be improved, and the other image frames in the video are compressed to a low resolution of 448*448. This not only improves the video resolution, but also improves the multimodal large model's ability to perceive the detailed content of the video, and saves the number of tokens processed by the multimodal large model, reduces resource usage, and improves the inference speed.

[0075] In order to further improve the accuracy of video processing, based on the above embodiments, in the embodiment of the present application, the image frame to be enhanced is split into multiple sub-images, including:

[0076] Determining an adjusted first resolution according to the first split ratio and the preset resolution;

[0077] Adjusting the resolution of the image frame to be improved based on the first resolution;

[0078] The image frame to be enhanced after the resolution adjustment is split into a plurality of sub-images according to the first splitting ratio.

[0079] In order to ensure that the resolution size of each sub-image obtained by splitting is the same, in an embodiment of the present application, when splitting the image frame to be improved into multiple sub-images, the adjusted first resolution can be determined based on the first splitting ratio and the preset resolution. Among them, the first splitting ratio can be pre-configured based on experience, for example, 12:1, 4:3, 3:3, etc. Among them, 12:1 can be understood as splitting the image to be improved into 12 parts in proportion on the width, and splitting the image in height into 1 part, that is, not performing the operation on the height of the image frame to be improved. 4:3 can be understood as splitting the image to be improved into 4 parts in proportion on the width, and splitting the image in height into 3 parts, then the image frame to be improved is split into 12 sub-images.

[0080] Specifically, assuming that the image size of the image frame to be enhanced is 800*1300, the first split ratio is 2:3, and the preset resolution is 448*448, then the adjusted first resolution is M*N, where M=896=2*448 and N=1344=3*448.

[0081] After obtaining the adjusted first resolution, the resolution of the image frame to be enhanced can be adjusted to the first resolution. For example, the resolution of the image frame to be enhanced with a resolution of 800*1300 is adjusted to 896*1344.

[0082] After the resolution of the image frame to be enhanced is adjusted, the image frame to be enhanced with the adjusted resolution may be split into a plurality of sub-images according to a first splitting ratio.

[0083] Specifically, if the image frame to be upscaled is split according to a first splitting ratio of 4:3, the image frame to be upscaled may be split into 12 sub-images. Figure 2 A schematic diagram of splitting an image frame to be enhanced provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the width of the image frame to be enhanced is divided into 4 parts in equal proportion, and the height of the image frame to be enhanced is divided into 3 parts in equal proportion.

[0084] In order to further improve the accuracy of video processing, based on the above embodiments, in the embodiment of the present application, the process of determining the first split ratio includes:

[0085] Determining a pixel ratio between the number of pixels in width and the number of pixels in height of the image frame to be enhanced;

[0086] determining the deviation between the pixel ratio and each candidate split ratio respectively;

[0087] The candidate split ratio corresponding to the minimum deviation is determined as the first split ratio.

[0088] Since the resolutions of the image frames to be improved in different videos are different, some have high resolutions and some have low resolutions, if the image frames to be improved are all split according to the same splitting ratio, the details in the image frames to be improved may not be well displayed. Therefore, in an embodiment of the present application, a dynamic resolution method can be used to process the image frames to be improved. In an embodiment of the present application, a plurality of candidate splitting ratios can be pre-configured, and the candidate splitting ratios can be determined by relevant personnel based on experience. In order to ensure that the pixel ratio of the image frame to be improved remains unchanged as much as possible, so as to retain the content included in the image frame to be improved as much as possible. In an embodiment of the present application, the candidate splitting ratio that is closest to the pixel ratio of the image frame to be improved can be selected from the candidate splitting ratios as the first splitting ratio.

[0089] When determining the first split ratio, the pixel ratio between the number of pixels in the width and the number of pixels in the height of the image frame to be enhanced can be determined. After obtaining the pixel ratio, the deviation between the pixel ratio and each candidate split ratio can be determined, and the candidate split ratio corresponding to the minimum deviation is determined as the first split ratio.

[0090] In order to further improve the efficiency of video processing, based on the above embodiments, in the embodiment of the present application, the process of determining each candidate split ratio includes:

[0091] Each candidate split ratio is constructed according to a preset minimum number of blocks and a preset maximum number of blocks, wherein the product of the two values ​​in any candidate split ratio is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks.

[0092] Due to the limitations of the model architecture of the multimodal large model, the number of input tokens is limited. If the image frame to be improved is split into too many sub-images, the amount of calculation will increase significantly and the efficiency will be low. Therefore, in an embodiment of the present application, a preset minimum number of blocks and a preset maximum number of blocks can be saved. The preset minimum number of blocks and the preset maximum number of blocks are empirical values, and the most appropriate and reasonable number of blocks can be summarized based on multiple experiments. For example, the preset minimum number of blocks can be 1, and the preset maximum number of blocks can be 12.

[0093] In an embodiment of the present application, each candidate split ratio can be constructed based on the preset minimum number of blocks and the preset maximum number of blocks, and the product of the two values ​​in any constructed candidate split ratio is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks.

[0094] Specifically, assuming that the preset minimum number of blocks is 1 and the preset maximum number of blocks is 12, the candidate split ratios constructed can be 12:1, 3:4, 4:3, 3:1, 2:3, etc. If the resolution of the image frame to be enhanced is 800*1300, then the first split ratio selected from these candidate split ratios is 2:3, and therefore, this first split ratio is closest to 800:1300.

[0095] In order to further improve the accuracy of video processing, based on the above embodiments, in an embodiment of the present application, after splitting the image frame to be enhanced into multiple sub-images, and before adjusting each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and inputting them into the visual encoder, the method further includes:

[0096] The input layer selects a second split ratio from each pre-saved candidate split ratio, and determines an adjusted second resolution according to the second split ratio and a preset resolution;

[0097] Adjusting the resolution of the image frame to be improved based on the second resolution;

[0098] The image frame to be enhanced after the resolution adjustment is split into multiple sub-images according to the second splitting ratio; wherein the splitting traces of the image frame to be enhanced after the resolution adjustment is split according to the second splitting ratio do not overlap with the splitting traces corresponding to the sub-images that have been obtained.

[0099] Since the image frame to be enhanced is split into different sub-images when it is split, it is inevitable that some continuous semantic information will be divided into different sub-images, for example, the same character is split into different sub-images. Therefore, in an embodiment of the present application, a multi-scale splitting method is also provided. After the image frame to be enhanced is split into multiple sub-images, before each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a preset resolution and input into the visual encoder, the image frame to be enhanced can be split again using a second splitting ratio. The second splitting ratio is different from the first splitting ratio.

[0100] In an embodiment of the present application, the input layer can select a second split ratio from each pre-saved candidate split ratio, thereby determining an adjusted second resolution based on the second split ratio and a preset resolution, and adjusting the resolution of the image frame to be enhanced based on the second resolution, and then splitting the image frame to be enhanced after the resolution adjustment into multiple sub-images according to the second split ratio. The process of determining the second resolution is similar to the process of determining the first resolution, and the process of splitting the image frame to be enhanced after the resolution adjustment according to the second split ratio is similar to the process of splitting the image frame to be enhanced after the resolution adjustment according to the first split ratio, and this embodiment of the present application will not be repeated.

[0101] In this embodiment of the present application, the splitting traces of the image frame to be enhanced after the resolution adjustment according to the second splitting ratio do not overlap with the splitting traces corresponding to the already obtained sub-images. In other words, even if a character in the already obtained sub-image is split into two sub-images, the second splitting with different traces can ensure that the character exists in one sub-image. Figure 3a An image splitting diagram provided in the embodiment of the present application Figure 1 , Figure 3b An image splitting diagram provided in the embodiment of the present application Figure 2 .like Figure 3a As shown in , the image to be enhanced is split into 6 sub-images, such as Figure 3b As shown in the figure, the image to be enhanced is split into 12 sub-images. Figure 3a and Figure 3b From the split traces of each sub-image in , we can see that any split traces are non-overlapping. The grid and straight lines in the figure are the split traces.

[0102] Specifically, in order to prevent the split traces from different times from overlapping, when selecting the second split ratio, the candidate split ratios that overlap with or are divisible by the first split ratio can be eliminated from the candidate split ratios. For example, if the minimum number of blocks is preset to 1, the maximum number of blocks is preset to 12, and the first split ratio is 2:3, then the remaining candidate split ratios after elimination are 3:2, 4:2, 5:2, 3:4, and 6:2. Then, among the remaining candidate split ratios, the most appropriate candidate split ratio is selected as the second split ratio according to the process of determining the first split ratio. The second split ratio can be 3:4.

[0103] In the embodiment of the present application, it is ensured that the split traces of the second split image and the first split do not overlap, so that the separated areas in the first split can be supplemented in the second split process, alleviating the boundary jagged effect.

[0104] To further improve the efficiency of video processing, based on the above embodiments, in an embodiment of the present application, the feature processing layer selects a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image, including:

[0105] The feature processing layer determines the weight corresponding to each sub-image based on the second feature matrix and position code corresponding to the received prompt text, the first feature matrix and position code corresponding to each image, the model dimension, and the weight calculation formula;

[0106] According to the weight corresponding to each sub-image, a preset number of key sub-images are selected from each sub-image.

[0107] In an embodiment of the present application, when selecting a preset number of key sub-images, the feature processing layer can determine the weight corresponding to each sub-image based on the second feature matrix and position code corresponding to the received prompt text, the first feature matrix and position code corresponding to each image, the model dimension, and the weight calculation formula.

[0108] The weight calculation formula can be expressed as follows:

[0109]

[0110] Among them, att represents the weight matrix; softmax() represents the normalized exponential function; Vb represents the matrix obtained by splicing the first feature matrices corresponding to all sub-images in a preset order; Vo represents the first feature matrix corresponding to other image frames in the video; Vg represents the first feature matrix corresponding to the image frame to be improved; Vt represents the second feature matrix corresponding to the prompt text; cat() represents the matrix splicing function, then cat(Vo, Vg, Vt) represents splicing the feature matrices Vo, Vg, and Vt in sequence; PE() represents position encoding, then PE(cat(Vo, Vg, Vt)) represents the encoding matrix corresponding to the spliced ​​feature matrix. How to determine the position encoding is an existing technology. The process of determining the position encoding can refer to the process of determining the position encoding in ViT (Vision Transformer), and the embodiments of this application will not go into details about this; D represents the dimension of the large language model, that is, the dimension of the hidden layer of the large language model.

[0111] Specifically, assuming there are currently three sub-images, each corresponding to a first characteristic matrix of 36*256 dimensions, then Vb, obtained by concatenating the first characteristic matrices corresponding to all sub-images in a preset order, is a matrix of (3*36)*256 dimensions. In other words, the matrix corresponding to rows 1-36 in Vb is the first characteristic matrix corresponding to the first sub-image, the matrix corresponding to rows 37-72 is the first characteristic matrix corresponding to the second sub-image, and the matrix corresponding to rows 73-118 is the first characteristic matrix corresponding to the third sub-image. Those skilled in the art can configure the method for concatenating to obtain Vb as needed.

[0112] The value corresponding to each item in the weight matrix att represents the weight of the corresponding sub-image. After obtaining the above weight matrix att, all items in att can be counted and the first preset number of sub-images with the largest weights can be determined as key sub-images.

[0113] In an embodiment of the present application, an adaptive visual token compression mechanism is proposed, which takes the second feature matrix Vt of the prompt text input by the user, the first feature matrix Vo (V1, V2, ... Vn) of other image frames in the video, and the first feature matrix Vg of each sub-image as conditions, and performs inner product with the first feature matrix Vb corresponding to the image frame to be improved. The values ​​are normalized to between 0 and 1 by calculating the softmax function, and the sub-images corresponding to the topk values ​​are selected as the key sub-images finally selected.

[0114] The video processing process of the multimodal large model is described below with reference to a specific embodiment. Figure 4 A schematic diagram of a video processing process of a multimodal large model provided in an embodiment of the present application is shown as follows: Figure 4 As shown, first, an image frame is selected from the video as the image frame to be enhanced, and dynamic resolution processing is performed on the image frame to be enhanced. Dynamic resolution processing is the process of processing the image frame to be enhanced after the resolution adjustment according to the first and second split ratios as described in the above embodiments. Simultaneously, the prompt text entered by the user is input into the text encoder to obtain a second feature matrix corresponding to the prompt text.

[0115] After obtaining each sub-image, each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a resolution of 448*448 and then input into the visual encoder to obtain the first feature matrix corresponding to each image.

[0116] After obtaining the first feature matrix corresponding to each sub-image, a preset number of sub-images are selected from the multiple sub-images as key sub-images based on an adaptive compression mechanism. The adaptive compression mechanism is the step described in the above embodiments where the feature processing layer selects a preset number of key sub-images from the multiple sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code.

[0117] After the key sub-image is determined, the first feature matrix corresponding to the key sub-image, the image frame to be enhanced, and each other image frame may be input into the multilayer perceptron for feature mapping to obtain a third feature matrix.

[0118] After obtaining the third feature matrix, the third feature matrix and the second feature matrix are input into the large language model to obtain a processing result corresponding to the video.

[0119] In the embodiment of the present application, a dynamic resolution solution is used to expand the resolution of a single image frame. In combination with other image frames whose resolutions are not expanded, an adaptive visual token compression mechanism is adopted to determine key sub-images, thereby reducing the total number of tokens processed by the large language model, improving processing speed and reducing resource usage.

[0120] In the embodiments of this application, multi-scale dynamic resolution improves the ability of a large multimodal model to process high-resolution single-frame images, reducing the jagged effect caused by blocking. Furthermore, the combination of a single high-resolution frame and multiple low-resolution frames enhances the large multimodal model's ability to process high-resolution video. An adaptive visual token compression mechanism reduces the total number of tokens processed by the large multimodal video model, improving processing speed and reducing resource usage.

[0121] Figure 5 A schematic diagram of the structure of a video processing device based on a multimodal large model provided in an embodiment of the present application, wherein the multimodal large model includes an input layer, a visual encoder, a feature processing layer, a multilayer perceptron and a large language model, such as Figure 5 As shown, the device includes:

[0122] A splitting module 501 is configured to select at least one image frame from the received video as an image frame to be enhanced at the input layer, and split the image frame to be enhanced into a plurality of sub-images;

[0123] A feature extraction module 502 is configured to adjust each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and input the resolutions to the visual encoder to obtain a first feature matrix corresponding to each image.

[0124] A screening module 503 is configured for the feature processing layer to select a preset number of key sub-images from the plurality of sub-images based on the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image;

[0125] A feature mapping module 504 is configured to input the first feature matrix corresponding to each of the key sub-images, the image frame to be enhanced, and each of the other image frames into the multilayer perceptron for feature mapping to obtain a third feature matrix;

[0126] The inference module 505 is configured to input the third feature matrix and the second feature matrix into the large language model to obtain a processing result corresponding to the video.

[0127] In one possible implementation, the splitting module 501 is specifically configured to select a preset number of candidate image frames from the video at the input layer; determine, for each candidate image frame, a first similarity between the candidate image frame and other candidate image frames; determine, based on each first similarity, a second similarity corresponding to the candidate image frame; and determine the candidate image frame corresponding to the highest second similarity as the image frame to be improved.

[0128] In a possible embodiment, the splitting module 501 is specifically used to determine the adjusted first resolution based on the first splitting ratio and the preset resolution; adjust the resolution of the image frame to be improved based on the first resolution; and split the image frame to be improved after the resolution is adjusted into multiple sub-images according to the first splitting ratio.

[0129] In one possible embodiment, the splitting module 501 is specifically used to determine the pixel ratio between the number of pixels in the width and the number of pixels in the height of the image frame to be improved; determine the deviation between the pixel ratio and each candidate splitting ratio respectively; and determine the candidate splitting ratio corresponding to the minimum deviation as the first splitting ratio.

[0130] In one possible embodiment, the splitting module 501 is specifically used to construct each candidate split ratio based on a preset minimum number of blocks and a preset maximum number of blocks, wherein the product of the two values ​​in any candidate split ratio is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks.

[0131] In a possible embodiment, the splitting module 501 is also used for the input layer to select a second splitting ratio from each pre-saved candidate splitting ratio, and determine the adjusted second resolution based on the second splitting ratio and the preset resolution; adjust the resolution of the image frame to be improved based on the second resolution; split the image frame to be improved after the resolution is adjusted into multiple sub-images according to the second splitting ratio; wherein the splitting traces of the image frame to be improved after the resolution is adjusted according to the second splitting ratio do not overlap with the splitting traces corresponding to the sub-images that have been obtained.

[0132] In a possible embodiment, the screening module 503 is specifically used for the feature processing layer to determine the weight corresponding to each sub-image based on the second feature matrix and position code corresponding to the received prompt text, the first feature matrix and position code corresponding to each image, the model dimension and the weight calculation formula; and select a preset number of key sub-images from each sub-image based on the weight corresponding to each sub-image.

[0133] Based on the above embodiments, the present application also provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, it includes: a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604;

[0134] The memory 603 stores a computer program. When the program is executed by the processor 601, the processor 601 performs the following steps:

[0135] The input layer selects at least one image frame from the received video as the image frame to be enhanced, splits the image frame to be enhanced into multiple sub-images; adjusts each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution, and then inputs them into the visual encoder to obtain a first feature matrix corresponding to each image;

[0136] The feature processing layer selects a preset number of key sub-images from the plurality of sub-images based on the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; inputs the first feature matrix corresponding to each key sub-image, the image frame to be enhanced, and each of the other image frames into the multilayer perceptron for feature mapping to obtain a third feature matrix;

[0137] The third feature matrix and the second feature matrix are input into the large language model to obtain a processing result corresponding to the video.

[0138] In a possible implementation, the input layer selects at least one image frame from the received video as the image frame to be enhanced, including:

[0139] The input layer selects a preset number of candidate image frames in the video; for each candidate image frame, determines a first similarity between the candidate image frame and other candidate image frames; based on each first similarity, determines a second similarity corresponding to the candidate image frame; and determines the candidate image frame corresponding to the highest second similarity as the image frame to be improved.

[0140] In a possible implementation, splitting the image frame to be enhanced into multiple sub-images includes:

[0141] Determining an adjusted first resolution according to the first split ratio and the preset resolution;

[0142] Adjusting the resolution of the image frame to be improved based on the first resolution;

[0143] The image frame to be enhanced after the resolution adjustment is split into a plurality of sub-images according to the first splitting ratio.

[0144] In a possible implementation, the process of determining the first split ratio includes:

[0145] Determining a pixel ratio between the number of pixels in width and the number of pixels in height of the image frame to be enhanced;

[0146] determining the deviation between the pixel ratio and each candidate split ratio respectively;

[0147] The candidate split ratio corresponding to the minimum deviation is determined as the first split ratio.

[0148] In a possible implementation, the process of determining each candidate split ratio includes:

[0149] Each candidate split ratio is constructed according to a preset minimum number of blocks and a preset maximum number of blocks, wherein the product of the two values ​​in any candidate split ratio is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks.

[0150] In a possible implementation, after splitting the image frame to be enhanced into multiple sub-images, and before adjusting each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and inputting the resolutions to the visual encoder, the method further includes:

[0151] The input layer selects a second split ratio from each pre-saved candidate split ratio, and determines an adjusted second resolution according to the second split ratio and a preset resolution;

[0152] Adjusting the resolution of the image frame to be improved based on the second resolution;

[0153] The image frame to be enhanced after the resolution adjustment is split into multiple sub-images according to the second splitting ratio; wherein the splitting traces of the image frame to be enhanced after the resolution adjustment is split according to the second splitting ratio do not overlap with the splitting traces corresponding to the sub-images that have been obtained.

[0154] In one possible implementation, the feature processing layer selects a preset number of key sub-images from the plurality of sub-images based on the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image, including:

[0155] The feature processing layer determines the weight corresponding to each sub-image based on the second feature matrix and position code corresponding to the received prompt text, the first feature matrix and position code corresponding to each image, the model dimension, and the weight calculation formula;

[0156] According to the weight corresponding to each sub-image, a preset number of key sub-images are selected from each sub-image.

[0157] The communication bus mentioned in the above-mentioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 602 is used for communication between the above-mentioned electronic device and other devices. The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0158] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processing processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0159] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program that can be executed by a processor, so that the processor implements the steps of the video processing method based on the multimodal large model as described above when executing the computer program.

[0160] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0164] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A video processing method based on a multimodal large model, characterized in that: The multimodal large model includes an input layer, a visual encoder, a feature processing layer, a multilayer perceptron and a large language model, and the method includes: The input layer selects at least one image frame in the received video as the image frame to be enhanced, and splits the image frame to be enhanced into multiple sub-images; each sub-image, the image frame to be enhanced, and other image frames in the video are adjusted to a preset resolution and then input into the visual encoder to obtain a first feature matrix corresponding to each image; The feature processing layer selects a preset number of key sub-images from the plurality of sub-images according to the second feature matrix corresponding to the received prompt text, the first feature matrix corresponding to each image, and the position code; inputs the first feature matrix corresponding to each of the key sub-images, the image frame to be enhanced, and each of the other image frames into the multi-layer perceptron for feature mapping to obtain a third feature matrix; The third feature matrix and the second feature matrix are input into the large language model to obtain a processing result corresponding to the video.

2. The method according to claim 1, characterized in that The input layer selects at least one image frame in the received video as the image frame to be enhanced, including: The input layer selects a preset number of candidate image frames in the video; for each candidate image frame, determines a first similarity between the candidate image frame and other candidate image frames; based on each first similarity, determines a second similarity corresponding to the candidate image frame; and determines the candidate image frame corresponding to the highest second similarity as the image frame to be improved.

3. The method according to claim 1, characterized in that The step of splitting the image frame to be enhanced into a plurality of sub-images comprises: Determining an adjusted first resolution according to the first split ratio and the preset resolution; Adjusting the resolution of the image frame to be improved based on the first resolution; The image frame to be enhanced after the resolution adjustment is split into a plurality of sub-images according to the first splitting ratio.

4. The method according to claim 3, characterized in that The process of determining the first split ratio includes: Determine a pixel ratio between the number of pixels in width and the number of pixels in height of the image frame to be enhanced; Determining the deviation between the pixel ratio and each candidate split ratio respectively; The candidate split ratio corresponding to the minimum deviation is determined as the first split ratio.

5. The method according to claim 4, characterized in that The process of determining each candidate split ratio includes: Each candidate split ratio is constructed according to a preset minimum number of blocks and a preset maximum number of blocks, wherein the product of two values ​​in any candidate split ratio is greater than the preset minimum number of blocks and less than or equal to the preset maximum number of blocks.

6. The method according to any one of claims 1 to 5, characterized in that: After splitting the image frame to be enhanced into a plurality of sub-images, and before adjusting each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and inputting them into the visual encoder, the method further includes: The input layer selects a second split ratio from each pre-saved candidate split ratio, and determines an adjusted second resolution according to the second split ratio and a preset resolution; Adjusting the resolution of the image frame to be improved based on the second resolution; The image frame to be enhanced after the resolution adjustment is split into multiple sub-images according to the second splitting ratio; wherein the splitting traces of the image frame to be enhanced after the resolution adjustment is split according to the second splitting ratio do not overlap with the splitting traces corresponding to the sub-images that have been obtained.

7. The method according to claim 1, characterized in that The feature processing layer selects a preset number of key sub-images from the multiple sub-images according to the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image, including: The feature processing layer determines the weight corresponding to each sub-image according to the second feature matrix and position code corresponding to the received prompt text, the first feature matrix and position code corresponding to each image, the model dimension and the weight calculation formula; According to the weight corresponding to each sub-image, a preset number of key sub-images are selected from each sub-image.

8. A video processing device based on a multimodal large model, characterized in that: The multimodal large model includes an input layer, a visual encoder, a feature processing layer, a multilayer perceptron and a large language model, and the device includes: A splitting module, used for the input layer to select at least one image frame in the received video as the image frame to be enhanced, and split the image frame to be enhanced into multiple sub-images; A feature extraction module, used for adjusting each sub-image, the image frame to be enhanced, and other image frames in the video to a preset resolution and inputting them into the visual encoder to obtain a first feature matrix corresponding to each image; A screening module, configured for the feature processing layer to select a preset number of key sub-images from the plurality of sub-images according to the second feature matrix corresponding to the received prompt text, and the first feature matrix and position code corresponding to each image; A feature mapping module, used for inputting the first feature matrix corresponding to each of the key sub-images, the image frame to be enhanced and each of the other image frames into the multi-layer perceptron for feature mapping to obtain a third feature matrix; An inference module is used to input the third feature matrix and the second feature matrix into the large language model to obtain a processing result corresponding to the video.

9. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the video processing method based on the multimodal large model as claimed in any one of claims 1 to 7 when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the video processing method based on a multimodal large model as described in any one of claims 1 to 7.

Images