Video compression method and device, equipment, storage medium and program product

Abstract

The invention relates to the technical field of video processing, and discloses a video compression method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-processed target video; determining a compression parameter corresponding to the target video based on the duration of the target video; and performing video compression processing on the target video according to the compression parameter to generate a compressed video corresponding to the target video. By implementing the technical scheme of the invention, adaptive compression for the target videos with different time lengths is realized, the video compression quality is improved, the detailed information of the video content is prevented from being lost, the understanding ability of the visual language model for the target video content with different lengths is improved, and the understanding accuracy of the video content is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video compression method, device, equipment, storage medium and program product. Background Art

[0002] Currently, many video language models are extensions of single-image visual language models (VLMs), which find it difficult to effectively handle the complexity of longer videos. In addition, the same compressor is usually used for long and short videos, keeping the compression ratio fixed, which leads to poor video compression quality, thus affecting the visual language model's ability to understand the content of long videos. Summary of the invention

[0003] In view of this, the present disclosure provides a video compression method, apparatus, device, storage medium and program product to solve the problem that poor video compression quality affects the understanding of video content.

[0004] In a first aspect, the present disclosure provides a video compression method, including: obtaining a target video to be processed; determining compression parameters corresponding to the target video based on the length of the target video; performing video compression processing on the target video according to the compression parameters to generate a compressed video corresponding to the target video.

[0005] In a second aspect, the present disclosure provides a video compression device, including: an acquisition module, used to acquire a target video to be processed; a parameter determination module, used to determine the compression parameters corresponding to the target video based on the length of the target video; and a compression module, used to perform video compression processing on the target video according to the compression parameters to generate a compressed video corresponding to the target video.

[0006] In a third aspect, the present disclosure provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the video compression method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0007] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the video compression method of the first aspect or any corresponding embodiment thereof.

[0008] In a fifth aspect, the present disclosure provides a computer program product, including computer instructions, which are used to enable a computer to execute the video compression method of the first aspect or any corresponding embodiment thereof.

[0009] The video compression method, device, equipment, storage medium and program product provided by the present disclosure determine the compression parameters matching the target video in combination with the duration of the target video, and then compress the target video according to the determined compression parameters to obtain the corresponding compressed video. Thus, different compression parameters are determined for target videos of different durations, and adaptive compression for target videos of different durations is achieved, the video compression quality is improved, and the loss of detailed information of the video content is avoided, which is conducive to improving the visual language model's ability to understand the content of target videos of different lengths, and ensures the accuracy of understanding the video content. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a flowchart of a video compression method according to an embodiment of the present disclosure;

[0012] Figure 2 is a flowchart of another video compression method according to an embodiment of the present disclosure;

[0013] Figure 3 is a schematic diagram of a visual language model and a compression method according to an embodiment of the present disclosure;

[0014] Figure 4 is a flowchart of another video compression method according to an embodiment of the present disclosure;

[0015] Figure 5 is a structural block diagram of a video compression device according to an embodiment of the present disclosure;

[0016] Figure 6 It is a schematic diagram of the hardware structure of the computer device of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0018] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0019] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0020] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0022] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0023] Currently, the same compressor is usually used for long videos and short videos to keep the compression ratio fixed. However, due to the complexity of longer videos, a fixed compression ratio is difficult to effectively process longer videos, affecting the compression quality of long videos.

[0024] In view of this, the technical solution disclosed in the present invention adapts to video compression of different lengths through dynamic compression to ensure the quality of video compression. At the same time, the pre-trained visual language model is deployed in the computer device, such as Figure 3 As shown, the visual language model includes a visual encoder and a language description model. Among them, the language description model is a model for describing the video content. The language description model can be trained based on a large language model architecture or based on a deep learning model architecture, which is not specifically limited here. For the specific training method of the visual language model, please refer to the following embodiment. Of course, connectors of different modes can also be provided in the visual language model to suit the understanding of content in different modes.

[0025] The visual language model can be used to perform image and video analysis on compressed videos to understand the text and images in the compressed videos and complete tasks such as image question answering and video question answering. Therefore, by processing the compressed videos through the visual language model, the detailed information of the video content is avoided from being lost, which is conducive to improving the visual language model's ability to understand the content of target videos of different lengths and ensuring the accuracy of understanding the video content.

[0026] According to an embodiment of the present disclosure, a video compression method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a video compression method is provided, which can be used in computer devices, such as computers, mobile phones, tablet computers, etc. Figure 1 is a flowchart of a video compression method according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0028] Step S101, obtaining a target video to be processed.

[0029] The target video is a video whose video content needs to be understood. In a specific embodiment, when understanding the video content of the target video, the user can input the target video through the interactive interface provided by the computer device, and accordingly, the computer device can obtain the target video input by the user.

[0030] In another specific implementation, the user can select a target video from a local video library, and the computer device can respond to the user's query instruction and extract the target video selected by the user from the local video library. Of course, the target video can also be collected online in real time. The method for obtaining the target video is not specifically limited here, and those skilled in the art can determine it according to actual needs.

[0031] Step S102: Determine compression parameters corresponding to the target video based on the duration of the target video.

[0032] The duration is the playback duration of the target video, such as 1s, 2s, 3s, etc., and target videos of different durations have different numbers of video frames. The compression parameter is a parameter required for feature compression of target videos of different durations.

[0033] Specifically, the target video is parsed according to the duration of the target video to determine the video format and video frames of the target video. The visual features of the target video are evaluated in combination with the video format of the target video and the images of each video frame. Then, the video features of the target video are analyzed using a dynamic compression algorithm to determine compression parameters that match the duration of the target video.

[0034] Step S103: perform video compression processing on the target video according to the compression parameters to generate a compressed video corresponding to the target video.

[0035] The video format, frame rate, resolution and other information of the target video during the compression process can be determined through the compression parameters, and the compression parameters are input into the visual language model. All video frames of the target video are compressed through the visual language model to compress all video frames of the target video together to obtain the corresponding compressed video, thereby including all the contents of the target video in the compressed video.

[0036] The video compression method provided in this embodiment determines the compression parameters that match the target video in combination with the duration of the target video, and then compresses the target video according to the determined compression parameters to obtain a corresponding compressed video. Thus, different compression parameters are determined for target videos of different durations, and adaptive compression for target videos of different durations is achieved, thereby improving the video compression quality, avoiding the loss of detailed information about the video content, and being conducive to improving the visual language model's ability to understand the content of target videos of different lengths, thereby ensuring the accuracy of understanding the video content.

[0037] In this embodiment, a video compression method is provided, which can be used in computer devices, such as computers, mobile phones, tablet computers, etc. Figure 2 is a flowchart of a video compression method according to an embodiment of the present disclosure. Figure 2 As shown, the process includes the following steps:

[0038] Step S201, obtaining a target video to be processed. Please refer to the description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0039] Step S202: Determine compression parameters corresponding to the target video based on the duration of the target video.

[0040] Specifically, the compression parameters include the amount of visual tokens, which is the process of converting video frames into a set of fixed-length digital representations. The visual tokens are used to convert complex visual information into a digital form that can be understood by computers, which is convenient for further text processing tasks to achieve video description, video summarization, video retrieval, visual question answering, etc. for target videos.

[0041] Accordingly, the above step S202 includes:

[0042] Step S2021: If the duration exceeds the preset duration, a key video frame of the target video is obtained.

[0043] The preset duration is the pre-set allowed video playback length, that is, for target video frames that do not exceed the preset duration, the target video can be kept uncompressed to obtain detailed information of the target video; for target video frames that exceed the preset duration, dynamic compression can be performed according to the duration of the target video to enhance the time details of the target video, thereby obtaining the video content of the target video.

[0044] The key video frame is a video frame that represents the video content, and the key video frame contains sufficient information in the target video. Specifically, the key video frame is each video frame of the target video, and can also be collected from multiple video frames of the target video in a uniform sampling manner. In order to ensure the integrity of the video content, it is preferred that the key video frame is all the video frames that constitute the target video.

[0045] The duration of the target video is compared with the preset duration to determine whether the duration of the target video exceeds the preset duration. If the duration of the target video exceeds the preset duration, it means that the target video is a long video. At this time, the target video needs to be compressed to improve the video processing efficiency. In order to ensure that the key content carried by the target video can be effectively compressed, the key video frames can be collected for the target video.

[0046] Step S2022, extracting visual features of key video frames, compressing the visual features based on a preset compression method, and obtaining a visual token quantity corresponding to the target video.

[0047] As described above, a visual encoder is provided in the visual language model, and the key video frames of the acquired target video are input into the visual encoder, and the visual features of the key video frames are extracted by the visual encoder. Specifically, given a target video V containing N frames = {X0,...,X N-1}, each key video frame X i are independently input into the visual encoder to extract the corresponding visual features F i That is, the output of the visual encoder can be expressed as F = {F0, ..., F N-1}, where F i ∈R (H×W)×C represents the visual features of the i-th key video frame.

[0048] The visual token quantity refers to the amount of visual information contained in a video frame. The preset compression method is a pre-set visual feature compression algorithm. Here, a dynamic visual feature compression algorithm is used to reduce the number of visual features. Specifically, the compressed visual features can be expressed as:

[0049] F^=Compressor(F,M), where F^={F^0,...,F^ M-1}, M is the number of visual tokens corresponding to the target video.

[0050] Step S2023, if the duration does not exceed the preset duration, obtain the visual token quantity corresponding to each video frame of the target video.

[0051] If the duration does not exceed the preset duration, it means that the target video is a short video. At this time, there is no need to compress the target video, and the video content of the target video can be directly understood. The frame rate, resolution, content complexity (such as details, motion, color changes, etc.) and visual coding information of the target video are analyzed to determine the visual information carried by each video frame in the target video, and the complexity and information density of the video frame are quantified based on the visual information to obtain the amount of visual tokens corresponding to the video frame. Specifically, the amount of visual tokens can be estimated by a visual language model, for example, by analyzing the pixel distribution, color histogram, texture features, etc. in the video frame.

[0052] Among them, the more visual information displayed per unit time, the larger the amount of visual tokens; the higher the resolution of the video, the more pixels contained in the video frame, the more visual information, and the higher the amount of visual tokens; the more complex the video content, the higher the amount of visual tokens; rapid scene changes or complex motion patterns in the target video will also increase the amount of visual tokens; different video encoding algorithms process visual information differently, which will also increase the amount of visual tokens in the video frame.

[0053] In some optional implementations, the preset compression method includes one of a dynamic pooling method, a dynamic token merging method, and a dynamic token reduction method.

[0054] The dynamic pooling method is to use the pooling operation as a downsampling method to sample the target video, such as Figure 3 As shown. The dynamic pooling method can be expressed as:

[0055] F^=AdaptiveAvgPool2d(F,[H,W]),

[0056] Among them, H = W ∈ [4, 28], H and W represent the height and width of the video frame respectively, then the number of visual tokens M = H × W.

[0057] Dynamic token merging is to merge the visual tokens of different video frames using cosine similarity after obtaining the visual features of each video frame, such as Figure 3 Specifically, the k-th smallest bipartite graph soft matching is applied here to obtain similar visual tokens:

[0058] F^=k-th-Bipartite-Soft-Matching(F,k)

[0059] Wherein, k=N / M∈[1,49], N represents the number of video frames of the target video.

[0060] The dynamic token reduction method allows the use of any number of tags to represent a video frame image, such as Figure 3 As shown. Dynamic token pruning is used to determine visual tokens. First, the video frame F is processed by a multi-layer perceptron (MLP) to obtain a set of logits. Then, these logits are processed by the Gumbel-Softmax function to obtain a probability distribution. The KeepTopK operation is used to retain the K elements with the highest probability from the probability distribution. Finally, the obtained K elements are determined as the final output F^, that is, the visual token corresponding to the target video. Specifically, its expression is:

[0061] F^=KeepTopK(GumbelSoftmax(MLP(F),M,F))

[0062] Among them, the number of visual tokens M belongs to [16,576], which represents the constraint or limitation of dynamic token pruning and is used to control the scale parameter of Gumbel-Softmax distribution. KeepTopK means retaining the K elements with the highest probability in the probability distribution after Gumbel-Softmax processing and discarding the other elements. MLP stands for Multilayer Perceptron. F represents the video frame vector or tensor input to MLP, and a set of logits is obtained after MLP processing.

[0063] GumbelSoftmax is an algorithm that combines Gumbel noise and Softmax function to generate samples with probability distribution. It alleviates the saturation problem of Softmax function by adding Gumbel noise to logits (i.e., the unactivated output of the last layer of the neural network), making the generated distribution smoother and more uniform. Gumbel distribution is a continuous probability distribution whose probability density function reaches its peak at x=0 and decreases rapidly as x increases or decreases. Softmax function is a function that converts any real number vector into a probability distribution. For a set of real numbers z, Softmax function ensures that the sum of all output values ​​is 1.

[0064] For dynamic pooling, dynamic token merging or dynamic token pruning, various compression methods can be tested here, and one is selected as the compression method used by the visual language model. Specifically, a simpler training process is set to evaluate the effectiveness of each compression method. The compressed videos obtained after compression according to each compression method are pre-trained, and then fine-tuned on a small part of the proposed dataset (about 11K mixed data). Video Multimodal Embedding (VideoMME) and Multimodal Video Description Question Answering (MSVD-QA) are used to evaluate the compression performance of each compression method. As shown in Table 1, the results show that the representation dynamic token pruning method is not effective when high-level understanding is required, while the simple adaptive dynamic pooling method performs slightly better than the dynamic token merging method. Therefore, the dynamic pooling method can be used as the default compression method for the visual language model.

[0065]

[0066]

[0067] In the above-mentioned implementation, visual features are compressed by a set dynamic pooling method, a dynamic token merging method or a dynamic token pruning method, thereby realizing dynamic compression of visual features to adapt to target videos of different lengths, thereby reducing the usage of visual tokens for long videos, so that the compressed video obtained after compression can accommodate more video frames, thereby improving the video compression quality.

[0068] Step S203, compress the target video according to the compression parameters to generate a compressed video corresponding to the target video. Please refer to the relevant description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0069] The video compression method provided in this embodiment, by comparing the duration of the target video with the preset duration, compresses the visual features of the target video when the duration of the target video exceeds the preset duration, so as to reduce the number of visual tokens used, so that more video frame content can be represented with less visual tokens used, thereby improving the compression quality of long videos. When the duration of the target video does not exceed the preset duration, the amount of visual tokens corresponding to each video frame in the target video is directly obtained, thereby keeping the visual tokens uncompressed to provide detailed video information, thereby ensuring the compression quality of short videos. Here, combined with the duration comparison relationship, the amount of visual tokens that match the target video is determined to ensure that the amount of visual tokens matches the target video, thereby achieving adaptive compression for target videos of different durations, avoiding the use of a fixed compression ratio that affects the compression quality of long videos, and further improving the video compression effect, which is conducive to the subsequent understanding of the video content and ensures the subsequent video content understanding ability.

[0070] In this embodiment, a video compression method is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 4 is a flowchart of a video compression method according to an embodiment of the present disclosure. Figure 4 As shown, the process includes the following steps:

[0071] Step S301, obtaining a target video to be processed. Please refer to the description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0072] Step S302: Determine compression parameters corresponding to the target video based on the duration of the target video. For details, please refer to the description of the corresponding steps in the above embodiment, which will not be repeated here.

[0073] Step S303: compress the target video according to the compression parameters to generate a compressed video corresponding to the target video. Please refer to the description of the corresponding steps in the above embodiment for details, which will not be repeated here.

[0074] Step S304: input the compressed video into a pre-trained visual language model, and generate video content represented by the compressed video through the visual language model.

[0075] The compressed video is input into the visual language model, and the video content in the compressed video is extracted and described through the language description model in the visual language model to obtain the video content represented by the compressed video.

[0076] Specifically, the training method of the above visual language model includes:

[0077] Step a1: collect a video question-answer pair dataset for video sample data, and use the video question-answer pair dataset to build a model training task, which is used to guide the output of the visual language model.

[0078] The video question-answer data pair dataset is the questions generated for the video samples and the answers corresponding to the questions. The video question-answer pairs can be generated using a prompt closed-source model. Specifically, the original videos in multiple datasets are used as video samples. The original videos include videos of different lengths. For multiple original videos, duplicate videos can be removed by original subtitles. At the same time, each subtitle is parsed into an original noun block, and the high-frequency blocks are downsampled to reduce the number of videos. When the prompt closed-source model is used to generate question-answer pairs, for the filtered multiple original videos, some are provided with original subtitles and videos, and some are provided with only videos. Since the original videos have sufficient diversity, there is no need to process the original videos here. In most cases, if the video length exceeds 128 seconds, up to 128 frames can be sampled for each video sample; otherwise, we sample at a rate of 1FPS. In addition, in order to enhance the model's ability to process longer video sequences, some videos are sampled up to 256 frames. As a result, the duration of the video frame is variable, making the scale of the video question-answer pair dataset larger.

[0079] After obtaining the video sample, the prompt description information is set to perform in-depth reasoning on the visual question and answer of the video sample. Visual question and answer are closely related to the visual content of the video, so the model training task is constructed in combination with the video question and answer dataset. Specifically, it includes perception tasks (recognizing object entities and their attributes, positions and movements in the video), general tasks (re-captioning tasks, detailed descriptions, sentiment analysis and story writing), time tasks (intensive video captioning, question and answer with timestamps and general time question and answer), reasoning tasks (complex details in visual modalities) and formatting tasks (guiding the model to output answers according to the requirements of multiple-choice questions and answers, and providing concise answers for open-ended questions and answers).

[0080] Step a2: Use the video question-answering dataset and model training task to pre-train the model and generate an initial visual language model.

[0081] Filter out the wrong responses and answers that do not conform to the specified format or are rejected by the closed-source model for various reasons from the video question-answer pair dataset, and convert the timestamps in the video question-answer pairs into different formats, such as <frame3>to "frameof 3s". Then, save the sampled video frames directly instead of the original video to ensure that the visual language model can see the exact same video frames as the closed-source model.

[0082] All parameters of the visual language model are unfrozen, and the model is trained in combination with the video question-answering dataset and model training tasks to enhance its overall visual perception ability. In the pre-training stage, the number of labels for each video frame can be randomly sampled in the interval [16,576]. A simple system prompt (such as "You are a helpful visual assistant") can be used. For the language description model, 2×10 -5 The learning rate is 1, and the batch size is 128. The extraction of visual features can use half of the learning rate. The initial model of visual language with basic visual perception ability is generated through pre-training.

[0083] Step a3, performing visual optimization and video optimization on the initial visual language model to obtain a visual language model.

[0084] Fine-tune the visual parameters and video parameters in the visual language initial model while keeping other parameters in a frozen state, thereby achieving visual optimization and video optimization of the visual language initial model and obtaining a visual language model that meets the needs of video understanding.

[0085] In the above implementation, a large-scale question-answer pair dataset is used to construct a model training task, so that the scope of the model training task is more extensive. The model pre-training is performed in combination with the question-answer pair dataset and the model training task, and then the visual parameters and video parameters of the model are fine-tuned to obtain the final visual language model, thereby ensuring that the visual language model can have a good video content understanding ability, so that the visual language model can accurately identify the video content of the target video, so as to effectively process the video data according to the user's question.

[0086] In some optional implementations, the above step a3 may include:

[0087] Step a31, collecting the visual data set corresponding to the video sample, and using the visual data set to perform visual optimization on the initial model of the visual language.

[0088] The visual dataset is a dataset containing visual question-answer pairs. Specifically, the visual dataset is collected from a large amount of publicly accessible data, for example, data is collected from two publicly accessible categories, general visual question answering (VQA) and OCR. The visual dataset is used to fine-tune the visual parameters of the visual language initial model, and the visual dataset is used to enhance the model's visual instruction ability.

[0089] Step a32, obtaining structured video data and visual compression features of the video sample, where the video data includes time information and video image information.

[0090] The time information is the timestamp information of the video frame, and the video image information is the frame image information corresponding to each video frame. The structured video data is the data obtained by structuring the video frame. Specifically, the structured video data can be expressed as: 1s: ; 2s: ;.......

[0091] Visual compression features are features obtained by extracting visual features from video frames in a video sample by a visual encoder and compressing them using a preset compression method. Specifically, the video sample is input into the visual encoder of the visual language model for visual encoding, the visual features of the video frame are extracted, and the visual features are compressed using a visual feature compression algorithm to obtain the corresponding visual compression features.

[0092] Step a33, replacing the video image information with visual compression features to obtain optimized structured video data, and using the optimized structured video data to perform video optimization on the initial model of the visual language.

[0093] Step a34, generating a visual language model based on the optimization results of visual optimization and video optimization.

[0094] The model input uses structured video data as input, in which the timestamps for each video frame are included, and the compressed visual features replace the video image information. , and obtain the optimized structured video data. The optimized structured video data is input into the language description model of the visual language model to fine-tune the video parameters to enhance the model's video instruction capability. During the training process, the maximum context window can be set to 16K, and the number of features for each video frame is from the interval [16, min (N max / T,576)] randomly selected, where N max Represents the maximum number of visual features, T represents the number of video frames, and in the above-collected dataset, the maximum number of frames can reach 256. Then, the visual optimization results of the model and the video optimization results are fused to obtain a visual language model with fine-tuned parameters.

[0095] In the above implementation, after pre-training is completed, the visual language initial model is visually optimized and video optimized to achieve visual instruction tuning and video instruction tuning of the model, thereby improving the visual language model's processing capabilities for videos and video content.

[0096] In some optional implementations, the above method may further include: testing the visual language model according to at least one preset question-and-answer method, and generating a test result for the visual language model.

[0097] The preset question-and-answer modes are pre-set test modes, including open-ended video question-and-answer, multiple-choice video question-and-answer, and multiple-choice multi-image question-and-answer. Among them, open-ended video question-and-answer aims to prompt free-form answers to questions related to the video content; multiple-choice video question-and-answer aims to instruct to select the correct option from multiple candidate options in the context of the question and answer; and multiple-choice multi-image question-and-answer is similar to multiple-choice video question-and-answer, involving presenting a question with multiple candidate answers and several images.

[0098] The visual language model is tested using one or more preset question-and-answer methods to generate corresponding model test results. The visual language model's understanding performance of video content is evaluated through the model test results, so that the visual language model can be trained more robustly in combination with the test results to create a general model suitable for various tasks.

[0099] The video compression method provided in this embodiment, by inputting the compressed video into a pre-trained visual language model, generates video content represented by the compressed video by the visual language model, which helps to more effectively process video data and achieve accurate recognition of video content.

[0100] In this embodiment, a video compression device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0101] This embodiment provides a video compression device, such as Figure 5 As shown, including:

[0102] The acquisition module 401 is used to acquire the target video to be processed.

[0103] The parameter determination module 402 is used to determine the compression parameters corresponding to the target video based on the duration of the target video.

[0104] The compression module 403 is used to perform video compression processing on the target video according to the compression parameters to generate a compressed video corresponding to the target video.

[0105] In some optional implementations, the parameter determination module 402 includes:

[0106] The key video frame acquisition unit is used to acquire the key video frame of the target video if the duration exceeds a preset duration.

[0107] The first token quantity acquisition unit is used to extract visual features of key video frames, compress the visual features based on a preset compression method, and obtain the visual token quantity corresponding to the target video.

[0108] The second token quantity acquisition unit is used to acquire the visual token quantity corresponding to each video frame of the target video if the duration does not exceed the preset duration.

[0109] In some optional implementations, the preset compression method includes one of a dynamic pooling method, a dynamic token merging method, and a dynamic token reduction method.

[0110] In some optional embodiments, the above device further includes:

[0111] The video content generation module is used to input the compressed video into a pre-trained visual language model and generate video content represented by the compressed video through the visual language model.

[0112] In some optional embodiments, the above device further includes:

[0113] Model training module, used to train visual language models.

[0114] Specifically, the model training module includes:

[0115] The question-answer pair collection unit is used to collect a video question-answer pair dataset for video sample data, and use the video question-answer pair dataset to build a model training task, which is used to guide the output of the visual language model.

[0116] The pre-training unit is used to pre-train the model using the video question-answering dataset and model training tasks to generate an initial visual language model.

[0117] The optimization unit is used to perform visual optimization and video optimization on the initial visual language model to obtain a visual language model.

[0118] In some optional embodiments, the optimization unit includes:

[0119] The visual optimization subunit is used to collect the visual data set corresponding to the video sample, and use the visual data set to perform visual optimization on the initial model of the visual language.

[0120] The data acquisition subunit is used to acquire structured video data and visual compression features of video samples, where the video data includes time information and video image information.

[0121] The video optimization subunit is used to replace video image information with visual compression features to obtain optimized structured video data, and use the optimized structured video data to perform video optimization on the visual language initial model.

[0122] The model fusion subunit is used to generate a visual language model based on the optimization results of visual optimization and video optimization.

[0123] In some optional embodiments, the above device may further include:

[0124] The testing module is used to test the visual language model according to at least one preset question-and-answer method and generate a test result for the visual language model.

[0125] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0126] The video compression device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0127] The video compression device provided in this embodiment determines the compression parameters that match the target video in combination with the duration of the target video, and then compresses the target video according to the determined compression parameters to obtain the corresponding compressed video. Thus, different compression parameters are determined for target videos of different durations, and adaptive compression for target videos of different durations is achieved, thereby improving the video compression quality, avoiding the loss of detailed information of the video content, and being conducive to improving the ability of the visual language model to understand the content of target videos of different lengths, thereby ensuring the accuracy of understanding the video content.

[0128] The present disclosure also provides a computer device having the above Figure 5 The video compression device shown.

[0129] See also Figure 6 , Figure 6 is a schematic diagram of a computer device provided by an optional embodiment of the present disclosure, such as Figure 6 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0130] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0131] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0132] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0134] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0135] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0136] The computer device also includes a communication interface, which is used for the computer device to communicate with other devices or a communication network.

[0137] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0138] A part of the present disclosure may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present disclosure through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0139] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A video compression method, characterized in that: The method comprises: Get the target video to be processed; Determining compression parameters corresponding to the target video based on the duration of the target video; The target video is compressed according to the compression parameters to generate a compressed video corresponding to the target video.

2. The method according to claim 1, characterized in that The step of determining a compression parameter corresponding to the target video based on the duration of the target video includes: If the duration exceeds a preset duration, obtaining a key video frame of the target video; Extracting visual features of the key video frame, compressing the visual features based on a preset compression method, and obtaining a visual token amount corresponding to the target video; Wherein, the compression parameter includes the visual token amount.

3. The method according to claim 2, characterized in that Also includes: If the duration does not exceed the preset duration, the amount of visual tokens corresponding to each video frame of the target video is obtained.

4. The method according to claim 2, characterized in that: The preset compression method includes one of a dynamic pooling method, a dynamic token merging method, and a dynamic token reduction method.

5. The method according to claim 1, characterized in that Also includes: Inputting the compressed video into a pre-trained visual language model, and generating video content represented by the compressed video through the visual language model; The training method of the visual language model includes: Collecting a video question-answer pair dataset for the video sample data, and using the video question-answer pair dataset to construct a model training task, wherein the model training task is used to guide the output of the visual language model; Using the video question-answering dataset and the model training task to perform model pre-training to generate an initial visual language model; Visual optimization and video optimization are performed on the initial visual language model to obtain the visual language model.

6. The method according to claim 5, characterized in that The performing visual optimization and video optimization on the initial visual language model to obtain the visual language model includes: Collecting a visual data set corresponding to the video sample, and using the visual data set to perform visual optimization on the initial model of the visual language; Acquire structured video data and visual compression features of a video sample, wherein the video data includes time information and video image information; Replacing the video image information with the visual compression feature to obtain optimized structured video data, and performing video optimization on the visual language initial model using the optimized structured video data; The visual language model is generated based on the optimization results of the visual optimization and the video optimization.

7. The method according to claim 5 or 6, characterized in that: Also includes: The visual language model is tested according to at least one preset question-and-answer method to generate a test result for the visual language model.

8. A video compression device, characterized in that: The device comprises: An acquisition module, used to acquire a target video to be processed; A parameter determination module, used to determine the compression parameters corresponding to the target video based on the duration of the target video; The compression module is used to perform video compression processing on the target video according to the compression parameters to generate a compressed video corresponding to the target video.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the video compression method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the video compression method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the video compression method according to any one of claims 1 to 7.

Images