Video processing method and device, intelligent agent, electronic equipment, storage medium and program product

Abstract

The invention provides a video processing method and device, an intelligent agent, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models, video processing and the like. The video processing method comprises the following steps: for at least two first video frames in a video sequence to be processed, determining at least two second video frames in the first video frames according to time sequence characteristics between each first video frame and other video frames except the first video frames in the video sequence to be processed, the time sequence features represent the similarity between adjacent video frames; and according to a cross-modal feature between the task description and the second video frame, compressing other target frames except the first target frame in the second video frame to obtain a processed video sequence, the first target frame being determined according to the cross-modal feature, and the second target frame being determined according to the cross-modal feature. The cross-modal feature characterizes a similarity between the task description and the second video frame.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the technical fields of computer vision, deep learning, large models and video processing, and more specifically to a video processing method and device, an intelligent body, an electronic device, a storage medium and a program product. Background Art

[0002] With the development of computer technology, artificial intelligence technology has also been developed. For example, Multimodal Large Language Models (MLLMs) came into being.

[0003] Multimodal large language models can integrate and understand multiple types of input data, such as text, images, audio, and video. In one example, multimodal large language models have shown strong potential in tasks such as video understanding and video question answering. Summary of the invention

[0004] The present disclosure provides a video processing method and device, an intelligent agent, an electronic device, a storage medium and a program product.

[0005] According to one aspect of the present disclosure, a video processing method is provided, including: for at least two first video frames in a video sequence to be processed, determining at least two second video frames in the first video frames according to timing features between each of the first video frames and other video frames in the video sequence to be processed except the first video frame, wherein the timing features characterize the similarity between adjacent video frames; and, according to cross-modal features between a task description and the second video frames, compressing other target frames in the second video frames except the first target frame to obtain a processed video sequence, wherein the first target frame is determined according to the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frames.

[0006] According to another aspect of the present disclosure, a video processing device is provided, including: a determination module, for determining at least two second video frames in the at least two first video frames in a video sequence to be processed, according to timing features between each of the first video frames and other video frames in the video sequence to be processed except the first video frame, wherein the timing features characterize the similarity between adjacent video frames; and a first compression processing module, for compressing other target frames in the second video frames except the first target frame according to cross-modal features between a task description and the second video frames, to obtain a processed video sequence, wherein the first target frame is determined according to the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frames.

[0007] According to another aspect of the present disclosure, an intelligent agent is provided, including: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large model based on the target task, and obtaining output information by calling the large model to execute a video processing method; and an output module for outputting the output information obtained by the processing module.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0009] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program or instructions, wherein the computer program or instructions implement the steps of the above method when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0013] Figure 1 The system architecture to which the video processing method according to the embodiment of the present disclosure can be applied is schematically shown;

[0014] Figure 2 The flowchart of the video processing method according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 3 An example schematic diagram of a process of determining at least two second video frames in a first video frame according to a timing feature between each first video frame and other video frames in a video sequence to be processed except the first video frame is schematically shown according to an embodiment of the present disclosure;

[0016] Figure 4 Schematically shows an example schematic diagram of a process of compressing target frames other than the first target frame in the second video frame according to the cross-modal features between the task description and the second video frame to obtain a processed video sequence according to an embodiment of the present disclosure;

[0017] Figure 5 An example schematic diagram of a video processing process according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 6 An example schematic diagram schematically shows a process of compressing second target frames in second video frames according to spatial features between each second video frame and other second video frames to obtain a target video sequence according to an embodiment of the present disclosure;

[0019] Figure 7 An example schematic diagram of a video processing process according to another embodiment of the present disclosure is schematically shown;

[0020] Figure 8 A block diagram schematically shows a video processing device according to an embodiment of the present disclosure;

[0021] Fig. 9 A block diagram schematically shows a structure of an intelligent agent of a large model according to an embodiment of the present disclosure; and

[0022] Fig.10 A block diagram of an electronic device suitable for implementing a video processing method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0024] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0025] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0026] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0027] Multimodal large language models have shown great potential in tasks such as video understanding and video question answering. However, the context length of current language models is usually limited to about 8k tokens, and long videos (e.g., hour-long videos) require more than 200,000 tokens to be fully represented. In addition, due to the significant temporal and spatial redundant information in long videos, only a small number of frames contain valuable visual content.

[0028] In one example, the number of frames can be reduced by uniform sampling or the number of tokens can be reduced by fixed dimensionality reduction. However, these methods either lose key information or require high computational costs, making it difficult to efficiently adapt to long video scenarios.

[0029] To this end, the embodiment of the present disclosure proposes a video processing scheme. For example, for at least two first video frames in a video sequence to be processed, at least two second video frames are determined in the first video frame according to the temporal features between each first video frame and other video frames in the video sequence to be processed except the first video frame, wherein the temporal features characterize the similarity between adjacent video frames; and, according to the cross-modal features between the task description and the second video frame, other target frames in the second video frame except the first target frame are compressed to obtain a processed video sequence, wherein the first target frame is determined according to the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frame.

[0030] According to the embodiments of the present disclosure, by analyzing the temporal features between the frames in the video sequence to be processed, the similarities between the first video frames are utilized to accurately determine at least two second video frames from at least two first video frames, thereby ensuring the representativeness of the selected frames and the integrity of the information. On this basis, by analyzing the cross-modal features between the task description and at least two second video frames, other target frames other than the selected first target frame can be compressed to reduce the amount of data without losing key information, thereby improving the efficiency and accuracy of video processing.

[0031] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0032] In the technical solution of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0033] Figure 1 The system architecture to which the video processing method according to the embodiment of the present disclosure can be applied is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0034] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0035] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0036] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0037] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0038] It should be noted that the video processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the video processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The video processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the video processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] Alternatively, the video processing method provided in the embodiment of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the video processing apparatus provided in the embodiment of the present disclosure may also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0042] The above describes the system architecture to which the video processing method provided by the present disclosure can be applied. Figure 2 For example, the video processing process of the present disclosure is further described.

[0043] Figure 2 The flowchart of the video processing method according to the embodiment of the present disclosure is schematically shown.

[0044] like Figure 2 As shown, the video processing method 200 includes operations S210-S220.

[0045] In operation S210, for at least two first video frames in a video sequence to be processed, at least two second video frames are determined in the first video frames based on timing features between each first video frame and other video frames in the video sequence to be processed except the first video frame, wherein the timing features characterize the similarity between adjacent video frames.

[0046] In operation S220, based on the cross-modal features between the task description and the second video frame, other target frames in the second video frame except the first target frame are compressed to obtain a processed video sequence, wherein the first target frame is determined based on the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frame.

[0047] The video processing method 200 can be applied to the fields of video question-answering, video summarization, long video surveillance analysis, etc. The input data of the video processing method 200 may include a video sequence to be processed of the video type. The video sequence to be processed refers to a set of video frames selected from a video stream that meet the requirements or specific conditions in the field of computer vision and video processing. At least two first video frames in the video frame set have a temporal relationship, that is, at least two first video frames in the video sequence to be processed are continuous in time. The method for intercepting the video sequence to be processed can be configured according to actual business needs and is not limited here. For example, the method for intercepting the video sequence to be processed may include at least one of the following: intercepting the video sequence to be processed using video editing software, intercepting the video sequence to be processed using a command line tool, and intercepting the video sequence to be processed using a programming language combined with a corresponding library.

[0048] The first video frame refers to a single image in the video sequence to be processed. For each first video frame in the video sequence to be processed, the temporal features between the first video frame and other video frames in the video sequence to be processed can be determined respectively. The temporal features refer to the features that characterize the similarity between adjacent video frames according to the temporal sequence of the first video frames. For example, the temporal features may include content changes, motion vectors, and optical flows between frames.

[0049] The method for determining the timing characteristics can be configured according to actual business needs and is not limited here. For example, the method for determining the timing characteristics may include at least one of the following: determining the timing characteristics based on histogram comparison, determining the timing characteristics based on the first preset model, and determining the timing characteristics based on optical flow. The first preset model may include, for example, a convolutional neural network (CNN) and a DINOv2 network.

[0050] After obtaining the timing features between each first video frame and other video frames in the video sequence to be processed, at least two second video frames can be determined in the first video frame according to the respective timing features of each first video frame. The second video frame may refer to a representative or critical video frame. For example, redundant frames in the time dimension may be removed according to the respective timing features of each first video frame to reduce the number of input video frames, obtain at least two second video frames, and reduce the computational burden for subsequent steps.

[0051] The input data of the video processing method 200 may also include a text-type task description. The task description refers to a text description used to guide the process of video processing. In one example, the task description may be a real-time query input by a user, for example, the task description may include questions or keywords. In another example, the task description may be a preset task goal, for example, the task description may include a keyword in a video summary.

[0052] After obtaining at least two second video frames, for each second video frame, the cross-modal features between the second video frame and the task description can be determined respectively. Cross-modal features refer to the features of the similarity between the task description and the second video frame, which can express the intrinsic connection and semantic information of different modal data. For example, the task description can be processed by text feature extraction to obtain text features. The second video frame is processed by image feature extraction to obtain image features. On this basis, cross-modal features can be determined based on text features and image features. The specific text feature extraction and processing method, image feature extraction and processing method, and feature fusion method can be configured according to actual business needs and are not limited here.

[0053] For example, the text feature extraction processing method may include at least one of the following: a text feature extraction method based on a bag-of-words (BoW) model, a text feature extraction method based on a TF-IDF model, a text feature extraction method based on an n-gram model, a text feature extraction method based on word embedding (Word Embedding), and a text feature extraction method based on a deep learning model, etc.

[0054] For another example, the image feature extraction processing method may include at least one of the following: a traditional feature extraction method and an image feature extraction method based on a deep learning model. The traditional feature extraction method may include at least one of the following: SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF), etc. The image feature extraction method based on a deep learning model may include at least one of the following: a variational autoencoder (VAE) model, a convolutional variational autoencoder (CVAE) model, and a bidirectional transformer (i.e., Bidirectional Transformer) model, etc.

[0055] For another example, the feature fusion method may include at least one of the following: a cross-modal feature determination method based on vector concatenation, a cross-modal feature determination method based on a Transformer model, a cross-modal feature determination method based on an attention mechanism, a cross-modal feature determination method based on a bidirectional transformer model, and a cross-modal feature determination method based on a multi-layer perceptron (MLP).

[0056] After obtaining the cross-modal features of each second video frame and the task description, the first target frame can be determined in the second video frame based on these cross-modal features. The first target frame refers to the key frame selected according to the task description, and these key frames need to be retained without compression due to their content, importance or other features. On this basis, the other target frames except the first target frame in the at least two second video frames can be compressed. Compression processing refers to the process of encoding video frames or reducing the amount of data, which can reduce the demand for storage space or transmission bandwidth. Compression processing may include at least one of the following: removing redundant information, reducing resolution, using a specific encoding algorithm, etc.

[0057] After the other target frames except the first target frame in the at least two second video frames are compressed, the first target frames and the other video frames after the compression processing can be used as a processed video sequence. The processed video sequence contains the key information in the original video sequence to be processed, but exists in a smaller amount of data, so as to facilitate storage or transmission.

[0058] According to the embodiments of the present disclosure, by analyzing the temporal features between the frames in the video sequence to be processed, the similarities between the first video frames are utilized to accurately determine at least two second video frames from at least two first video frames, thereby ensuring the representativeness of the selected frames and the integrity of the information. On this basis, by analyzing the cross-modal features between the task description and at least two second video frames, other target frames other than the selected first target frame can be compressed to reduce the amount of data without losing key information, thereby improving the efficiency and accuracy of video processing.

[0059] In the embodiment of the present disclosure, after obtaining the video sequence to be processed, at least two second video frames can be determined in the first video frame according to the timing characteristics between each first video frame and other video frames in the video sequence to be processed except the first video frame. Figure 3 Take as an example to illustrate the compression process based on time series features.

[0060] Figure 3 An example schematic diagram of a process of determining at least two second video frames in a first video frame according to a timing feature between each first video frame and other video frames in a video sequence to be processed except the first video frame according to an embodiment of the present disclosure is schematically shown.

[0061] like Figure 3 As shown, in 300, taking the example of a video sequence to be processed 310 including 9 first video frames shot at intervals of 1 millisecond and a second width of the second window of 3 milliseconds, the process of determining at least two second video frames in the first video frame according to the timing characteristics between each first video frame and other video frames in the video sequence to be processed except the first video frame is explained.

[0062] In one example, the video sequence to be processed 310 can be subjected to window processing based on the second window to obtain a second number of second frame sets. The second window can be a non-overlapping window, which helps to reduce the global computational complexity. For example, the second frame set 301 can include the first video frame from the 0th millisecond to the 2nd millisecond, that is, the first video frame 1, the first video frame 2, and the first video frame 3; the second frame set 302 can include the first video frame from the 3rd millisecond to the 5th millisecond, that is, the first video frame 4, the first video frame 5, and the first video frame 6; the third frame set 303 can include the first video frame from the 6th millisecond to the 8th millisecond, that is, the first video frame 7, the first video frame 9, and the first video frame 9.

[0063] After obtaining the second number of second frame sets, for each second frame set, feature extraction can be performed on at least two first video frames in the second frame set to obtain frame features of at least two first video frames, and similarity analysis can be performed based on the frame features to obtain similarity information, which can include the average similarity between each first video frame and other video frames in the second frame set. On this basis, at least two second video frames can be determined from the first video frames included in the second frame set according to the similarity information.

[0064] In an example, the frame features of each first video frame may be extracted in a manner as shown in the following formula (1), and the average similarity of each first video frame may be determined in a manner as shown in the following formula (2).

[0065]

[0066] in, represents the number of the first video frame in the video sequence to be processed, represents the i-th first video frame, Characterize the frame features of the i-th first video frame, J is the width of the second window, Characterize the frame features of the j-th first video frame, Characterize the cosine similarity between the frame features of the i-th first video frame and the frame features of the j-th first video frame, Characterizes the average similarity of the i-th first video frame.

[0067] Taking the second frame set 302 as an example, similarity analysis can be performed on at least two first video frames in the second frame set 302 to obtain similarity information. For example, the first similarity between the first video frame 4 and the first video frame 5 can be determined, the second similarity between the first video frame 4 and the first video frame 6 can be determined, and the average similarity 304_1 of the first video frame 4 can be determined based on the first similarity and the second similarity. Alternatively, the third similarity between the first video frame 5 and the first video frame 6 can be determined, and the average similarity 304_2 of the first video frame 5 can be determined based on the first similarity and the third similarity. Alternatively, the average similarity 304_3 of the first video frame 6 can be determined based on the second similarity and the third similarity.

[0068] On this basis, at least two second video frames can be determined from the first video frame 4, the first video frame 5, and the first video frame 6 according to the average similarity 304_1, the average similarity 304_2, and the average similarity 304_3. In one example, the first video frame with a higher average similarity can be deleted, while the first video frame with a lower average similarity is retained. For example, the average similarity 304_2 of the first video frame 5 is lower, so the first video frame 5 can be determined as the second video frame 305.

[0069] According to an embodiment of the present disclosure, by performing window processing on the video sequence to be processed based on the second window, the video sequence is divided into a second number of second frame sets according to the set width of the second window, so that each second frame set contains a fixed number of first video frames, which provides a basis for subsequent similarity analysis and second video frame selection. By performing a comprehensive similarity analysis on at least two first video frames in each second frame set, and based on the average similarity between each first video frame and other video frames in the second frame set except the first video frame, at least two second video frames are determined from at least two first video frames, ensuring that the selected second video frames are representative in terms of information volume and visual changes, thereby ensuring that key frames can be screened out from the video sequence, while ignoring those redundant frames that are visually highly similar to the key frames, thereby improving the efficiency of video processing.

[0070] In another example, a semantic model may be used to perform semantic analysis on the visual features of each first video frame to obtain a key point extraction result, and at least two second video frames may be determined in the first video frame based on the key point extraction result. For example, the semantic model may include an object detection model or a scene recognition model. Alternatively, pixel changes between frames may be detected to select frames with significant changes. For example, the detection method of pixel changes between frames may include edge detection or motion analysis.

[0071] According to an embodiment of the present disclosure, by using a semantic model to perform in-depth semantic analysis on the visual features of each of at least two first video frames, key point information of the first video frame can be extracted. On this basis, by determining at least two second video frames in the first video frame according to the key point extraction result, key frames representing the entire video sequence can be screened, which not only improves the accuracy of video processing, but also can store more video sequences in a limited storage space by screening key frames, and can also speed up the upload and download speed of video sequences due to the smaller file size, thereby improving transmission efficiency.

[0072] In the embodiment of the present disclosure, after obtaining at least two second video frames, other target frames in the second video frames except the first target frame may be compressed according to the cross-modal features between the task description and the second video frames to obtain a processed video sequence. Figure 4 Take as an example to illustrate the compression process based on cross-modal features.

[0073] Figure 4 The diagram schematically shows an example of a process of compressing target frames other than the first target frame in the second video frame according to the cross-modal features between the task description and the second video frame to obtain a processed video sequence according to an embodiment of the present disclosure.

[0074] like Figure 4 As shown, in 400, taking the second video frame including N as an example, the process of compressing the target frames other than the first target frame in the second video frame according to the cross-modal features between the task description and the second video frame to obtain the processed video sequence 405 is described. N is a positive integer.

[0075] In one example, the attention evaluation value between each second video frame and the task description can be determined respectively. The attention evaluation value can be used to characterize the correlation between the task description and the second video frame. For example, the attention evaluation value 402_1 between the task description 401 and the second video frame 1, the attention evaluation value 402_2 between the task description 401 and the second video frame 2, and so on, the attention evaluation value 402_N between the task description 401 and the second video frame N can be determined.

[0076] In one example, each second video frame may include at least two pieces of position information. For each second video frame, the visual features of the second video frame and the text features of the task description are determined so as to project the visual features into a space aligned with the text features. On this basis, the degree of match between each second video frame and the task description can be calculated based on the visual features of the second video frame and the text features of the task description to obtain an attention weight, which characterizes the global correlation between each piece of position information and the task description. After obtaining the attention weight, the attention weight can be normalized according to the initial resolution of the video sequence to be processed and the number of tokens of the task description after word segmentation to adapt to scenes of different frame sizes or query lengths to obtain an attention evaluation value. The attention evaluation value can characterize the overall match between the second video frame and the task description.

[0077] In a specific example, the attention evaluation value may be determined as shown in the following formula (3).

[0078]

[0079] in, Characterize the visual features of the i-th second video frame, represents the spatial resolution of each second video frame, Q represents the text features of the task description, The number of tokens representing the task description after word segmentation, Characterize a multi-layer perceptron for converting visual features into representations that are aligned with textual features.

[0080] According to an embodiment of the present disclosure, for each second video frame, an attention weight is obtained by analyzing its visual features and the text features of the task description, and the attention weight reflects the global correlation between each position in the second video frame and the task description, so that the in-frame area most relevant to the task description can be identified. On this basis, the attention weight is normalized according to the initial resolution and the number of tokens after the task description is processed by word segmentation to obtain the attention evaluation value of each second video frame, ensuring that the attention evaluation values ​​between different second video frames and task descriptions are comparable, so that the importance of each second video frame can be effectively evaluated and compared. In this way, key frames can be accurately identified and non-key frames can be compressed, so that more video sequences can be stored in a limited storage space, and while reducing the volume of the video sequence, the visual quality of the video sequence can be maintained as much as possible through key frames, thereby improving the efficiency and accuracy of video content analysis.

[0081] After obtaining the respective attention evaluation values ​​of each second video frame, the first target frame can be determined in the second video frame according to the attention evaluation values ​​of each second video frame. The first target frame can refer to a key frame with high-resolution features related to the task description. For example, it can be determined that the first target frame 404 includes the second video frame 3, the second video frame 5, ..., the second video frame N, then the other video frames 403 except these first target frames 404 include the second video frame 1, the second video frame 2 and the second video frame 3.

[0082] In one example, the second video frames can be sorted in descending order according to the attention evaluation value of each second video frame to obtain a video frame identification sequence. On this basis, the second video frames with the first first number of identifications in the video frame identification sequence can be determined as first target frames. The first target frame represents a frame that does not need to be compressed, and the other video frames represent frames that need to be compressed.

[0083] For example, when the first number is 0, it indicates that the context length limit is insufficient to retain the high-resolution features of any frame, that is, all second video frames need to be reduced in dimension. Alternatively, when the first number is greater than 0, the second video frames of the first number of identifiers in the video frame identifier sequence can be determined as the high-resolution first target frames that need to be retained.

[0084] In one example, the first number can be determined based on a target constraint. The target constraint can be used to dynamically adjust the number of high-resolution first target frames and the number of other low-resolution target frames under the context length limit to maximize the token utilization of the large model used to process the video sequence to be processed. The context length limit can be determined based on the maximum number of tokens of the large model, which affects the number of video frames that can be processed and the number of tokens per frame, thereby affecting the performance and efficiency of the model. In one example, the context length limit can be set comprehensively based on model computing resources and task requirements. For example, the context length limit can be set to 8k.

[0085] In a specific example, the first number may be determined as shown in the following formula (4).

[0086]

[0087] in, Characterizes the first number, Characterizes the context length limit, represents the number of tokens after word segmentation of the task description, T represents the number of the second video frames, Characterizes the spatial resolution of each second video frame, Characterizes the compression resolution, which can be dynamically adjusted based on context constraints.

[0088] According to an embodiment of the present disclosure, by intelligently determining the number of first target frames and the number of other video frames based on target constraints, the processing efficiency of video sequences is improved while ensuring that the token utilization of large models is maximized.

[0089] According to an embodiment of the present disclosure, by sorting the second video frames from large to small according to the attention evaluation value between each second video frame and the task description, a video frame identification sequence is obtained, which ensures that the order of the second video frames corresponds to their importance, that is, the second video frames with higher attention evaluation values ​​are arranged in front. On this basis, the second video frames corresponding to the first number of video frame identifications are selected and determined as the first target frames. Since these first target frames represent the key video content most relevant to the task description, the key frames in the video sequence can be quickly identified and extracted, thereby improving the pertinence and efficiency of video content processing.

[0090] After determining the first target frame, the other target frames except the first target frame in the at least two second video frames may be compressed to obtain a processed video sequence 405. Compression processing may refer to a process of encoding the target frame or reducing the amount of data to reduce the demand for storage space or transmission bandwidth. Compression processing may include removing redundant information, reducing resolution, using a specific encoding algorithm, etc.

[0091] In one example, other second video frames may be compressed according to the compression resolution determined by the context length limit to obtain compressed frames. The first number of first target frames and compressed frames are used as a processed video sequence. For example, the second video frame 1 may be compressed to obtain compressed frame 1; the second video frame 2 may be compressed to obtain compressed frame 2; the second video frame 4 may be compressed to obtain compressed frame 4. On this basis, compressed frame 1, compressed frame 2, second video frame 3, compressed frame 4, second video frame 5, ..., second video frame N may be determined as a processed video sequence 405.

[0092] According to the embodiments of the present disclosure, by determining the appropriate compression resolution according to the context length limit, it is ensured that the second video frame can still meet the requirements of large model processing after compression, while reducing the amount of data and improving the efficiency of storage and transmission. On this basis, by compressing the target frames other than the first number of first target frames according to the compression resolution, the obtained compressed frames eliminate the redundant information between the second video frames while maintaining the key visual information, reducing the space occupied by data. Since the size of the compressed video sequence is reduced, it helps to store more video sequences in a limited storage space, thereby facilitating the storage and transmission of video sequences. By combining the uncompressed first target frame and the compressed frame, the processed video sequence obtained not only retains the key frames of the video content, but also reduces the overall data volume by compressing other frames, making the processed video sequence more suitable for network transmission and storage, while meeting the efficiency requirements of large model processing, thereby improving the efficiency of video processing.

[0093] According to an embodiment of the present disclosure, by calculating the attention evaluation value between each second video frame and the task description, the first target frame most relevant to the task description can be identified. On this basis, by compressing the target frames other than the first target frame in the second video frame, the visual quality of the video sequence can be maintained as much as possible through the first target frame while reducing the volume of the video sequence, so that the compressed video sequence still has a good viewing experience. In addition, since the amount of data in the compressed video sequence is small, more video content can be stored in the same storage space, and the smaller file size can also speed up the upload and download speed of the video sequence, thereby improving the transmission efficiency. As a result, not only the key content of the video sequence is retained, but also the efficiency of video processing is significantly improved by removing or reducing the resolution or quality of non-key frames.

[0094] In another example, semantic matching information can also be determined based on the visual features of the second video frame and the text features of the task description, and the first target frame can be screened from at least two second video frames based on the respective semantic matching information of each second video frame. The semantic matching information can be used to characterize the semantic similarity or difference between the visual features and the text features. Alternatively, the first target frame can be obtained by determining which frames are most relevant to the task description through a pre-trained task classification model. On this basis, the other target frames in the at least two second video frames except the first target frame are also compressed to obtain a processed video sequence.

[0095] According to an embodiment of the present disclosure, by analyzing the visual features of each second video frame and the text features of the task description, the semantic matching information obtained can quantify the semantic similarity between the content of the second video frame and the task description, and provide an objective measure of the importance of the second video frame. On this basis, by determining the first target frame in the second video frame according to the respective semantic matching information of each second video frame, these first target frames contain visual information that is critical to completing the task. By compressing other target frames except these first target frames, the amount of data in the video stream is reduced while maintaining the integrity of key information. Thus, it is possible to improve the efficiency of storage and transmission by compressing non-key frames while ensuring that key visual content is not lost.

[0096] In the embodiment of the present disclosure, the video processing process may include a compression link based on temporal features and a compression link based on cross-modal features. Figure 5 Take as an example to illustrate the video processing process provided by an embodiment of the present disclosure.

[0097] Figure 5 An example schematic diagram of a video processing process according to an embodiment of the present disclosure is schematically shown.

[0098] like Figure 5 As shown, in 500, the video processing process is described by taking the video sequence 510 to be processed including 9 first video frames shot at intervals of 1 millisecond and the second width of the second window being 3 milliseconds as an example. In this example, the video processing process may include a compression link based on temporal features and a compression link based on cross-modal features.

[0099] For the compression link based on temporal features, the video sequence to be processed 510 can be subjected to window processing based on the second window to obtain a second number of second frame sets. For example, the second frame set 501 can include the first video frames from the 0th millisecond to the 2nd millisecond, that is, the first video frame 1, the first video frame 2, and the first video frame 3; the second frame set 502 can include the first video frames from the 3rd millisecond to the 5th millisecond, that is, the first video frame 4, the first video frame 5, and the first video frame 6; the third frame set 503 can include the first video frames from the 6th millisecond to the 8th millisecond, that is, the first video frame 7, the first video frame 9, and the first video frame 9.

[0100] After obtaining the second number of second frame sets, for each second frame set, the second video frames can be determined in each second frame set according to the timing characteristics between each first video frame in the set and other first video frames in the set except the first video frame. For example, the second video frame 2 can be determined in the second frame set 501, the second video frame 5 and the second video frame 6 can be determined in the second frame set 502, and the second video frame 7 can be determined in the second frame set 503.

[0101] For the compression link based on cross-modal features, the cross-modal features between the task description and the second video frame can be used to compress the target frames other than the first target frame in the second video frame. For example, if the first target frame in the second video frame 2, the second video frame 5, the second segment video frame 6, and the second video frame 7 is the second video frame 2 and the second video frame 6, the second video frame 5 can be compressed to obtain the compressed frame 5, and the second video frame 7 can be compressed to obtain the compressed frame 7. On this basis, the second video frame 2, the compressed frame 5, the second video frame 6, and the compressed frame 7 can be determined as the processed video sequence 504.

[0102] Through the compression link based on temporal features, redundant frames can be dynamically reduced based on inter-frame similarity, avoiding the problem of key frame loss in traditional uniform sampling methods. Through the compression link based on cross-modal features, the guidance of task description can be used to achieve task-driven key frame retention and improve the representation ability of semantically related features.

[0103] In the embodiment of the present disclosure, after obtaining the processed video sequence, the second target frame in the second video frame can also be compressed according to the spatial features between each second video frame and other second video frames to obtain a target video sequence. Figure 6 Take as an example to illustrate the compression process based on spatial features.

[0104] Figure 6An example schematic diagram of a process of compressing second target frames in second video frames according to spatial features between each second video frame and other second video frames to obtain a target video sequence according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 6 As shown, in 600, taking the processed video sequence 610 including 8 second video frames shot at intervals of 1 millisecond and the first width of the first window being 4 milliseconds as an example, the process of compressing the second target frames in the second video frames according to the spatial features between each second video frame and other second video frames to obtain the target video sequence is explained.

[0106] In one example, in response to the number of frames of the processed video sequence 610 exceeding the context length limit, the second target frame in the second video frame can be compressed according to the spatial features between each second video frame and other second video frames to obtain a target video sequence 605, so as to further reduce the spatial resolution and minimize the computational burden. Each second video frame includes at least two position information. The position information may refer to a two-dimensional coordinate position within the second video frame. The spatial feature characterizes the similarity of adjacent second video frames at at least two position information, and the second target frame may be determined according to the spatial feature.

[0107] According to an embodiment of the present disclosure, when the number of frames of the processed video sequence exceeds the context length limit, by performing spatial feature analysis on each second video frame, identifying and compressing second target frames that are similar to other frames in at least two position information, redundant data can be further reduced, which helps to store more video sequences in a limited storage space, and can maintain the visual quality of the video sequence as much as possible through the second target frame while reducing the volume of the video sequence, so that the compressed video sequence still has a good viewing experience. In addition, since the amount of data in the compressed video sequence is small, more video content can be stored in the same storage space, and the smaller file size can also speed up the upload and download speed of the video sequence, thereby improving the transmission efficiency of the video efficiency.

[0108] For example, based on the first window, the processed video sequence 610 can be subjected to window processing to obtain a first number of first frame sets. The first window can be a sliding window to ensure that each second video frame can be comprehensively considered in each window, which is conducive to capturing time dependency. For example, the first frame set 601 can include the second video frames from the 0th millisecond to the 3rd millisecond, that is, the second video frame 1, the second video frame 2, the second video frame 3, and the second video frame 4; the first frame set 602 can include the second video frames from the 4th millisecond to the 7th millisecond, that is, the second video frame 5, the second video frame 6, the second video frame 7, and the second video frame 8.

[0109] After obtaining the first number of first frame sets, the similarity of each piece of position information may be determined for each first frame set according to each piece of position information.

[0110] Taking the first frame set 602 as an example, the first frame in the first frame set 602, that is, the second video frame 5, can be used as a reference, and the second video frame 5 and other frames in the first frame set 602 can be respectively analyzed pixel by pixel for similarity to obtain the similarity of each position information. For example, the first similarity between the second video frame 5 and the second video frame 6 at the position information 603_1, the second similarity between the second video frame 5 and the second video frame 7 at the position information 603_1, and the third similarity between the second video frame 5 and the second video frame 8 at the position information 603_1 can be determined, and the similarity 604_1 of the second video frame 5 at the position information 603_1 is determined based on the first similarity, the second similarity and the third similarity. Similarly, the similarity 604_2 of the second video frame 5 at the position information 603_2 can be determined.

[0111] In a specific example, the similarity at each location information may be determined as shown in the following formula (5).

[0112]

[0113] in, Represents position information, h represents the vertical position, w represents the horizontal position, represents the pixel value of the first frame in the first frame set at the position information, Representing the pixel values ​​of other frames in the first frame set at the position information, Characterizes the similarity between the first frame in the first frame set and other frames in the first frame set at the position information.

[0114] After obtaining the respective similarities of each position information, for each second video frame in each first frame set, in response to the existence of a similarity greater than a similarity threshold in the similarities of the position information, the second video frame can be determined as a second target frame. In one example, a second video frame having more position information with higher similarities can be determined as a second target frame. After obtaining the second target frame, for each second target frame, the resolution at the position information with a similarity greater than the similarity threshold can be compressed according to the position information with higher similarity, so as to obtain a target video sequence 605.

[0115] In a specific example, the compression method of the second target frame may be as shown in the following formula (6).

[0116]

[0117] in, Represents position information, h represents the vertical position, w represents the horizontal position, Characterizes the similarity threshold, which can be configured according to actual business needs and is not limited here. For example, the similarity threshold can be 0.8~0.9 to balance between retaining information and reducing tokens.

[0118] According to the embodiments of the present disclosure, for each second video frame, by comparing the similarity of at least two position information with a preset similarity threshold, the second target frame with highly similar pixels is intelligently identified, thereby avoiding unnecessary compression processing. On this basis, for each determined second target frame, by compressing the resolution at the position information where the similarity is greater than the similarity threshold, this local compression strategy effectively reduces the amount of data while maintaining the coherence of the video content.

[0119] According to an embodiment of the present disclosure, by performing window processing on the processed video stream based on the first window, the processed video sequence is divided into at least two first frame sets according to the width of the first window, and each first frame set contains a fixed number of second video frames. For each first frame set, the similarity of each position information is determined according to each position information, and the second target frame with highly similar pixels is identified based on the similarity of each position information. On this basis, by compressing these second target frames, the data volume of the video stream can be reduced while maintaining the integrity of the key visual content.

[0120] In another example, for each second video frame, a region to be compressed can be determined based on the attention distribution between the second video frame and other second video frames, and the region to be compressed can represent an unimportant feature region. On this basis, the resolution in the region to be compressed can be compressed to obtain a processed video frame 605. In addition, the region of interest in the second video frame can be determined, and the high-resolution features of the region of interest are retained, and the remaining regions are reduced in dimension.

[0121] According to the embodiments of the present disclosure, for each second video frame, the area to be compressed is intelligently determined based on the attention distribution between the second video frame and other second video frames, thereby ensuring the pertinence and effectiveness of the compression process. On this basis, by compressing the resolution in the area to be compressed to obtain a processed video frame, it is possible to improve the efficiency of storage and transmission by compressing video frames with high similarity while ensuring that key visual information is not lost.

[0122] In the embodiment of the present disclosure, the video processing process may further include a compression link based on temporal features, a compression link based on cross-modal features, and a compression link based on spatial features. Figure 7 Take as an example to illustrate the video processing process provided by an embodiment of the present disclosure.

[0123] Figure 7 An example schematic diagram of a video processing process according to another embodiment of the present disclosure is schematically shown.

[0124] like Figure 7 As shown, in 700, the video processing process is described by taking the video sequence 710 to be processed including 9 first video frames shot at intervals of 1 millisecond and the second width of the second window being 3 milliseconds as an example. In this example, the video processing process may include a compression link based on temporal features, a compression link based on cross-modal features, and a compression link based on spatial features.

[0125] For the compression link based on temporal features, the video sequence to be processed 710 can be subjected to window processing based on the second window to obtain a second number of second frame sets. For example, the second frame set 701 can include the first video frame from the 0th millisecond to the 2nd millisecond, that is, the first video frame 1, the first video frame 2, and the first video frame 3; the second frame set 702 can include the first video frame from the 3rd millisecond to the 5th millisecond, that is, the first video frame 4, the first video frame 5, and the first video frame 6; the third frame set 703 can include the first video frame from the 6th millisecond to the 8th millisecond, that is, the first video frame 7, the first video frame 9, and the first video frame 9.

[0126] After obtaining the second number of second frame sets, for each second frame set, the second video frames can be determined in each second frame set according to the timing characteristics between each first video frame in the set and other first video frames in the set. For example, the second video frame 2 can be determined in the second frame set 701, the second video frame 5 and the second video frame 6 can be determined in the second frame set 702, and the second video frame 7 can be determined in the second frame set 703.

[0127] For the compression link based on cross-modal features, the cross-modal features between the task description and the second video frame can be used to compress the target frames other than the first target frame in the second frame. For example, if the first target frame in the second video frame 2, the second video frame 5, the second video frame 6, and the second video frame 7 is the second video frame 2 and the second video frame 6, the second video frame 5 can be compressed to obtain the compressed frame 5, and the second video frame 7 can be compressed to obtain the compressed frame 7.

[0128] For the compression link based on spatial features, for each second video frame or compressed frame, by comparing the similarity of at least two pieces of position information with a preset similarity threshold, a second target frame with highly similar pixels can be intelligently identified. For example, the second video frame 2, the compressed frame 5, the second video frame 6, and the compressed frame 7 can be determined as the second target frame 704_1, and the compressed frame 7 can be determined as the second target frame 704_2.

[0129] For each determined second target frame, the resolution at the position information whose similarity is greater than the similarity threshold can be compressed. For example, the position information 705_1 in the second target frame 704_1 can be compressed to obtain the compressed second target frame 2; the position information 706_1 in the second target frame 704_2 can be compressed to obtain the compressed second target frame 7. On this basis, the compressed second target frame 2, the compressed frame 5, the second video frame 6 and the compressed second target frame 7 can be determined as the target video sequence 706.

[0130] Through the above-mentioned compression link based on spatial features, spatial tokens can be further compressed through pixel-level similarity within the frame. On this basis, by combining the multi-level compression mechanism based on temporal features, cross-modal features and spatial features, the model can efficiently process long videos within a limited context length, while taking into account both accuracy and efficiency.

[0131] The above are only exemplary embodiments, but are not limited thereto, and may also include other video processing methods known in the art as long as they can improve the efficiency and accuracy of video processing.

[0132] It should be noted that in the embodiments of the present invention, certain software, components, models and other existing solutions in the industry may be mentioned, which should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present invention, but it does not mean that the present invention has or must use this solution.

[0133] Based on the above video processing method, the present invention also provides a video processing device. Figure 8 The device is described in detail.

[0134] Figure 8 The block diagram schematically shows a video processing device according to an embodiment of the present disclosure.

[0135] like Figure 8 As shown, the video processing device 800 may include a determination module 810 and a first compression processing module 820 .

[0136] The determination module 810 is used to determine at least two second video frames in the first video frames for at least two first video frames in the video sequence to be processed, according to the timing features between each first video frame and other video frames in the video sequence to be processed except the first video frame, wherein the timing features characterize the similarity between adjacent video frames.

[0137] The first compression processing module 820 is used to compress the target frames other than the first target frame in the second video frame according to the cross-modal features between the task description and the second video frame to obtain a processed video sequence, wherein the first target frame is determined according to the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frame.

[0138] According to an embodiment of the present disclosure, the first compression processing module 820 may include a first determination submodule, a second determination submodule and a first compression processing submodule.

[0139] The first determination submodule is used to determine an attention evaluation value between each second video frame and the task description.

[0140] The second determination submodule is used to determine the first target frame in the second video frame according to the attention evaluation value.

[0141] The first compression processing submodule is used to perform compression processing on other target frames to obtain a processed video sequence.

[0142] According to an embodiment of the present disclosure, each second video frame includes at least two pieces of position information.

[0143] According to an embodiment of the present disclosure, the first determining submodule may include a first determining unit and a normalizing unit.

[0144] The first determination unit is used to determine an attention weight according to the visual features of the second video frame and the text features of the task description, wherein the attention weight represents the global correlation between each position information and the task description.

[0145] The normalization unit is used to normalize the attention weight according to the initial resolution of the video sequence to be processed and the number of tokens after the task description is processed by word segmentation to obtain an attention evaluation value.

[0146] According to an embodiment of the present disclosure, the second determining submodule may include a sorting unit and a second determining unit.

[0147] The sorting unit is used to sort the second video frames in descending order according to the attention evaluation value of each second video frame to obtain a video frame identification sequence.

[0148] The second determining unit is used to determine the second video frame of each of the first number of identifiers in the video frame identifier sequence as the first target frame.

[0149] According to an embodiment of the present disclosure, the first target frame represents a frame that does not need to be compressed, and the other target frames represent frames that need to be compressed, and the first number is determined based on a target constraint. The target constraint is used to adjust the number of the first target frames and the number of the other target frames under a context length limit so as to maximize the token utilization of the large model used to process the video sequence to be processed, wherein the context length limit is determined based on the maximum number of tokens of the large model.

[0150] According to an embodiment of the present disclosure, the first compression processing submodule may include a first compression processing unit and a third determination unit.

[0151] The first compression processing unit is used to perform compression processing on other target frames according to a compression resolution to obtain compressed frames, wherein the compression resolution is determined according to a context length restriction.

[0152] The third determining unit is configured to use the first number of first target frames and the compressed frames as a processed video sequence.

[0153] According to an embodiment of the present disclosure, the first compression processing module 820 may further include a third determination submodule, a fourth determination submodule and a second compression processing submodule.

[0154] The third determination submodule is used to determine semantic matching information according to the visual features of the second video frame and the text features of the task description.

[0155] The fourth determination submodule is used to determine the first target frame in the second video frames according to the semantic matching information of each second video frame.

[0156] The second compression processing submodule is used to perform compression processing on other target frames to obtain a processed video sequence.

[0157] According to an embodiment of the present disclosure, the video processing device 800 may further include a second compression processing module.

[0158] The second compression processing module is used for compressing the second target frame in the second video frame according to the spatial features between each second video frame and other second video frames in response to the number of frames of the processed video sequence exceeding the context length limit to obtain a target video sequence. Each second video frame includes at least two position information, the spatial features characterize the similarity of adjacent second video frames at at least two position information, and the second target frame is determined according to the spatial features.

[0159] According to an embodiment of the present disclosure, the second compression processing module 820 may include a first windowing submodule, a fifth determining submodule and a third compression processing submodule.

[0160] The first window submodule is used to perform window processing on the processed video sequence based on the first window to obtain a first number of first frame sets.

[0161] The fifth determination submodule is used to determine the similarity of each piece of position information according to each piece of position information for each first frame set.

[0162] The third compression processing submodule is used to compress the second target frame in the first frame set based on the similarity of each position information to obtain a target video sequence.

[0163] According to an embodiment of the present disclosure, the third compression processing submodule may include a fourth determination unit and a second compression processing unit.

[0164] The fourth determining unit is configured to determine the second video frame as the second target frame in response to a similarity greater than a similarity threshold among the similarities of the respective position information.

[0165] The second compression processing unit is used to compress the resolution of the position information at which the similarity in the second target frame is greater than the similarity threshold to obtain a processed video frame.

[0166] According to an embodiment of the present disclosure, the second compression processing module 820 may include a sixth determination submodule and a fourth compression processing submodule.

[0167] The sixth determination submodule is used to determine the area to be compressed based on the attention distribution between the second video frame and other second video frames.

[0168] The fourth compression processing submodule is used to compress the resolution in the to-be-compressed area to obtain a processed video frame.

[0169] According to an embodiment of the present disclosure, the determination module 810 may include a second windowing submodule, a first analysis submodule and a seventh determination submodule.

[0170] The second window submodule is used to perform window processing on the video sequence to be processed based on the second window to obtain a second number of second frame sets.

[0171] The first analysis submodule is used to perform similarity analysis on at least two first video frames in each second frame set to obtain similarity information, wherein the similarity information includes an average similarity between each first video frame and other first video frames in the second frame set.

[0172] The seventh determination submodule is used to determine at least two second video frames in the first video frame according to the similarity information.

[0173] According to an embodiment of the present disclosure, the determination module 810 may include a second analysis submodule and an eighth determination submodule.

[0174] The second analysis submodule is used to perform semantic analysis on the visual features of each first video frame by using a semantic model to obtain a key point extraction result.

[0175] An eighth determination submodule is used to determine at least two second video frames in the first video frame according to the key point extraction result.

[0176] Fig. 9 The structural block diagram of the intelligent agent of the large model according to the embodiment of the present disclosure is schematically shown.

[0177] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig. 9 As shown, the AI ​​agent 900 may include five core modules: an input module 910 , a processing module 920 , and an output module 950 .

[0178] The input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world. It enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0179] In an example, the input module 910 may input the video sequence to be processed described above.

[0180] The processing module 920 can be used to determine the target task based on the input information received by the input module 910, determine the large model based on the target task, obtain output information by calling the large model to execute the video processing method described above, and pass the output information to the output module 930.

[0181] In an example, the processing module 920 may perform the video processing method as described above by calling the large model to obtain a processed video sequence. In another example, the processing module 920 may perform the video processing method as described above by calling the large model to obtain a target video sequence.

[0182] In a specific example, the processing module 920 may include a control module 921, a storage module 922, and a computing module 923. The control module 921 is the core support for the AI ​​agent 900 to process complex tasks. The control module 921 may execute the video processing method described above.

[0183] In the example, the control module 921 will continuously interact with the storage module 922, the computing module 923 and / or the output module 930 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 921 acts as a single initiator to initiate communication with the storage module 922, the computing module 923 and / or the output module 930, and there is no communication coupling between the storage module 922, the computing module 923 and the output module 930.

[0184] In the example, the performance of the control module 921 may be closely related to the large model on which the AI ​​agent 900 is based. In order to give full play to the capabilities of the large language model, the internal structure of the control module 921 may be designed to be highly configurable and scalable to cope with various types of tasks and requirements in real scenarios.

[0185] The storage module 922 may be responsible for memorizing information such as historical conversations, event streams, etc. The at least two second video frames and the processed video sequence as described above may be included in the storage module 922 .

[0186] In the example, after the AI ​​agent 900 obtains the video sequence to be processed, the AI ​​agent 900 may determine to process the video sequence to be processed to obtain at least two second video frames. The at least two second video frames may be stored in the storage module 922. The AI ​​agent 900 may obtain at least two second video frames from the storage module 922 and feed them back to the control module 921. Then, the control module 921 may further process the at least two second video frames fed back to obtain a processed video sequence, and pass the processed video sequence to the output module 930.

[0187] The operation module 923 may be viewed as a predefined tool library, and the tools for determining temporal features, cross-modal features, spatial features, etc. as described above may be included in the operation module 923 .

[0188] In the example, when the AI ​​agent 900 needs to determine the timing features, the relevant tool for determining the timing features can be called from the operation module 923 and fed back to the control module 921. Then, the control module 921 can use the tool for determining the timing features fed back to process the video sequence to be processed, obtain the respective timing features of each video frame, and pass the respective timing features of each video frame to the output module 930. It can be understood that although the large language model has excellent language understanding and generation capabilities, it is the same as a human being, and the tasks that can be solved without the help of any tools are very limited. When the AI ​​agent 900 is given the ability to call tools, it can achieve tasks such as completing the determination of timing features with the help of tools for determining timing features.

[0189] The output module 930 may output the processed video sequence and the target video sequence described above.

[0190] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0191] Fig.10 A block diagram of an electronic device suitable for implementing a video processing method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0192] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0193] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0194] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as video processing methods. For example, in some embodiments, the video processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the video processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the video processing method in any other appropriate manner (e.g., by means of firmware).

[0195] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0196] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0197] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0198] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0199] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0200] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0201] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0202] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video processing method, comprising: For at least two first video frames in a video sequence to be processed, determining at least two second video frames in the first video frames according to a temporal feature between each of the first video frames and other video frames in the video sequence to be processed except the first video frame, wherein the temporal feature represents a similarity between adjacent video frames; and According to the cross-modal features between the task description and the second video frame, compression processing is performed on other target frames in the second video frame except the first target frame to obtain a processed video sequence, wherein the first target frame is determined according to the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frame.

2. The method according to claim 1, wherein: The step of compressing the target frames other than the first target frame in the second video frame according to the cross-modal feature between the task description and the second video frame to obtain a processed video sequence includes: determining an attention evaluation value between each of the second video frames and the task description; determining the first target frame in the second video frame according to the attention evaluation value; and The other target frames are compressed to obtain the processed video sequence.

3. The method according to claim 2, wherein: Each of the second video frames includes at least two pieces of position information; Determining the attention evaluation value between each of the second video frames and the task description comprises: Determining an attention weight according to visual features of the second video frame and text features of the task description, wherein the attention weight represents a global correlation between each of the position information and the task description; and The attention weight is normalized according to the initial resolution of the video sequence to be processed and the number of tokens of the task description after word segmentation processing to obtain the attention evaluation value.

4. The method according to claim 2, wherein: The determining the first target frame in the second video frame according to the attention evaluation value comprises: Sort the second video frames in descending order according to the attention evaluation value of each second video frame to obtain a video frame identification sequence; and The second video frame of each of the first number of identifiers in the video frame identifier sequence is determined as the first target frame.

5. The method according to claim 4, wherein: The first target frame represents a frame that does not need to be compressed, the other target frames represent frames that need to be compressed, and the first number is determined based on a target constraint; The target constraint is used to adjust the number of the first target frames and the number of the other target frames under the context length limit so as to maximize the token utilization of the large model used to process the video sequence to be processed, wherein the context length limit is determined based on the maximum number of tokens of the large model.

6. The method according to claim 4, wherein: The compressing the other target frames to obtain the processed video sequence comprises: compressing the other target frames according to a compression resolution to obtain compressed frames, wherein the compression resolution is determined according to the context length limit; and The first number of first target frames and the compressed frames are taken as the processed video sequence.

7. The method according to claim 1, wherein: The step of compressing the target frames other than the first target frame in the second video frame according to the cross-modal feature between the task description and the second video frame to obtain a processed video sequence includes: Determining semantic matching information based on visual features of the second video frame and text features of the task description; Determining the first target frame in the second video frame according to the semantic matching information of each of the second videos; and The other target frames are compressed to obtain the processed video sequence.

8. The method according to any one of claims 1 to 7, further comprising: In response to the number of frames of the processed video sequence exceeding the context length limit, compressing second target frames in the second video frames according to spatial features between each of the second video frames and other second video frames to obtain a target video sequence; Each of the second video frames includes at least two pieces of position information, the spatial feature represents the similarity of adjacent second video frames at each piece of position information, and the second target frame is determined based on the spatial feature.

9. The method according to claim 8, wherein: The compressing the second target frames in the second video frames according to the spatial features between each of the second video frames and other second video frames to obtain the target video sequence comprises: Based on the first window, performing window processing on the processed video sequence to obtain a first number of first frame sets; For each of the first frame sets, determining the similarity of each of the position information according to each of the position information; and Based on the similarity of each of the position information, the second target frame in the first frame set is compressed to obtain a target video sequence.

10. The method according to claim 9, wherein: The compressing the second target frame in the first frame set based on the similarity of each position information to obtain the target video sequence includes: In response to a similarity greater than a similarity threshold among the similarities of the position information, determining the second video frame as the second target frame; The resolution of the position information in the second target frame where the similarity is greater than the similarity threshold is compressed to obtain a processed video frame.

11. The method according to claim 8, wherein: The compressing the second target frames in the second video frames according to the spatial features between each of the second video frames and other second video frames to obtain the target video sequence comprises: Determining a region to be compressed based on attention distribution between the second video frame and other second video frames; and The resolution in the area to be compressed is compressed to obtain a processed video frame.

12. The method according to any one of claims 1 to 7, wherein: The determining at least two second video frames in the first video frame according to the time characteristics between each of the first video frames and other video frames in the video sequence to be processed except the first video frame comprises: Based on the second window, performing window processing on the video sequence to be processed to obtain a second number of second frame sets; For each of the second frame sets, Performing similarity analysis on at least two first video frames in the second frame set to obtain similarity information, wherein the similarity information includes an average similarity between each of the first video frames and other first video frames in the second frame set; and The at least two second video frames are determined in the first video frame according to the similarity information.

13. The method according to any one of claims 1 to 7, wherein: The determining at least two second video frames in the first video frame according to the time characteristics between each of the first video frames and other video frames in the video sequence to be processed except the first video frame comprises: Using a semantic model, performing semantic analysis on visual features of each of the first video frames to obtain key point extraction results; and According to the key point extraction result, the at least two second video frames are determined in the first video frame.

14. A video processing device, comprising: a determination module, configured to determine, for at least two first video frames in a video sequence to be processed, at least two second video frames in the first video frames according to a temporal feature between each of the first video frames and other video frames in the video sequence to be processed except the first video frame, wherein the temporal feature represents a similarity between adjacent video frames; and The first compression processing module is used to compress the other target frames in the second video frame except the first target frame according to the cross-modal features between the task description and the second video frame to obtain a processed video sequence, wherein the first target frame is determined according to the cross-modal features, and the cross-modal features characterize the similarity between the task description and the second video frame.

15. The device according to claim 14, wherein: The first compression processing module comprises: A first determination submodule, configured to determine an attention evaluation value between each of the second video frames and the task description; a second determination submodule, configured to determine the first target frame in the second video frame according to the attention evaluation value; and The first compression processing submodule is used to perform compression processing on the other target frames to obtain the processed video sequence.

16. The device according to claim 15, wherein: Each of the second video frames includes at least two pieces of position information; The first determining submodule includes: A first determining unit, configured to determine an attention weight according to visual features of the second video frame and text features of the task description, wherein the attention weight represents a global correlation between each of the position information and the task description; and A normalization unit is used to normalize the attention weight according to the initial resolution of the video sequence to be processed and the number of tokens of the task description after word segmentation processing to obtain the attention evaluation value.

17. The device according to claim 15, wherein: The second determining submodule includes: a sorting unit, configured to sort the second video frames in descending order according to the attention evaluation value of each of the second video frames to obtain a video frame identification sequence; and The second determining unit is configured to determine the second video frames of the first first number of identifiers in the video frame identifier sequence as the first target frame.

18. The device according to claim 17, wherein: The first target frame represents a frame that does not need to be compressed, the other target frames represent frames that need to be compressed, and the first number is determined based on a target constraint; The target constraint is used to adjust the number of the first target frames and the number of the other target frames under the context length limit so as to maximize the token utilization of the large model used to process the video sequence to be processed, wherein the context length limit is determined based on the maximum number of tokens of the large model.

19. The device according to claim 17, wherein: The first compression processing submodule includes: a first compression processing unit, configured to perform compression processing on the other target frames according to a compression resolution to obtain compressed frames, wherein the compression resolution is determined according to the context length limit; and The third determining unit is configured to use the first number of first target frames and the compressed frames as the processed video sequence.

20. The device according to claim 14, wherein: The first compression processing module also includes: A third determination submodule, configured to determine semantic matching information based on visual features of the second video frame and text features of the task description; a fourth determination submodule, configured to determine the first target frame in the second video frames according to the semantic matching information of each of the second video frames; and The second compression processing submodule is used to perform compression processing on the other target frames to obtain the processed video sequence.

21. The device according to any one of claims 14 to 20, further comprising: A second compression processing module is configured to compress second target frames in the second video frames according to spatial features between each of the second video frames and other second video frames in response to the number of frames of the processed video sequence exceeding the context length limit, so as to obtain a target video sequence; Each of the second video frames includes at least two pieces of position information, the spatial feature represents the similarity between adjacent second video frames at each piece of position information, and the second target frame is determined based on the spatial feature.

22. The device according to claim 21, wherein The second compression processing module includes: A first windowing submodule, configured to perform windowing processing on the processed video sequence based on a first window to obtain a first number of first frame sets; a fifth determining submodule, configured to determine, for each of the first frame sets, a similarity of each of the position information according to each of the position information; and The third compression processing submodule is used to compress the second target frame in the first frame set based on the similarity of each position information to obtain a target video sequence.

23. The device according to claim 22, wherein: The third compression processing submodule includes: a fourth determining unit, configured to determine the second video frame as the second target frame in response to a similarity greater than a similarity threshold among the similarities of the respective position information; The second compression processing unit is used to compress the resolution of the position information at which the similarity in the second target frame is greater than the similarity threshold to obtain a processed video frame.

24. The device according to claim 21, wherein The second compression processing module includes: A sixth determination submodule, configured to determine a to-be-compressed area based on the attention distribution between the second video frame and other second video frames; The fourth compression processing submodule is used to compress the resolution in the area to be compressed to obtain a processed video frame.

25. The device according to any one of claims 14 to 20, wherein: The determination module comprises: A second windowing submodule, configured to perform windowing processing on the video sequence to be processed based on a second window to obtain a second number of second frame sets; a first analysis submodule, configured to perform similarity analysis on at least two first video frames in each second frame set to obtain similarity information, wherein the similarity information includes an average similarity between each first video frame and other first video frames in the second frame set; and A seventh determination submodule is configured to determine the at least two second video frames in the first video frame according to the similarity information.

26. The device according to any one of claims 14 to 20, wherein: The determination module comprises: a second analysis submodule, configured to perform semantic analysis on visual features of each of the first video frames using a semantic model to obtain a key point extraction result; and An eighth determination submodule is used to determine the at least two second video frames in the first video frame according to the key point extraction result.

27. An intelligent agent comprises: An input module, used for receiving input information; a processing module, configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and obtain output information by calling the large model to execute the method of claims 1 to 13; as well as An output module is used to output the output information obtained by the processing module.

28. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 13.

29. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

30. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images