Video processing method and device, electronic equipment and storage medium

Abstract

The invention provides a video processing method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, in particular to the fields of large models, deep learning and image processing. According to the specific implementation scheme, the method comprises the steps of determining a key frame in image frames based on visual features of each image frame in T image frames of a video; based on the event prototype corresponding to the key frame, coding the key frame to obtain coding information of the key frame; on the basis of features of a text associated with the video processing task, encoding other image frames except the key frame in the image frames to obtain encoding information of other image frames; and obtaining a video processing result based on the coding information of the key frame and the coding information of other image frames.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to the fields of large models, deep learning, and image processing. Background Art

[0002] With the success of Large Language Model (LLM) in the field of artificial intelligence, its application in vision-language fusion has been greatly expanded. In some application scenarios, it is necessary to fully understand the content of the video, and encoding the video frames is an important part of understanding the video. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device and storage medium for video processing.

[0004] According to one aspect of the present disclosure, there is provided a video processing method, comprising:

[0005] Determine a key frame in the image frame based on a visual feature of each of T image frames of the video; wherein T is an integer greater than or equal to 2;

[0006] Based on the event prototype corresponding to the key frame, the key frame is encoded to obtain the encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame;

[0007] Based on the features of the text associated with the video processing task, encoding other image frames except the key frames in the image frames to obtain encoding information of the other image frames;

[0008] Based on the encoding information of the key frame and the encoding information of other image frames, a video processing result is obtained.

[0009] According to another aspect of the present disclosure, there is provided a video processing device, comprising:

[0010] A key frame determination module, configured to determine a key frame in an image frame based on visual features of each of T image frames of a video; wherein T is an integer greater than or equal to 2;

[0011] A key frame encoding module is used to encode the key frame based on the event prototype corresponding to the key frame to obtain the encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame;

[0012] The other image frame encoding module is used to encode other image frames except the key frames in the image frames based on the features of the text associated with the video processing task to obtain encoding information of the other image frames;

[0013] The result determination module is used to obtain the video processing result based on the encoding information of the key frame and the encoding information of other image frames.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] The technical solution of the disclosed embodiment can retain the event details of the key frame as much as possible by encoding the key frame using the event prototype corresponding to the key frame, and can determine and retain the information related to the text in the other image frames and compress the content not related to the text by using the features of the text associated with the video processing task to encode the other image frames. Therefore, the technical solution of the disclosed embodiment can make the best use of the features of the key frame and other image frames, retain the key event information of the video while reducing the memory usage, so as to obtain good coding efficiency and information integrity.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a video processing method provided by an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of an application example of the video processing method provided by an embodiment of the present disclosure;

[0025] Figure 3 is a schematic block diagram of a video processing device provided by an embodiment of the present disclosure;

[0026] Figure 4 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0027] Figure 5 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0028] Figure 6 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0029] Figure 7 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0030] Figure 8 The block diagram is a block diagram of an electronic device for implementing the video processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] The video processing method provided by the embodiments of the present disclosure can be applied to scenarios where video semantics need to be understood, such as video retrieval, video question answering, video summarization, etc. Specifically, it can be applied to the process of determining the relevance between video frames and text input by users.

[0033] Figure 1 The video processing method provided by an embodiment of the present disclosure is shown. The method can be applied to a video processing device, which can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal can be a user equipment (User Equipment, UE) such as a mobile device, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.; the server can be a stand-alone server or a server cluster. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the method may include the following steps S110 to S140.

[0034] Step S110 , determining a key frame in the image frame based on the visual features of each of the T image frames of the video; wherein T is an integer greater than or equal to 2.

[0035] In the disclosed embodiment, the video may be any video in the video library, such as a video selected by a user. In some application scenarios, the user may specify a video and input text, and the video processing device may encode the video according to the text to compress the video, or extract content associated with the text from the video, or generate summary information, reply information, etc. associated with the text.

[0036] In the disclosed embodiment, T may be the number of image frames in the video, and visual features of each image frame may be extracted for each of the T image frames. The visual features of an image frame may be understood as reflecting the visual content of the image frame. For example, the visual features of an image frame may be features extracted from the image frame using a convolutional neural network. For another example, the visual features of an image frame may include brightness, contrast, color distribution, texture features, etc. of the image frame.

[0037] Optionally, a visual encoder can be used to extract features from each frame of the video to obtain a set of visual features corresponding to the video. where v t ∈R N×d represents the features of the tth frame, R represents a two-dimensional matrix with a size of N×d, N is the number of image blocks, and d is the feature dimension, that is, v t is a two-dimensional matrix of size N×d. For example, the visual encoder can use a pre-trained ViT model (EVA-CLIP). Optionally, the visual features of each image frame can also be pooled to reduce the dimension of the visual features. For example, v t Through spatial pooling, the dimension is reduced to f t =Pool(v t ), where Pool() represents pooling, f t ∈R P×d , where P is the number of blocks after pooling.

[0038] Optionally, the key frame may include one or more video frames representing the core content of the video among the T image frames. In other words, the key frame may be a redundant video frame without repeated frames, header frames, and tail frames.

[0039] Optionally, the video processing device can determine the key frames in T image frames through clustering. The purpose of clustering can be to divide the T image frames into multiple classes / clusters. Through clustering, the clustering index of each image frame can be obtained, and the clustering index indicates whether a single image frame is the center of a class / cluster, and the distance between the image frame and other image frames. Using the clustering index, the representative frame of each class / cluster can be determined in the T image frames, and the key frames are determined based on this. Exemplarily, the visual features of each image frame can be processed based on the density peak clustering algorithm (Density Peaks Clustering based on K-NearestNeighbor, DPC-KNN) to obtain the local density index and distance index of each image frame. Based on the local density index and distance index of each image frame, the importance score of each image frame is obtained, and based on the importance score of each image frame, it can be determined whether the image frame is a key frame.

[0040] Optionally, the video processing device may generate a frame importance mask b={b t}, when the image frame is a key frame, mark the mask value b corresponding to the image frame t =1, otherwise the mask value corresponding to the image frame is marked as b t =0, thereby dividing the image frames into key frames and other image frames, wherein the other image frames can also be called redundant frames.

[0041] Optionally, the key frame may also include one or more video frames associated with the text in the T image frames. For example, if the text is "What is the color in the background?", the key frame may include a video frame with background details.

[0042] Step S120: Encode the key frame based on the event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame.

[0043] In the disclosed embodiment, a key frame may correspond to an event prototype. The event prototype may be understood as an abstract representation describing an important event associated with the key frame. The event prototype may include aggregate features extracted from the visual features of the key frame and the visual features of the neighboring frames of the key frame. Therefore, compared with the visual features of the key frame, the event prototype corresponding to the key frame integrates the visual features of the neighboring frames, retains the temporal dynamic features of the video, and generates a feature with more global representation power. Optionally, the event prototype can be obtained by de-redundancy and aggregation, so that a more compact feature can be generated, which can reduce the consumption of computing resources for subsequent processing, while highlighting the core information of the event.

[0044] Optionally, the visual features of the key frame and the visual features of the key frame's neighboring frames can be weighted and aggregated to obtain an event prototype. The event prototype may include statistical features of the key frame and its redundant frames, such as the mean and maximum values ​​of each visual feature, so as to extract important information about the event. The statistical features of the event prototype can be used to encode the key frame to obtain the encoding information of the key frame.

[0045] In the disclosed embodiment, based on the event prototype corresponding to the key frame, the key frame can be finely encoded (for example, high-resolution encoding), without pooling operation during the encoding process, retaining spatial details, and obtaining the encoding information of the key frame.

[0046] Optionally, the encoding information of the key frame may include the spatial elements in the video picture and the presentation method of the spatial elements, such as the composition of the video picture, the layout of objects, light and shadow effects, color matching, etc.

[0047] Optionally, the number of key frames may be one or more, and each key frame is encoded based on an event prototype corresponding to the key frame to obtain encoding information.

[0048] Step S130: Based on the features of the text associated with the video processing task, encode the image frames other than the key frames in the image frames to obtain encoding information of the other image frames.

[0049] Exemplarily, the text associated with the video processing task may include query information for a user to initiate a query / retrieval, or a question raised by a user regarding the video, or a statement by a user describing the video processing task. The features of the text may represent information about the content or structure of the text, for example, the features of the text may include semantic features, structural features, etc.

[0050] Optionally, the text can be converted into features via a text encoder where q r ∈R d' , R is a real number, specifically the number of text words, and d' is the text feature dimension.

[0051] In the disclosed embodiment, image frames other than key frames are redundant frames in the video. The features of the text can be used to guide the video processing device to perform coarse-grained encoding (such as low-resolution encoding) on ​​other image frames. For example, content related to the text can be determined and retained in other image frames, and content unrelated to the text can be compressed, rather than relying solely on the global average of visual features. This can reduce memory pressure while retaining the temporal dynamics of the video and obtaining encoding information for other image frames.

[0052] Step S140: Obtain a video processing result based on the encoding information of the key frame and the encoding information of other image frames.

[0053] In the disclosed embodiment, the coding information of the key frame can be combined with the coding information of other image frames. The coding information of the key frame can retain the event details, and the coding information of other image frames can retain the information related to the text in other image frames, thereby obtaining the coding result of the entire video. The coding result of the video can be decoded to obtain the processing result of the video. The processing result of the video can be text or video, and has a high correlation with the input text. Optionally, the processing result of the video can be displayed to the user, thereby completing tasks such as video retrieval, video question and answer, and video summary.

[0054] In the related art, the understanding of videos still faces significant challenges, especially in long videos where there is a lot of redundant information between video frames, such as repeated frames and frames irrelevant to the problem, resulting in excessive memory usage. In addition, different problems have different focuses on video frames, and this semantic relevance is difficult to be fully reflected in existing static coding methods. Although average pooling or attention mechanisms can reduce the coding cost of video frames, key visual details or temporal dynamics are often lost. The above method provided by the embodiment of the present disclosure can retain the event details of the key frame as much as possible by encoding the key frame using the event prototype corresponding to the key frame, and by encoding other image frames using the features of the text associated with the video processing task, it can determine and retain information related to the text in other image frames, and compress content that is not related to the text. Therefore, the technical solution of the embodiment of the present disclosure can make optimal use of the features of key frames and other image frames, while reducing memory usage, retaining the key event information of the video, thereby obtaining good coding efficiency and information integrity.

[0055] In some embodiments, the above step S140, based on the encoding information of the key frame and the encoding information of other image frames, obtains the processing result of the video, including:

[0056] Based on the encoding information of the key frame and the encoding information of other image frames, one or more of a target segment in the video, summary information of the video, and reply information for the text are determined.

[0057] In the disclosed embodiments, it can be applied to video question-answering or other scenarios that require understanding of video semantics. For example: video retrieval (for example, quickly finding relevant clips based on text questions), video summarization (for example, extracting key content related to the question and generating a summary), and interactive video understanding (for example, answering video-related questions like an intelligent assistant). Through the encoding information of key frames and the encoding information of other image frames, one or more of the target clip in the video, the summary information of the video, and the reply information to the text can be determined. Users can view the target clip in the video, the summary information of the video, or the reply information to the text through electronic devices.

[0058] Optionally, based on the encoding information of the key frame and the encoding information of other image frames, the semantic relevance score between each image frame and the text can be calculated, and the target segment can be determined or summary information or reply information can be generated based on the semantic relevance score.

[0059] For example, based on the semantic relevance score, segments of a plurality of consecutive video frames whose relevance scores are higher than a preset threshold may be selected to obtain a target segment in the video.

[0060] For another example, the target segment can be determined based on the semantic relevance score, and then key information can be extracted from the target segment and semantic synthesis can be performed to generate concise summary information.

[0061] Optionally, LLM may be used to determine a target segment or generate summary information or generate response information based on the encoding information of the key frame and the encoding information of other image frames.

[0062] For example, the encoding information of the key frames and the encoding information of other image frames can be provided as input to the LLM, and the LLM generates a response in natural language form, that is, reply information to the text.

[0063] According to the above embodiment, an intuitive and accurate reply to a text may be made through a target segment in a video, summary information of a video, or reply information to a text, thereby improving the user experience.

[0064] In some embodiments, the above step S110, determining a key frame in the image frame based on the visual features of each of the T image frames of the video, includes:

[0065] Clustering the image frames based on the visual features of each image frame to obtain clustering information;

[0066] Based on the clustering information, M representative frames are determined in the image frame; wherein M is a positive integer less than or equal to T;

[0067] Based on the features of the text, key frames are determined in the representative frames.

[0068] In the disclosed embodiment, clustering can be performed on the T image frames based on the visual features of each image frame, that is, the T image frames of the video are divided into several categories or clusters of image frames. The image frames in the same category or cluster are similar, while the image frames in different categories or clusters are quite different. The clustering information of each image frame can be used to determine the distance between the image frame and other image frames (for example, adjacent frames), and whether the image frame should be the center of a category or cluster.

[0069] Optionally, the clustering information may include a local density index ρ t and / or distance indicator δ t . Local density refers to the neighborhood density of an image frame in the visual feature space, indicating the degree of similarity between it and the neighboring frames. The higher the local density, the more similar the content of the image frame is to the neighboring frames. The distance index is used to measure the distance between an image frame and a high-density image frame, indicating the uniqueness of the image frame. The larger the distance index, the more representative the image frame is in terms of content. For example, the visual features of each image frame can be processed based on the density peak clustering algorithm to obtain the local density index and distance index of each image frame.

[0070] In the disclosed embodiment, the representative frame can be understood as a landmark frame of a video event. Among multiple consecutive frames with similar content, the representative frame can be the first frame, the last frame, or a video frame with some changes (such as the beginning or end of an action). For example, in a video of a person speaking, the representative frame can be a video frame with a significant change in the shape of the mouth.

[0071] Optionally, the local density index ρ of each image frame can be t and the distance index δ t , get the importance score s of each image frame t =ρ t ·δ t , the M image frames with higher importance scores can be determined as M representative frames, that is, the video frames with larger local density and larger distance index are considered to be representative frames.

[0072] Optionally, a representative frame may correspond to an event prototype, which is feature information of the event represented by the representative frame. In the disclosed embodiment, a key frame is determined in the M representative frames based on the features of the text, so that a key event can be selected from the M events represented by the M representative frames in the video based on the features of the text, and the representative frame of the key event is determined as the key frame.

[0073] According to the above embodiment, by using the clustering information of each image frame, M representative frames can be determined from T image frames, thereby preliminarily filtering out redundant video frames. And by using the features of the text to determine the key frames from the M representative frames, the key frames associated with the text in the video can be further accurately determined.

[0074] In some embodiments, determining a key frame in a representative frame based on features of the text includes:

[0075] Based on the event prototype corresponding to each representative frame and the features of the text, the association information corresponding to each representative frame is obtained; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, and i is a positive integer less than or equal to M; the association information is used to characterize the association between the event prototype and the features of the text;

[0076] Based on the associated information corresponding to each representative frame, a key frame is determined in the representative frame.

[0077] In the disclosed embodiment, the event prototype corresponding to each representative frame can be fused with the features of the text to obtain association information. The association information may include the degree of association between the event prototype corresponding to each representative frame and the features of the text, and / or the visual features in the event prototype corresponding to each representative frame that are associated with the features of the text.

[0078] Optionally, for each representative frame (taking the i-th representative frame as an example), the visual features of the i-th representative frame and the visual features of the neighboring frames of the representative frame (for example, the ik-th representative frame to the i+k-th representative frame, k is a positive integer) can be weighted and aggregated to obtain statistical features. The statistical features may include the mean, maximum value, etc. of each visual feature, so as to extract important information of the event and obtain the event prototype corresponding to the representative frame.

[0079] Optionally, when the association information includes the degree of association between the event prototype corresponding to each representative frame and the feature of the text, the event prototype with a higher degree of association may be determined, and the representative frame corresponding to the event prototype may be determined as a key frame.

[0080] Optionally, when the associated information includes visual features associated with the features of the text in the event prototype corresponding to each representative frame, the representative frame that meets preset conditions (such as a larger area) can be determined based on information such as the area, position, or color of the associated visual features in the video frame, and then the key frame can be determined among the M representative frames.

[0081] According to the above embodiment, the selection of key frames combines the characteristics of the text and the information of the event prototype, and can determine the key frames corresponding to the events that have a high correlation with the text, thereby further accurately determining the key frames in the video, and improving the accuracy of subsequent distinguishing the encoding of key frames and other image frames, thereby effectively improving the encoding efficiency and ensuring information integrity.

[0082] In some embodiments, based on the event prototype corresponding to each representative frame and the features of the text, the associated information corresponding to each representative frame is obtained, including:

[0083] The aggregate features of the text features and the statistical features of the event prototype corresponding to each representative frame are spliced ​​respectively to obtain the spliced ​​features corresponding to each representative frame;

[0084] The splicing features are processed using a multi-layer perceptron to obtain the associated information corresponding to each representative frame.

[0085] In the disclosed embodiments, the aggregated features of the features of the text can be understood as combining the features of multiple text words in the text to represent the global semantics of the text. For example, the values ​​on each dimension in a set of feature vectors of the text can be averaged to obtain a new vector with the same dimension, i.e., the aggregated features of the features of the text.

[0086] Optionally, the statistical features of the event prototype may include the mean and maximum values ​​of the visual features contained in the event prototype, and the important information of the event may be extracted through the statistical features of the event prototype. For example, the mean value of the event prototype may be obtained by averaging the values ​​on each dimension in the feature vector corresponding to each visual feature, and the maximum value on each dimension in the event prototype may be obtained by selecting the maximum value.

[0087] In the embodiment of the present disclosure, the splicing features corresponding to each representative frame can be input into a multi-layer perceptron, and the multi-layer perceptron can be guided by the features of the text to output the association information corresponding to each representative frame.

[0088] For example, the associated information s l =U(Max(m l )||Avg(m l )||Agg(Q)), where Agg(Q) is an aggregation operation on the features of the text (such as taking the average) to represent the global semantics of the text, and Max(m l ) is the feature row with the largest norm in the event prototype, Avg(m l ) is the characteristic mean of the event prototype. Max(m l )||Avg(m l )||Agg(Q) is the sum of Agg(Q) and Avg(m l ) and Max(m l ) is concatenated to obtain the concatenated features, U() represents the calculation of the multi-layer perceptron (MLP), and its output represents the associated information s corresponding to the frame l .

[0089] According to the above embodiment, by splicing and inputting the aggregated features of the text features and the statistical features of the event prototype corresponding to the representative frame into a multi-layer perceptron, the associated information corresponding to each representative frame can be accurately obtained, thereby screening out key frames that are more helpful for text problems, and improving the accuracy of subsequent distinguishing encoding of key frames and other image frames, thereby effectively improving encoding efficiency and ensuring information integrity.

[0090] In some embodiments, the above step S120, encoding the key frame based on the event prototype corresponding to the key frame to obtain the encoding information of the key frame, includes:

[0091] Clustering is performed based on the visual features of the key frames to obtain the spatial object prototypes corresponding to the key frames;

[0092] Based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame, the key frame is encoded to obtain encoding information of the key frame.

[0093] Optionally, clustering can be performed in the feature space of the key frame based on the visual features of the key frame. The spatial object prototype corresponding to the key frame may include spatial information in the key frame obtained according to different granularities and different clustering methods, that is, the visual features in the feature space of the key frame will be divided into multiple spatial objects (such as regions or blocks).

[0094] Optionally, the feature space of the key frames can be divided into several categories or clusters based on the visual features of each key frame. The visual features in the same category or cluster are similar, while the visual features in different categories or clusters are quite different. Through each layer of clustering, the feature space can be divided into different feature areas. Each category or cluster represents the semantic information of the feature area in the key frame (such as area shape, texture, etc.).

[0095] Optionally, visual features can be clustered layer by layer according to different granularities (coarse-grained to fine-grained), for example, the first layer is divided into large areas (such as foreground and background), and the second layer is further subdivided into more specific objects (such as people, vehicles).

[0096] Optionally, based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame, different spatial regions of the key frame may be encoded with different resolutions to obtain encoding information of the key frame.

[0097] According to the above embodiment, the key frames can be encoded in combination with the spatial object prototypes corresponding to the key frames, which can more accurately capture the spatial details in the key frames and improve the accuracy of subsequent distinguishing the encoding of the key frames and other image frames, thereby effectively improving the encoding efficiency and ensuring information integrity.

[0098] In some embodiments, based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame, the key frame is encoded to obtain the encoding information of the key frame, including:

[0099] Perform nonlinear combination on the spatial object prototypes corresponding to the key frames and the event prototypes corresponding to the key frames to obtain the splicing features corresponding to the key frames;

[0100] The multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

[0101] In the disclosed embodiment, the event prototype corresponding to the key frame and the spatial object prototype corresponding to the key frame can be nonlinearly combined, and the splicing features can be input into a multi-layer perceptron. The multi-layer perceptron can determine and retain the spatial details in the key frame and fuse the event detail information to obtain the encoding information of the key frame.

[0102] Optionally, the splicing features of the nonlinear combination of the event prototype corresponding to the key frame and the spatial object prototype corresponding to the key frame can be input into the multi-layer perceptron, or the multi-layer perceptron can also perform nonlinear combination of the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame.

[0103] According to the above embodiment, a multi-layer perceptron can be used to process the nonlinear combination of event prototypes and spatial object prototypes, and learn more feature representations of event prototypes and spatial object prototypes, so as to further optimize the spatial details in the key frames and improve the accuracy of key frame encoding, thereby effectively improving the encoding efficiency and ensuring information integrity.

[0104] In some embodiments, the above step S130, encoding other image frames except the key frames in the image frames based on the features of the text associated with the video processing task, to obtain encoding information of other image frames, includes:

[0105] Clustering is performed in the feature space of other image frames based on the visual features of other image frames to obtain the spatial object prototypes corresponding to other image frames;

[0106] Based on the spatial object prototypes corresponding to the other image frames and the features of the text, the other image frames are encoded to obtain the encoding information of the other image frames.

[0107] In the disclosed embodiments, clustering of different granularities can be performed in the feature space of other image frames based on the visual features of other image frames to obtain multi-level spatial object prototypes corresponding to the other image frames. The multi-level spatial object prototypes corresponding to the other image frames may include spatial information in other image frames obtained according to different granularities and different clustering methods. That is, the visual features in the feature space of other image frames will be divided into multiple spatial objects (such as regions or blocks).

[0108] Optionally, different spatial regions of other image frames may be encoded at different resolutions based on the spatial object prototypes corresponding to other image frames and the features of the text to obtain encoding information of other image frames. For example, spatial regions unrelated to the text may be encoded at low resolution and compressed.

[0109] According to the above embodiment, on the basis of encoding other image frames based on text features, spatial object prototypes corresponding to other image frames can also be combined to selectively retain spatial information in other image frames based on the text, thereby effectively improving subsequent encoding efficiency.

[0110] In some embodiments, based on the spatial object prototypes corresponding to the other image frames and the features of the text, the other image frames except the key frame in the T image frames are encoded to obtain the encoding information of the other image frames, including:

[0111] Determine the attention information of the spatial object prototype based on the features of the text;

[0112] Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain the encoding information of the other image frames.

[0113] In the disclosed embodiment, the attention mechanism is used to interact the features of the text with the spatial object prototypes corresponding to other image frames to generate a text-guided visual feature, i.e., the encoding information of other image frames, thereby compressing the spatial information of other image frames.

[0114] Optionally, the attention information may include the weight of each aggregation result in the spatial object prototype. When the weight between the aggregation result and the features of the text is low, that is, the spatial region corresponding to the aggregation result is less associated with the text, the information corresponding to the aggregation result may be compressed.

[0115] According to the above embodiment, the redundant spatial regions in other image frames can be accurately determined by attention information determined by the features of the text, thereby further accurately compressing the spatial regions of other image frames and effectively improving the subsequent encoding efficiency.

[0116] In order to more clearly understand the technical solution of the embodiment of the present disclosure, a specific application example is provided below. Figure 2 A schematic diagram of the video processing method is shown in FIG. Figure 2 As shown, the video processing method may include: a visual and text encoding module 21, a dynamic event prototype estimation module 22, and a compact collaborative encoding module 23.

[0117] The visual and text encoding module 21 can be used to extract visual features of the image frame and features of the text, and can specifically be composed of the following steps:

[0118] (1) The visual encoder extracts features from each frame of the video where v t ∈R N×d represents the features of the tth frame, R represents a set of real numbers, which can be specifically represented as a two-dimensional matrix with a size of N×d, where N is the number of image blocks and d is the feature dimension. The visual encoder can use the pre-trained ViT model (EVA-CLIP). t Through spatial pooling, the dimension is reduced to f t =Pool(v t ), where Pool() represents pooling, f t ∈R P×d , where P is the number of blocks after pooling.

[0119] (2) The text encoder converts text into features where q r ∈R d' , R is a real number, specifically the number of text words, and d' is the text feature dimension.

[0120] The dynamic event prototype estimation module 22 may also be referred to as a DPE (Dynamic Performance Evaluation) module, which is designed to dynamically filter key frames from video frame features and generate event prototypes. The module may specifically be composed of the following steps:

[0121] (1) Redundancy removal and clustering: Based on the Density Peaks Clustering based on K-Nearest Neighbor (DPC-KNN), the density index ρ of each video frame is calculated. t and the distance index δ t ;

[0122] (2) Frame importance score; based on the density index ρ t and the distance index δ t , the importance score s of each image frame t =ρ t ·δ t . Take the M frames with the highest scores as representative frames, and generate the event prototype m corresponding to each representative frame l ;

[0123] (3) Event prototype optimization: The score of the event prototype corresponding to each representative frame is regressed through a multi-layer perceptron and guided by the features of the text. Specifically, the associated information s l=U(Max(m l )||Avg(m l )||Agg(Q)), where Agg(Q) is an aggregation operation on the features of the text (such as taking the average) to represent the global semantics of the text, and Max(m l ) is the feature row with the largest norm in the event prototype, Avg(m l ) is the characteristic mean of the event prototype. Max(m l )||Avg(m l )||Agg(Q) is the sum of Agg(Q) and Avg(m l ) and Max(m l ) is concatenated to obtain the concatenated features, U() represents the calculation of the multi-layer perceptron (MLP), and its output represents the associated information s corresponding to the frame l , the model is able to learn the association between event prototypes and text, thereby screening out key frames that are more helpful in answering text questions.

[0124] The compact collaborative coding module 23 can be based on the optimized event prototype and the frame importance mask b={b t}, and encode the key frames and redundant frames (i.e., other image frames except the key frames) into fine-grained and coarse-grained features respectively. This module can be specifically composed of the following steps:

[0125] (1) Spatial object prototype extraction: cluster the visual features of key frames and other image frames respectively to extract multi-level spatial object prototypes G t ;

[0126] (2) Detail perception coding (cone cells); the frame importance mask b t = 1 is determined as a key frame, and the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame are nonlinearly combined to obtain the splicing features corresponding to the key frame; the splicing features corresponding to the key frame are processed by a multi-layer perceptron to obtain the encoding information U of the key frame A , without pooling operation, spatial details can be preserved.

[0127] (3) Coarse perceptual coding (rod cells); the frame importance mask b t = 0 is determined as a redundant frame, and the attention information of the spatial object prototype is determined based on the characteristics of the text; based on the attention information and the spatial object prototypes corresponding to other image frames, the other image frames are encoded to obtain the encoding information U of the other image frames. BCoarse perceptual coding is guided by the features of the text and encodes the redundant frames at a low resolution. It can extract information related to the text in the redundant frames instead of relying solely on the global average of visual features. For example, if the text is "What is the color in the background?", it is necessary to focus on the part related to the background details.

[0128] (4) Combine detail perception coding and coarse perception coding to form the final frame embedding (i.e., the processing result of the video): t =U A ·U B .

[0129] It can be seen that the video processing method provided by the embodiment of the present disclosure can, first of all, extract key frames based on the visual features of each video frame, perform semantic optimization in combination with the features of the text, and dynamically achieve task adaptation. Secondly, the biologically inspired coding mechanism can be used to collaborate fine-grained and coarse-grained coding to make optimal use of the characteristics of key frames and redundant frames. In this process, the features of the text are introduced into redundant frame compression and key frame optimization, thereby improving the correlation between video semantics and text, and enhancing text-visual fusion. Finally, in terms of temporal dynamic modeling and memory efficiency in long video processing, it far exceeds traditional static coding schemes and has significant advantages in understanding long videos.

[0130] According to an embodiment of the present disclosure, the present disclosure also provides a video processing device, Figure 3 A schematic block diagram of a video processing device provided by an embodiment of the present disclosure is shown. Figure 3 As shown, the device comprises:

[0131] A key frame determination module 310, configured to determine a key frame in an image frame based on visual features of each of T image frames of a video, wherein T is an integer greater than or equal to 2;

[0132] A key frame encoding module 320 is used to encode the key frame based on the event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame;

[0133] The other image frame encoding module 330 is used to encode other image frames except the key frames in the image frames based on the features of the text associated with the video processing task to obtain encoding information of other image frames;

[0134] The result determination module 340 is used to obtain the video processing result based on the encoding information of the key frame and the encoding information of other image frames.

[0135] In some embodiments, the result determination module 340 is specifically configured to:

[0136] Based on the encoding information of the key frame and the encoding information of other image frames, one or more of a target segment in the video, summary information of the video, and reply information for the text are determined.

[0137] In some embodiments, Figure 4 As shown, the key frame determination module 310 includes:

[0138] A first clustering submodule 410, configured to cluster the image frames based on visual features of each image frame to obtain clustering information of each image frame;

[0139] The representative frame determination submodule 420 is used to determine M representative frames in the image frame based on the clustering information of each image frame; wherein M is a positive integer less than or equal to T;

[0140] The key frame determination submodule 430 is used to determine the key frames in the representative frames based on the features of the text.

[0141] In some embodiments, Figure 5 As shown, the key frame determination submodule 430 includes:

[0142] The association determination unit 510 is used to obtain association information corresponding to each representative frame based on the event prototype corresponding to each representative frame and the features of the text; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, and i is a positive integer less than or equal to M; the association information is used to characterize the association between the event prototype and the features of the text;

[0143] The key frame determining unit 520 is configured to determine a key frame in the representative frame based on the associated information corresponding to each representative frame.

[0144] In some embodiments, the association determination unit 510 is specifically configured to:

[0145] The aggregate features of the text features and the statistical features of the event prototype corresponding to each representative frame are spliced ​​respectively to obtain the spliced ​​features corresponding to each representative frame;

[0146] A multi-layer perceptron is used to process the splicing features corresponding to each representative frame to obtain the associated information corresponding to each representative frame.

[0147] In some embodiments, Figure 6 As shown, the key frame encoding module 320 includes:

[0148] A second clustering submodule 610 is used to perform clustering based on the visual features of the key frames to obtain the spatial object prototypes corresponding to the key frames;

[0149] The key frame encoding submodule 620 is used to encode the key frame based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain encoding information of the key frame.

[0150] In some embodiments, the key frame encoding submodule 620 is specifically configured to:

[0151] Perform nonlinear combination on the spatial object prototypes corresponding to the key frames and the event prototypes corresponding to the key frames to obtain the splicing features corresponding to the key frames;

[0152] The multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

[0153] In some embodiments, Figure 7 As shown, other image frame encoding modules 330 include:

[0154] A third clustering submodule 710 is used to perform clustering in a feature space of other image frames based on visual features of other image frames to obtain spatial object prototypes corresponding to other image frames;

[0155] The other image frame encoding submodule 720 is used to encode other image frames based on the spatial object prototypes corresponding to the other image frames and the features of the text to obtain encoding information of the other image frames.

[0156] In some embodiments, the other image frame encoding submodule 720 is specifically used to:

[0157] Determine the attention information of the spatial object prototype based on the features of the text;

[0158] Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain the encoding information of the other image frames.

[0159] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0160] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0161] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0162] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0163] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 807 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0164] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0165] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as video processing methods. For example, in some embodiments, the video processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the video processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the video processing method in any other appropriate manner (eg, by means of firmware).

[0166] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

[0171] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0172] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0173] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A video processing method, comprising: Determining a key frame in T image frames of the video based on visual features of each image frame; wherein T is an integer greater than or equal to 2; Based on the event prototype corresponding to the key frame, the key frame is encoded to obtain the encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame; Based on the features of the text associated with the video processing task, encoding other image frames except the key frame in the image frames to obtain encoding information of the other image frames; Based on the encoding information of the key frame and the encoding information of the other image frames, a processing result of the video is obtained.

2. The method according to claim 1, wherein: The obtaining the processing result of the video based on the encoding information of the key frame and the encoding information of the other image frames includes: Based on the encoding information of the key frame and the encoding information of the other image frames, one or more of a target segment in the video, summary information of the video, and reply information for the text are determined.

3. The method according to claim 1 or 2, wherein: Determining a key frame in the image frames based on visual features of each of the T image frames of the video includes: Clustering the image frames based on the visual features of each image frame to obtain clustering information; Based on the clustering information, determining M representative frames in the image frame; wherein M is a positive integer less than or equal to T; The key frame is determined in the representative frame based on the feature of the text.

4. The method according to claim 3, wherein: The step of determining the key frame in the representative frame based on the feature of the text includes: Based on the event prototype corresponding to each of the representative frames and the features of the text, the association information corresponding to each of the representative frames is obtained; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, and i is a positive integer less than or equal to M; the association information is used to characterize the association between the event prototype and the features of the text; The key frame is determined in the representative frame based on the associated information corresponding to each of the representative frames.

5. The method according to claim 4, wherein: The obtaining of the associated information corresponding to each representative frame based on the event prototype corresponding to each representative frame and the feature of the text includes: The aggregated features of the features of the text and the statistical features of the event prototype corresponding to each of the representative frames are spliced ​​respectively to obtain the spliced ​​features corresponding to each of the representative frames; The splicing features are processed using a multi-layer perceptron to obtain associated information corresponding to each of the representative frames.

6. The method according to any one of claims 1 to 5, wherein: The step of encoding the key frame based on the event prototype corresponding to the key frame to obtain the encoding information of the key frame includes: Clustering is performed based on visual features of the key frames to obtain spatial object prototypes corresponding to the key frames; Based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame, the key frame is encoded to obtain encoding information of the key frame.

7. The method according to claim 6, wherein: The step of encoding the key frame based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain the encoding information of the key frame includes: Performing nonlinear combination on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain the splicing feature corresponding to the key frame; A multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

8. The method according to any one of claims 1 to 7, wherein: The encoding of other image frames except the key frame in the image frames based on the features of the text associated with the video processing task to obtain encoding information of the other image frames includes: Performing clustering based on visual features of the other image frames to obtain spatial object prototypes corresponding to the other image frames; Based on the spatial object prototypes corresponding to the other image frames and the features of the text, the other image frames are encoded to obtain encoding information of the other image frames.

9. The method according to claim 8, wherein: The encoding of the other image frames except the key frame in the image frames based on the spatial object prototypes corresponding to the other image frames and the features of the text to obtain the encoding information of the other image frames includes: determining attention information based on features of the text; Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain the encoding information of the other image frames.

10. A video processing device, comprising: A key frame determination module, configured to determine a key frame in T image frames of a video based on visual features of each image frame; wherein T is an integer greater than or equal to 2; A key frame encoding module, used to encode the key frame based on the event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame; Another image frame encoding module is used to encode other image frames in the image frames except the key frame based on the characteristics of the text associated with the video processing task to obtain encoding information of the other image frames; The result determination module is used to obtain the processing result of the video based on the encoding information of the key frame and the encoding information of the other image frames.

11. The device according to claim 10, wherein: The result determination module is further used for: Based on the encoding information of the key frame and the encoding information of the other image frames, one or more of a target segment in the video, summary information of the video, and reply information for the text are determined.

12. The device according to claim 10 or 11, wherein: The key frame determination module comprises: A first clustering submodule, configured to cluster the image frames based on visual features of each image frame to obtain clustering information of each image frame; A representative frame determination submodule, used to determine M representative frames in the image frame based on the clustering information; wherein M is a positive integer less than or equal to T; The key frame determination submodule is used to determine the key frame in the representative frame based on the features of the text.

13. The device according to claim 12, wherein: The key frame determination submodule includes: An association determination unit is used to obtain association information corresponding to each representative frame based on the event prototype corresponding to each representative frame and the features of the text; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, and i is a positive integer less than or equal to M; the association information is used to characterize the association between the event prototype and the features of the text; The key frame determining unit is used to determine the key frame in the representative frame based on the associated information corresponding to each of the representative frames.

14. The device according to claim 13, wherein: The association determination unit is further used for: The aggregated features of the features of the text and the statistical features of the event prototype corresponding to each of the representative frames are spliced ​​respectively to obtain the spliced ​​features corresponding to each of the representative frames; The splicing features are processed using a multi-layer perceptron to obtain associated information corresponding to each of the representative frames.

15. The device according to any one of claims 10 to 14, wherein: The key frame encoding module comprises: A second clustering submodule, used for clustering based on the visual features of the key frame to obtain the spatial object prototype corresponding to the key frame; The key frame encoding submodule is used to encode the key frame based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain the encoding information of the key frame.

16. The device according to claim 15, wherein: The key frame encoding submodule is further used for: Performing nonlinear combination on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain the splicing feature corresponding to the key frame; A multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

17. The device according to any one of claims 10 to 16, wherein: The other image frame encoding module comprises: A third clustering submodule, configured to perform clustering based on visual features of the other image frames to obtain spatial object prototypes corresponding to the other image frames; The other image frame encoding submodule is used to encode the other image frames based on the spatial object prototypes corresponding to the other image frames and the characteristics of the text to obtain the encoding information of the other image frames.

18. The device according to claim 17, wherein: The other image frame encoding submodule is further used for: determining attention information based on features of the text; Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain the encoding information of the other image frames.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images