Video processing method, device, electronic device and storage medium

Abstract

The present disclosure provides a video processing method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence, particularly to the field of large models, deep learning, and image processing. A specific implementation scheme is as follows: based on the visual features of each of T image frames in a video, a key frame is determined in the image frame; based on the event prototype corresponding to the key frame, the key frame is encoded to obtain encoding information of the key frame; based on the features of the text associated with the video processing task, the image frames other than the key frame are encoded to obtain encoding information of the other image frames; based on the encoding information of the key frame and the encoding information of the other image frames, the video processing result is obtained.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular to the fields of large models, deep learning, and image processing. Background Art

[0002] With the success of Large Language Models (LLMs) in the field of artificial intelligence, their application in vision-language fusion has been greatly expanded. In some application scenarios, a full understanding of the video content is required, and encoding video frames is an important part of understanding the video. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, electronic device, and storage medium for video processing.

[0004] According to one aspect of the present disclosure, a video processing method is provided, comprising:

[0005] Determining a key frame in the image frame based on visual features of each of T image frames in the video, wherein T is an integer greater than or equal to 2;

[0006] Based on the event prototype corresponding to the key frame, the key frame is encoded to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame;

[0007] Based on the features of the text associated with the video processing task, encoding the image frames other than the key frames in the image frames to obtain encoding information of the other image frames;

[0008] Based on the encoding information of the key frames and the encoding information of other image frames, a video processing result is obtained.

[0009] According to another aspect of the present disclosure, there is provided a video processing apparatus, comprising:

[0010] a key frame determination module, configured to determine a key frame in an image frame based on visual features of each of T image frames of a video; wherein T is an integer greater than or equal to 2;

[0011] A key frame encoding module is used to encode the key frame based on the event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame;

[0012] Other image frame encoding modules are used to encode other image frames except key frames in the image frames based on the features of the text associated with the video processing task to obtain encoding information of other image frames;

[0013] The result determination module is used to obtain the video processing result based on the coding information of the key frames and the coding information of other image frames.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the computer program implements any one of the methods according to the embodiments of the present disclosure.

[0020] The technical solution of the disclosed embodiment can retain the event details of the keyframe as much as possible by encoding the keyframe using the event prototype corresponding to the keyframe. Moreover, by encoding other image frames using the characteristics of the text associated with the video processing task, it can determine and retain the information related to the text in the other image frames and compress the content unrelated to the text. Therefore, the technical solution of the disclosed embodiment can make optimal use of the characteristics of the keyframe and other image frames, while retaining the key event information of the video while reducing memory usage, thereby achieving good encoding efficiency and information integrity.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a flowchart of a video processing method provided by an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of an application example of the video processing method provided by an embodiment of the present disclosure;

[0025] Figure 3 is a schematic block diagram of a video processing device provided by an embodiment of the present disclosure;

[0026] Figure 4 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0027] Figure 5 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0028] Figure 6 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0029] Figure 7 is a schematic block diagram of a video processing device provided by another embodiment of the present disclosure;

[0030] Figure 8 is a block diagram of an electronic device for implementing the video processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] The video processing method provided by the embodiments of the present disclosure can be applied to scenarios requiring understanding of video semantics, such as video retrieval, video question answering, and video summarization. Specifically, it can be applied to the process of determining the relevance between video frames and user-entered text.

[0033] Figure 1 The video processing method provided by an embodiment of the present disclosure is shown. The method can be applied to a video processing device, which can be deployed in an electronic device. The electronic device is, for example, a single or multi-machine terminal, server or other processing device. Among them, the terminal can be a user equipment (UE) such as a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc.; the server can be a single server or a server cluster. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in a memory. For example Figure 1 As shown, the method may include the following steps S110 to S140.

[0034] Step S110 : determining a key frame in the image frames based on visual features of each of T image frames in the video; wherein T is an integer greater than or equal to 2.

[0035] In the disclosed embodiments, the video can be any video in the video library, such as a video selected by the user. In some application scenarios, the user can specify a video and enter text, and the video processing device can encode the video based on the text to compress the video, extract content associated with the text from the video, or generate summary information or reply information associated with the text.

[0036] In the disclosed embodiment, T may be the number of image frames in a video, and visual features of each of the T image frames may be extracted. The visual features of an image frame may be understood as reflecting the visual content of the image frame. For example, the visual features of an image frame may be features extracted from the image frame using a convolutional neural network. For another example, the visual features of an image frame may include brightness, contrast, color distribution, texture features, etc. of the image frame.

[0037] Optionally, a visual encoder can be used to extract features from each frame of the video to obtain the corresponding visual feature set of the video. where v t ∈R N×d Represents the features of the t-th frame, R represents a two-dimensional matrix, its size is N×d, N is the number of image blocks, d is the feature dimension, that is, v t is a two-dimensional matrix of size N×d. For example, the visual encoder can use a pre-trained ViT model (EVA-CLIP). Optionally, the visual features of each image frame can be pooled to reduce the dimension of the visual features. For example, v t Reduce the dimension to f by spatial pooling t =Pool(v t ), where Pool() represents pooling, f t ∈R P×d , where P is the number of blocks after pooling.

[0038] Optionally, the key frame may include one or more video frames representing the core content of the video among the T image frames. In other words, the key frame may be a video frame excluding redundant frames such as repeated frames, header frames, and tail frames.

[0039] Optionally, the video processing device can determine the key frames in T image frames through clustering. The purpose of clustering can be to divide the T image frames into multiple classes / clusters. Through clustering, the clustering index of each image frame can be obtained, and the clustering index indicates whether a single image frame is the center of a class / cluster, and the distance between the image frame and other image frames. Using the clustering index, the representative frame of each class / cluster can be determined in the T image frames, and the key frames can be determined based on this. Exemplarily, the visual features of each image frame can be processed based on the density peaks clustering algorithm (Density Peaks Clustering based on K-NearestNeighbor, DPC-KNN) to obtain the local density index and distance index of each image frame. Based on the local density index and distance index of each image frame, the importance score of each image frame is obtained. Based on the importance score of each image frame, it can be determined whether the image frame is a key frame.

[0040] Optionally, the video processing apparatus may generate a frame importance mask b={b t}, when the image frame is a key frame, mark the mask value b corresponding to the image frame t =1, otherwise the mask value corresponding to the image frame is marked as b t =0, thereby dividing the image frames into key frames and other image frames, wherein the other image frames can also be called redundant frames.

[0041] Optionally, the key frame may also include one or more video frames associated with the text in the T image frames. For example, if the text is "What is the color in the background?", the key frame may include a video frame with background details.

[0042] Step S120 : encoding the key frame based on the event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame.

[0043] In the disclosed embodiment, a key frame may correspond to an event prototype. The event prototype may be understood as an abstract representation describing an important event associated with the key frame. The event prototype may include aggregated features extracted from the visual features of the key frame and the visual features of the neighboring frames of the key frame. Therefore, compared with the visual features of the key frame, the event prototype corresponding to the key frame integrates the visual features of the neighboring frames, retains the temporal dynamic features of the video, and generates a feature with more global representation power. Optionally, the event prototype can be obtained by de-redundancy and aggregation, so as to generate a more compact feature, which can reduce the consumption of computing resources for subsequent processing while highlighting the core information of the event.

[0044] Optionally, the visual features of the key frame and the visual features of the neighboring frames of the key frame can be weighted and aggregated to obtain an event prototype. The event prototype may include statistical features of the key frame and its redundant frames, for example, the mean and maximum values ​​of each visual feature, so as to extract important information of the event. The statistical features of the event prototype can be used to encode the key frame to obtain the encoding information of the key frame.

[0045] In the disclosed embodiment, based on the event prototype corresponding to the key frame, the key frame can be finely encoded (for example, high-resolution encoding), without pooling operation during the encoding process, retaining spatial details, and obtaining the encoding information of the key frame.

[0046] Optionally, the encoding information of the key frame may include the spatial elements in the video picture and the presentation method of the spatial elements, such as the composition of the video picture, the layout of objects, light and shadow effects, color matching, etc.

[0047] Optionally, the number of key frames may be one or more, and each key frame is encoded based on an event prototype corresponding to the key frame to obtain encoding information.

[0048] Step S130 : Based on the features of the text associated with the video processing task, encode the image frames other than the key frames in the image frames to obtain encoding information of the other image frames.

[0049] For example, the text associated with the video processing task may include query information for a user to initiate a query / retrieval, a question raised by the user regarding the video, or a statement by the user describing the video processing task. The features of the text may represent information about the content or structure of the text, for example, semantic features, structural features, etc.

[0050] Optionally, the text can be converted into features through a text encoder where q r ∈R d' , R is a real number, specifically the number of text words, and d' is the text feature dimension.

[0051] In the disclosed embodiment, image frames other than key frames are redundant frames in the video. The features of the text can be used to guide the video processing device to perform coarse-grained encoding (such as low-resolution encoding) on ​​other image frames. For example, content related to the text can be determined and retained in other image frames, and content unrelated to the text can be compressed, rather than relying solely on the global average of visual features. This can reduce memory pressure while retaining the temporal dynamics of the video and obtaining encoding information for other image frames.

[0052] Step S140: Obtain a video processing result based on the encoding information of the key frame and the encoding information of other image frames.

[0053] In the disclosed embodiment, the encoding information of key frames can be combined with the encoding information of other image frames. The encoding information of key frames can retain event details, and the encoding information of other image frames can retain information related to text in other image frames, thereby obtaining the encoding result of the entire video. The encoding result of the video can be decoded to obtain the processing result of the video. The processing result of the video can be text or video, and has a high correlation with the input text. Optionally, the processing result of the video can be displayed to the user, thereby completing tasks such as video retrieval, video question and answer, and video summarization.

[0054] In the related art, the understanding of videos still faces significant challenges, especially in long videos where there is a large amount of redundant information between video frames, such as repeated frames and frames irrelevant to the problem, resulting in excessive memory usage. In addition, different problems have different focuses on video frames, and this semantic relevance is difficult to be fully reflected in existing static coding methods. Although average pooling or attention mechanisms can reduce the encoding cost of video frames, they often lose key visual details or temporal dynamics. The above method provided by the embodiment of the present disclosure can retain the event details of the key frames as much as possible by encoding the key frames using the event prototypes corresponding to the key frames, and by encoding other image frames using the features of the text associated with the video processing task, it can determine and retain information related to the text in other image frames, and compress content that is not related to the text. Therefore, the technical solution of the embodiment of the present disclosure can make optimal use of the features of key frames and other image frames, while reducing memory usage and retaining the key event information of the video, thereby achieving good coding efficiency and information integrity.

[0055] In some embodiments, the above step S140, based on the encoding information of the key frame and the encoding information of other image frames, obtains the video processing result, including:

[0056] Based on the encoding information of the key frame and the encoding information of other image frames, one or more items of a target segment in the video, summary information of the video, and reply information for the text are determined.

[0057] In the disclosed embodiments, it can be applied to video question-answering or other scenarios that require understanding of video semantics. For example: video retrieval (for example, quickly finding relevant clips based on text questions), video summarization (for example, extracting key content related to the question and generating a summary), and interactive video understanding (for example, answering video-related questions like an intelligent assistant). Through the encoding information of key frames and the encoding information of other image frames, one or more of the target clip in the video, the summary information of the video, and the reply information for the text can be determined. Users can view the target clip in the video, the summary information of the video, or the reply information for the text through electronic devices.

[0058] Optionally, based on the encoding information of the key frame and the encoding information of other image frames, the semantic relevance score between each image frame and the text can be calculated, and the target segment can be determined or summary information or response information can be generated based on the semantic relevance score.

[0059] For example, based on the semantic relevance score, a segment of a plurality of consecutive video frames whose relevance scores are higher than a preset threshold may be selected to obtain a target segment in the video.

[0060] For another example, the target segment can be determined based on the semantic relevance score, and then key information can be extracted from the target segment and semantic synthesis can be performed to generate concise summary information.

[0061] Optionally, the LLM may be used to determine a target segment or generate summary information or response information based on the encoding information of the key frame and the encoding information of other image frames.

[0062] For example, the encoding information of the key frames and the encoding information of other image frames can be provided as input to the LLM, and the LLM generates a response in natural language form, that is, reply information to the text.

[0063] According to the above embodiment, an intuitive and accurate reply to a text can be made through a target segment in a video, summary information of a video, or reply information to a text, thereby improving the user experience.

[0064] In some embodiments, the above step S110 of determining a key frame in the image frames based on the visual features of each of the T image frames of the video includes:

[0065] Clustering the image frames based on the visual features of each image frame to obtain clustering information;

[0066] Based on the clustering information, M representative frames are determined in the image frame; where M is a positive integer less than or equal to T;

[0067] Based on the features of the text, key frames are determined in the representative frames.

[0068] In an embodiment of the present disclosure, clustering can be performed among T image frames based on the visual features of each image frame, that is, the T image frames of the video are divided into several categories or clusters of image frames. The image frames in the same category or cluster are similar, while the image frames in different categories or clusters are quite different. The clustering information of each image frame can be used to determine the distance between the image frame and other image frames (for example, adjacent frames), and whether the image frame should be the center of a category or cluster.

[0069] Optionally, the clustering information may include a local density index ρ t and / or distance indicator δ t . Local density refers to the neighborhood density of an image frame in the visual feature space, indicating the degree of similarity between it and the neighboring frames. The higher the local density, the more similar the content of the image frame is to the neighboring frames. The distance index is used to measure the distance between an image frame and a high-density image frame, indicating the uniqueness of the image frame. The larger the distance index, the more representative the image frame is in terms of content. For example, the visual features of each image frame can be processed based on the density peak clustering algorithm to obtain the local density index and distance index of each image frame.

[0070] In the disclosed embodiments, a representative frame can be understood as a landmark frame of a video event. Among multiple consecutive frames with similar content, a representative frame can be the first frame, the last frame, or a frame that shows some change (such as the beginning or end of an action). For example, in a video of a person speaking, a representative frame can be a frame where the mouth shape changes significantly.

[0071] Alternatively, the local density index ρ of each image frame can be used t and distance index δ t , get the importance score s of each image frame t =ρ t ·δ t , the M image frames with higher importance scores can be determined as M representative frames, that is, the video frames with larger local density and larger distance index are considered to be representative frames.

[0072] Optionally, a representative frame may correspond to an event prototype, which is characteristic information of the event represented by the representative frame. In the disclosed embodiment, a key frame is determined from the M representative frames based on text features, thereby enabling the selection of a key event from the M events represented by the M representative frames in the video based on text features, and determining the representative frame of the key event as the key frame.

[0073] According to the above embodiment, using the clustering information of each image frame, M representative frames can be determined from T image frames, thereby initially filtering out redundant video frames. Furthermore, using the characteristics of the text to determine key frames from the M representative frames, the key frames associated with the text in the video can be further accurately determined.

[0074] In some embodiments, determining a key frame in a representative frame based on features of the text includes:

[0075] Based on the event prototype corresponding to each representative frame and the features of the text, the association information corresponding to each representative frame is obtained; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, where i is a positive integer less than or equal to M; the association information is used to represent the association between the event prototype and the features of the text;

[0076] Based on the associated information corresponding to each representative frame, a key frame is determined in the representative frame.

[0077] In the disclosed embodiments, the event prototype corresponding to each representative frame may be fused with the features of the text to obtain association information. The association information may include the degree of association between the event prototype corresponding to each representative frame and the features of the text, and / or the visual features of the event prototype corresponding to each representative frame that are associated with the features of the text.

[0078] Optionally, for each representative frame (taking the i-th representative frame as an example), the visual features of the i-th representative frame and the visual features of the neighboring frames of the representative frame (for example, the ik-th representative frame to the i+k-th representative frame, where k is a positive integer) can be weighted and aggregated to obtain statistical features. The statistical features may include the mean, maximum value, etc. of each visual feature, so that important information of the event can be extracted and the event prototype corresponding to the representative frame can be obtained.

[0079] Optionally, when the association information includes the degree of association between the event prototype corresponding to each representative frame and the feature of the text, the event prototype with a higher degree of association may be determined, and the representative frame corresponding to the event prototype may be determined as a key frame.

[0080] Optionally, when the associated information includes visual features associated with the features of the text in the event prototype corresponding to each representative frame, the representative frame that meets the preset conditions (for example, a larger area) can be determined based on information such as the area, position, or color of the associated visual features in the video frame, and then the key frame can be determined among the M representative frames.

[0081] According to the above embodiment, the selection of key frames combines the characteristics of the text and the information of the event prototype, and can determine the key frames corresponding to events that have a high correlation with the text, thereby further accurately determining the key frames in the video, and improving the accuracy of subsequent distinguishing encoding of key frames and other image frames, thereby effectively improving encoding efficiency and ensuring information integrity.

[0082] In some embodiments, based on the event prototype and text features corresponding to each representative frame, the associated information corresponding to each representative frame is obtained, including:

[0083] The aggregated features of the text features and the statistical features of the event prototype corresponding to each representative frame are spliced ​​separately to obtain the spliced ​​features corresponding to each representative frame;

[0084] The splicing features are processed using a multi-layer perceptron to obtain the associated information corresponding to each representative frame.

[0085] In the disclosed embodiments, the aggregated features of text features can be understood as combining the features of multiple words in a text to represent the global semantics of the text. For example, the values ​​in each dimension of a set of feature vectors for a text can be averaged to obtain a new vector with the same dimensions, i.e., the aggregated features of the text features.

[0086] Optionally, the statistical features of the event prototype may include the mean and maximum values ​​of the visual features contained in the event prototype. Important information about the event can be extracted from the statistical features of the event prototype. For example, the mean value of the event prototype can be obtained by averaging the values ​​in each dimension of the feature vector corresponding to each visual feature, while the maximum value in each dimension of the event prototype can be obtained by selecting the maximum value in each dimension of the event prototype.

[0087] In the embodiment of the present disclosure, the splicing features corresponding to each representative frame can be input into a multi-layer perceptron, and the multi-layer perceptron can be guided by the features of the text to output the associated information corresponding to each representative frame.

[0088] For example, the associated information s l =U(Max(m l )||Avg(m l )||Agg(Q)), where Agg(Q) is an aggregation operation on the features of the text (such as taking the mean), which is used to represent the global semantics of the text, and Max(m l ) is the feature row with the largest norm in the event prototype, Avg(m l ) is the characteristic mean of the event prototype. Max(m l )||Avg(m l )||Agg(Q) is the sum of Agg(Q) and Avg(m l ) and Max(m l ) is spliced ​​together to obtain the spliced ​​features, U() represents the calculation of the multi-layer perceptron (MLP), and its output represents the associated information s corresponding to the frame l .

[0089] According to the above embodiment, by splicing the aggregated features of the text features and the statistical features of the event prototype corresponding to the representative frame into a multi-layer perceptron, the associated information corresponding to each representative frame can be accurately obtained, thereby screening out key frames that are more helpful for text problems, and improving the accuracy of subsequent distinguishing encoding of key frames and other image frames, thereby effectively improving encoding efficiency and ensuring information integrity.

[0090] In some embodiments, the above step S120, encoding the key frame based on the event prototype corresponding to the key frame to obtain the encoding information of the key frame, includes:

[0091] Clustering is performed based on the visual features of key frames to obtain the spatial object prototypes corresponding to the key frames;

[0092] The key frames are encoded based on the spatial object prototypes and the event prototypes corresponding to the key frames to obtain encoding information of the key frames.

[0093] Optionally, clustering can be performed in the feature space of the keyframes based on the visual features of the keyframes. The spatial object prototypes corresponding to the keyframes can include spatial information in the keyframes obtained according to different granularities and different clustering methods. In other words, the visual features in the feature space of the keyframes are divided into multiple spatial objects (such as regions or blocks).

[0094] Optionally, the feature space of the key frames can be divided into several categories or clusters based on the visual features of each key frame. The visual features in the same category or cluster are similar, while the visual features in different categories or clusters are quite different. Through clustering at each layer, the feature space can be divided into different feature areas. Each category or cluster represents the semantic information of the feature area in the key frame (such as area shape, texture, etc.).

[0095] Optionally, visual features can be clustered layer by layer at different granularities (coarse to fine), for example, the first layer is divided into large areas (such as foreground and background), and the second layer is further subdivided into more specific objects (such as people, vehicles).

[0096] Optionally, different spatial regions of the key frame may be encoded with different resolutions based on the spatial object prototype and the event prototype corresponding to the key frame to obtain encoding information of the key frame.

[0097] According to the above embodiment, the key frame can be encoded in combination with the spatial object prototype corresponding to the key frame, which can more accurately capture the spatial details in the key frame and improve the accuracy of subsequent distinguishing encoding of the key frame and other image frames, thereby effectively improving the encoding efficiency and ensuring information integrity.

[0098] In some embodiments, the key frame is encoded based on the spatial object prototype and the event prototype corresponding to the key frame to obtain encoding information of the key frame, including:

[0099] Perform nonlinear combination on the spatial object prototypes and event prototypes corresponding to the key frames to obtain the splicing features corresponding to the key frames;

[0100] The multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

[0101] In the disclosed embodiment, the event prototype corresponding to the key frame and the spatial object prototype corresponding to the key frame can be nonlinearly combined, and the splicing features can be input into a multi-layer perceptron. The multi-layer perceptron can determine and retain the spatial details in the key frame and fuse the event detail information to obtain the encoding information of the key frame.

[0102] Optionally, the splicing features of the nonlinear combination of the event prototype corresponding to the key frame and the spatial object prototype corresponding to the key frame can be input into the multi-layer perceptron, or the multi-layer perceptron can also perform nonlinear combination on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame.

[0103] According to the above embodiment, a multi-layer perceptron can be used to process the nonlinear combination of event prototypes and spatial object prototypes, and learn more feature representations of event prototypes and spatial object prototypes, so as to further optimize the spatial details in the key frames and improve the accuracy of key frame encoding, thereby effectively improving encoding efficiency and ensuring information integrity.

[0104] In some embodiments, the above step S130, encoding the image frames other than the key frames based on the features of the text associated with the video processing task to obtain encoding information of the other image frames, includes:

[0105] Clustering is performed in the feature space of other image frames based on the visual features of other image frames to obtain spatial object prototypes corresponding to other image frames;

[0106] Based on the spatial object prototypes corresponding to the other image frames and the features of the text, the other image frames are encoded to obtain encoding information of the other image frames.

[0107] In the embodiment of the present disclosure, clustering of different granularities can be performed in the feature space of other image frames based on the visual features of other image frames to obtain multi-level spatial object prototypes corresponding to other image frames. The multi-level spatial object prototypes corresponding to other image frames may include spatial information in other image frames obtained according to different granularities and different clustering methods. That is, the visual features in the feature space of other image frames will be divided into multiple spatial objects (such as regions or blocks).

[0108] Optionally, different spatial regions of the other image frames may be encoded at different resolutions based on the spatial object prototypes corresponding to the other image frames and the characteristics of the text to obtain encoding information for the other image frames. For example, spatial regions unrelated to the text may be encoded at a low resolution and compressed.

[0109] According to the above embodiment, on the basis of encoding other image frames based on text features, the spatial object prototypes corresponding to other image frames can also be combined to selectively retain the spatial information in other image frames based on the text, thereby effectively improving the subsequent encoding efficiency.

[0110] In some embodiments, based on the spatial object prototypes and text features corresponding to the other image frames, the other image frames except the key frame are encoded to obtain encoding information of the other image frames, including:

[0111] Determine the attention information of spatial object prototypes based on text features;

[0112] Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain encoding information of the other image frames.

[0113] In the embodiment of the present disclosure, the features of the text and the spatial object prototypes corresponding to other image frames are interacted with each other through the attention mechanism to generate a text-guided visual feature, that is, the encoding information of other image frames, thereby compressing the spatial information of other image frames.

[0114] Optionally, the attention information may include the weight of each aggregation result in the spatial object prototype. When the weight between the aggregation result and the features of the text is low, that is, the spatial area corresponding to the aggregation result is less associated with the text, the information corresponding to the aggregation result can be compressed.

[0115] According to the above embodiment, the attention information determined by the text features can be used to accurately determine the redundant spatial regions in other image frames, thereby further accurately compressing the spatial regions of other image frames and effectively improving subsequent encoding efficiency.

[0116] In order to more clearly understand the technical solution of the embodiment of the present disclosure, a specific application example is provided below. Figure 2 Schematic diagram of the video processing method is shown in FIG. Figure 2 As shown, the video processing method may include: a visual and text encoding module 21 , a dynamic event prototype estimation module 22 , and a compact collaborative encoding module 23 .

[0117] The visual and text encoding module 21 can be used to extract visual features and text features of the image frame, and can specifically be composed of the following steps:

[0118] (1) The visual encoder extracts features from each frame of the video where v t ∈R N×d Represents the features of the tth frame, R represents a set of real numbers, which can be specifically represented as a two-dimensional matrix with a size of N×d, where N is the number of image blocks and d is the feature dimension. The visual encoder can use the pre-trained ViT model (EVA-CLIP). t Reduce the dimension to f by spatial pooling t =Pool(v t ), where Pool() represents pooling, f t ∈R P×d , where P is the number of blocks after pooling.

[0119] (2) Text encoder converts text into features where q r ∈R d' , R is a real number, specifically the number of text words, and d' is the text feature dimension.

[0120] The dynamic event prototype estimation module 22 may also be referred to as a DPE (Dynamic Performance Evaluation) module. The DPE module is designed to dynamically filter key frames from video frame features and generate event prototypes. The module may specifically comprise the following steps:

[0121] (1) Redundancy removal and clustering: Based on the Density Peaks Clustering based on K-Nearest Neighbor (DPC-KNN), the density index ρ of each video frame is calculated. t and distance index δ t ;

[0122] (2) Frame importance score; according to the density index ρ t and distance index δ t , the importance score s of each image frame t =ρ t ·δ t . Take the M frames with the highest scores as representative frames, and generate the event prototype m corresponding to each representative frame l ;

[0123] (3) Event prototype optimization: regress the score of the event prototype corresponding to each representative frame through a multi-layer perceptron and guide it through the characteristics of the text. Specifically, the associated information s l=U(Max(m l )||Avg(m l )||Agg(Q)), where Agg(Q) is an aggregation operation on the features of the text (such as taking the mean), which is used to represent the global semantics of the text, and Max(m l ) is the feature row with the largest norm in the event prototype, Avg(m l ) is the characteristic mean of the event prototype. Max(m l )||Avg(m l )||Agg(Q) is the sum of Agg(Q) and Avg(m l ) and Max(m l ) is spliced ​​together to obtain the spliced ​​features, U() represents the calculation of the multi-layer perceptron (MLP), and its output represents the associated information s corresponding to the frame l , the model is able to learn the association between event prototypes and text, thereby screening out key frames that are more helpful in answering text questions.

[0124] The compact collaborative coding module 23 can be based on the optimized event prototype and the frame importance mask b={b t} is encoded, and the key frames and redundant frames (i.e., other image frames except the key frames) are encoded into fine-grained and coarse-grained features respectively. This module can be specifically composed of the following steps:

[0125] (1) Spatial object prototype extraction: cluster the visual features of key frames and other image frames respectively to extract multi-level spatial object prototypes G t ;

[0126] (2) Detail perception coding (cone cells); the frame importance mask b t = 1 is determined as a key frame, and the spatial object prototype and the event prototype corresponding to the key frame are nonlinearly combined to obtain the splicing features corresponding to the key frame; the splicing features corresponding to the key frame are processed by a multi-layer perceptron to obtain the encoding information U of the key frame. A , without pooling operation, spatial details can be preserved.

[0127] (3) Coarse perceptual coding (rod cells); the frame importance mask b t = 0 is determined as a redundant frame, and the attention information of the spatial object prototype is determined based on the characteristics of the text; based on the attention information and the spatial object prototypes corresponding to other image frames, the other image frames are encoded to obtain the encoding information U of the other image frames. BCoarse perceptual coding uses the features of the text to encode redundant frames at low resolution. This allows for the extraction of text-related information from redundant frames, rather than relying solely on a global average of visual features. For example, if the text is "What is the color in the background?", then the focus should be on the details of the background.

[0128] (4) Combine detail perception coding and coarse perception coding to form the final frame embedding (i.e., the processing result of the video): t =U A ·U B .

[0129] It can be seen that the video processing method provided by the embodiment of the present disclosure can, first of all, extract key frames based on the visual features of each video frame, perform semantic optimization in combination with the features of the text, and dynamically achieve task adaptation. Secondly, the bio-inspired coding mechanism can be used to collaborate fine-grained and coarse-grained coding to make optimal use of the characteristics of key frames and redundant frames. In this process, the features of the text are introduced into redundant frame compression and key frame optimization, thereby improving the relevance of video semantics and text, and enhancing text-visual fusion. Finally, in terms of temporal dynamic modeling and memory efficiency in long video processing, it far exceeds traditional static coding schemes and has significant advantages in understanding long videos.

[0130] According to an embodiment of the present disclosure, the present disclosure further provides a video processing device, Figure 3 FIG. 1 shows a schematic block diagram of a video processing device provided by an embodiment of the present disclosure, as shown in FIG. Figure 3 As shown, the device includes:

[0131] A key frame determination module 310 is configured to determine a key frame in an image frame based on visual features of each of T image frames in a video, wherein T is an integer greater than or equal to 2;

[0132] A key frame encoding module 320 is configured to encode the key frame based on an event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is obtained based on the visual features of the key frame and the visual features of the neighboring frames of the key frame;

[0133] The other image frame encoding module 330 is used to encode the other image frames except the key frames in the image frames based on the features of the text associated with the video processing task to obtain encoding information of the other image frames;

[0134] The result determination module 340 is configured to obtain a video processing result based on the encoding information of the key frames and the encoding information of other image frames.

[0135] In some embodiments, the result determination module 340 is specifically configured to:

[0136] Based on the encoding information of the key frame and the encoding information of other image frames, one or more items of a target segment in the video, summary information of the video, and reply information for the text are determined.

[0137] In some embodiments, as Figure 4 As shown, the key frame determination module 310 includes:

[0138] A first clustering submodule 410 is configured to cluster the image frames based on visual features of each image frame to obtain clustering information of each image frame;

[0139] The representative frame determination submodule 420 is configured to determine M representative frames in the image frame based on the clustering information of each image frame; wherein M is a positive integer less than or equal to T;

[0140] The key frame determination submodule 430 is configured to determine key frames in the representative frames based on text features.

[0141] In some embodiments, as Figure 5 As shown, the key frame determination submodule 430 includes:

[0142] The association determination unit 510 is configured to obtain association information corresponding to each representative frame based on the event prototype corresponding to each representative frame and the features of the text; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, where i is a positive integer less than or equal to M; and the association information is used to represent the association between the event prototype and the features of the text;

[0143] The key frame determining unit 520 is configured to determine a key frame in the representative frame based on the associated information corresponding to each representative frame.

[0144] In some embodiments, the association determination unit 510 is specifically configured to:

[0145] The aggregated features of the text features and the statistical features of the event prototype corresponding to each representative frame are spliced ​​separately to obtain the spliced ​​features corresponding to each representative frame;

[0146] The multi-layer perceptron is used to process the splicing features corresponding to each representative frame to obtain the associated information corresponding to each representative frame.

[0147] In some embodiments, as Figure 6 As shown, the key frame encoding module 320 includes:

[0148] A second clustering submodule 610 is configured to perform clustering based on the visual features of the key frames to obtain spatial object prototypes corresponding to the key frames;

[0149] The key frame encoding submodule 620 is configured to encode the key frame based on the spatial object prototype and the event prototype corresponding to the key frame to obtain encoding information of the key frame.

[0150] In some embodiments, the key frame encoding submodule 620 is specifically configured to:

[0151] Perform nonlinear combination on the spatial object prototypes and event prototypes corresponding to the key frames to obtain the splicing features corresponding to the key frames;

[0152] The multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

[0153] In some embodiments, as Figure 7 As shown, the other image frame encoding module 330 includes:

[0154] A third clustering submodule 710 is configured to perform clustering in a feature space of other image frames based on visual features of other image frames to obtain spatial object prototypes corresponding to other image frames;

[0155] The other image frame encoding submodule 720 is used to encode other image frames based on the spatial object prototypes and text features corresponding to the other image frames to obtain encoding information of the other image frames.

[0156] In some embodiments, the other image frame encoding submodule 720 is specifically configured to:

[0157] Determine the attention information of spatial object prototypes based on text features;

[0158] Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain encoding information of the other image frames.

[0159] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0160] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0161] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0162] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0163] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 807 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0164] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0165] The computing unit 801 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the video processing method. For example, in some embodiments, the video processing method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the video processing method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the video processing method in any other appropriate manner (eg, by means of firmware).

[0166] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0167] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0168] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0169] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0170] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0171] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0172] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0173] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A video processing method, comprising: Determining a key frame in T image frames of the video based on visual features of each image frame, wherein T is an integer greater than or equal to 2; Encoding the key frame based on an event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is an abstract representation describing an important event associated with the key frame, the event prototype is obtained by weighted aggregation based on visual features of the key frame and visual features of neighboring frames of the key frame, the event prototype includes statistical features of the key frame and its redundant frames, and the key frame is encoded using the statistical features; Based on features of text associated with the video processing task, encoding other image frames in the image frames except the key frame to obtain encoding information of the other image frames; wherein the text associated with the video processing task includes query information for a user to initiate a query or retrieval, or a question raised by the user regarding the video, or a statement by the user describing the video processing task; the features of the text are used to guide coarse-grained encoding of the other image frames, determining and retaining content related to the text in the other image frames, and compressing content unrelated to the text; A processing result of the video is obtained based on the encoding information of the key frame and the encoding information of the other image frames.

2. The method according to claim 1, wherein The obtaining of the video processing result based on the encoding information of the key frame and the encoding information of the other image frames includes: Based on the encoding information of the key frame and the encoding information of the other image frames, one or more items of a target segment in the video, summary information of the video, and reply information for the text are determined.

3. The method according to claim 1 or 2, wherein: Determining a key frame in the image frames based on visual features of each of the T image frames in the video includes: Clustering the image frames based on the visual features of each image frame to obtain clustering information; Based on the clustering information, determining M representative frames in the image frame; wherein M is a positive integer less than or equal to T; The key frame is determined in the representative frame based on the feature of the text.

4. The method according to claim 3, wherein: The determining the key frame in the representative frame based on the feature of the text includes: Based on the event prototype corresponding to each representative frame and the features of the text, association information corresponding to each representative frame is obtained; wherein the event prototype corresponding to the i-th representative frame is obtained based on the visual features of the i-th representative frame and the visual features of the neighboring frames of the i-th representative frame, where i is a positive integer less than or equal to M; the association information is used to characterize the association between the event prototype and the features of the text; The key frame is determined in the representative frame based on the associated information corresponding to each representative frame.

5. The method according to claim 4, wherein The obtaining of association information corresponding to each representative frame based on the event prototype corresponding to each representative frame and the features of the text includes: The aggregated features of the text features and the statistical features of the event prototype corresponding to each representative frame are respectively spliced ​​to obtain spliced ​​features corresponding to each representative frame; The splicing features are processed using a multi-layer perceptron to obtain associated information corresponding to each representative frame.

6. The method according to claim 1 or 2, wherein: The step of encoding the key frame based on the event prototype corresponding to the key frame to obtain encoding information of the key frame includes: Clustering is performed based on the visual features of the key frames to obtain spatial object prototypes corresponding to the key frames; The key frame is encoded based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain encoding information of the key frame.

7. The method according to claim 6, wherein: The step of encoding the key frame based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain encoding information of the key frame includes: Performing nonlinear combination on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain the splicing feature corresponding to the key frame; A multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

8. The method according to claim 1 or 2, wherein: The encoding of the image frames other than the key frames in the image frames based on the features of the text associated with the video processing task to obtain encoding information of the other image frames includes: performing clustering based on visual features of the other image frames to obtain spatial object prototypes corresponding to the other image frames; Based on the spatial object prototypes corresponding to the other image frames and the features of the text, the other image frames are encoded to obtain encoding information of the other image frames.

9. The method according to claim 8, wherein The encoding of the image frames other than the key frame based on the spatial object prototypes corresponding to the other image frames and the features of the text to obtain encoding information of the other image frames includes: determining attention information based on features of the text; Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain encoding information of the other image frames.

10. A video processing device, comprising: A key frame determination module, configured to determine a key frame in T image frames of a video based on visual features of each image frame; wherein T is an integer greater than or equal to 2; A key frame encoding module is configured to encode a key frame based on an event prototype corresponding to the key frame to obtain encoding information of the key frame; wherein the event prototype is an abstract representation describing an important event associated with the key frame, and is obtained by weighted aggregation based on visual features of the key frame and visual features of neighboring frames of the key frame. The event prototype includes statistical features of the key frame and its redundant frames, and the key frame is encoded using the statistical features; an other image frame encoding module, configured to encode other image frames in the image frames, excluding the key frames, based on features of text associated with the video processing task, to obtain encoding information of the other image frames; wherein the text associated with the video processing task includes query information for a user to initiate a query or retrieval, or a question raised by the user regarding the video, or a statement by the user describing the video processing task; the features of the text are used to guide coarse-grained encoding of the other image frames, determining and retaining content related to the text in the other image frames, and compressing content unrelated to the text; The result determination module is used to obtain the processing result of the video based on the encoding information of the key frame and the encoding information of the other image frames.

11. The device according to claim 10, wherein The result determination module is further configured to: Based on the encoding information of the key frame and the encoding information of the other image frames, one or more items of a target segment in the video, summary information of the video, and reply information for the text are determined.

12. The device according to claim 10 or 11, wherein The key frame determination module includes: a first clustering submodule, configured to cluster the image frames based on visual features of each image frame to obtain clustering information of each image frame; a representative frame determination submodule, configured to determine M representative frames in the image frame based on the clustering information; wherein M is a positive integer less than or equal to T; The key frame determination submodule is used to determine the key frame in the representative frame based on the features of the text.

13. The device according to claim 12, wherein The key frame determination submodule includes: an association determination unit, configured to obtain association information corresponding to each representative frame based on an event prototype corresponding to each representative frame and features of the text; wherein the event prototype corresponding to the i-th representative frame is obtained based on visual features of the i-th representative frame and visual features of neighboring frames of the i-th representative frame, where i is a positive integer less than or equal to M; and the association information is used to characterize the association between the event prototype and the features of the text; The key frame determining unit is configured to determine the key frame in the representative frame based on the associated information corresponding to each representative frame.

14. The device according to claim 13, wherein The association determination unit is further configured to: The aggregated features of the text features and the statistical features of the event prototype corresponding to each representative frame are respectively spliced ​​to obtain spliced ​​features corresponding to each representative frame; The splicing features are processed using a multi-layer perceptron to obtain associated information corresponding to each representative frame.

15. The device according to claim 10 or 11, wherein The key frame encoding module includes: A second clustering submodule is used to perform clustering based on the visual features of the key frames to obtain spatial object prototypes corresponding to the key frames; The key frame encoding submodule is used to encode the key frame based on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain encoding information of the key frame.

16. The device according to claim 15, wherein The key frame encoding submodule is further used to: Performing nonlinear combination on the spatial object prototype corresponding to the key frame and the event prototype corresponding to the key frame to obtain the splicing feature corresponding to the key frame; A multi-layer perceptron is used to process the splicing features corresponding to the key frames to obtain the encoding information of the key frames.

17. The device according to claim 10 or 11, wherein The other image frame encoding modules include: a third clustering submodule, configured to perform clustering based on visual features of the other image frames to obtain spatial object prototypes corresponding to the other image frames; The other image frame encoding submodule is used to encode the other image frames based on the spatial object prototypes corresponding to the other image frames and the characteristics of the text to obtain encoding information of the other image frames.

18. The device according to claim 17, wherein The other image frame encoding submodules are further used for: determining attention information based on features of the text; Based on the attention information and the spatial object prototypes corresponding to the other image frames, the other image frames are encoded to obtain encoding information of the other image frames.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images