Video processing method and device based on spatiotemporal token merging and electronic equipment

Abstract

The application provides a video processing method and device based on spatiotemporal token merging, electronic equipment and storage medium, and belongs to the technical field of video processing, and comprises the following steps: acquiring a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first transformer block to obtain part of the first group of tokens, part of the second group of tokens and a group of time sequence merging tokens output by the first transformer block; splicing the part of the first group of tokens, the part of the second group of tokens and the group of time sequence merging tokens to obtain a third group of tokens; and inputting the third group of tokens into a second transformer block to obtain a fourth group of tokens output by the second transformer block. The video processing method based on spatiotemporal token merging provided by the application solves the technical problems of the related art video processing method based on spatiotemporal token merging, such as easy loss of video information and limited processing performance.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video processing method, device and electronic equipment based on spatiotemporal token merging. Background Art

[0002] The goal of video text retrieval is to quickly retrieve related videos or text from a database given a text or video query. Video processing requires inputting multiple frames of images, which increases the computational effort required for inference and hinders the deployment of video text retrieval models in real-world large-scale scenarios. Due to the redundant information between adjacent frames of a video, feeding multiple frames into the visual encoder generates many similar redundant tokens. By reducing the number of these redundant tokens, the computational effort required for video inference can be effectively reduced.

[0003] In related technologies, when processing video data, the number of redundant tokens is reduced by discarding meaningless frames and then filtering out detail-rich areas from the remaining frames. However, this strategy lacks contextual modeling of continuous time information, resulting in limited performance. There are also related schemes that reduce the number of redundant tokens by discarding information that appears earlier in the sequence and only retaining tokens with semantic importance. However, its frame-by-frame calculation method is inefficient.

[0004] It can be seen that the video processing method based on spatiotemporal token merging in the related art has technical problems such as easy loss of video information and limited processing performance. Summary of the Invention

[0005] The present invention provides a video processing method, device and electronic device based on spatiotemporal token merging, which are used to solve the technical problems of video processing methods based on spatiotemporal token merging in related technologies, such as easy loss of video information and limited processing performance.

[0006] The present invention provides a video processing method based on spatiotemporal token merging, comprising the following steps: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block, obtaining a portion of the first group of tokens, a portion of the second group of tokens and a group of temporal merging tokens output by the first converter block, wherein the group of temporal merging tokens is obtained by merging each temporal token pair in a group of temporal token pairs, the group of temporal token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, and the portion of the first group of tokens is composed of the first group of tokens except the group of temporal token pairs. The part of the second group of tokens is composed of tokens in the second group of tokens except the group of temporal token pairs; the part of the first group of tokens, the part of the second group of tokens and the group of temporal merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and the part of the third group of tokens, the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0007] According to a video processing method based on spatiotemporal token merging provided by the present invention, the first converter block includes a first multi-head self-attention module and a first feedforward neural network module; the first group of tokens and the second group of tokens are input into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens output by the first converter block, including: inputting the first group of tokens and the second group of tokens into the first multi-head self-attention module to obtain a group of first token value feature averages output by the first multi-head self-attention module, wherein the group of first token value feature averages includes each token in the first group of tokens and the second group of tokens in all attention The method comprises the following steps: calculating the average value feature of the value in the force head; calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on the average value of the value feature of the group of first tokens, and determining a group of similarity values; selecting m similarity values ​​with the largest values ​​from the group of similarity values ​​in order from large to small, and determining the group of time series token pairs corresponding to the m similarity values, wherein m is a constant; merging each time series token pair in the group of time series token pairs to obtain the group of time series merged tokens; and performing an update operation on the token features of the part of the first group of tokens, the part of the second group of tokens and the group of time series merged tokens.

[0008] According to a video processing method based on spatiotemporal token merging provided by the present invention, the similarity between each token in the first group of tokens and each token in the second group of tokens is calculated based on the average value of the first token value features of the group, and a group of similarity values ​​is determined, including: calculating the cosine similarity between the average value of the first token value features of each token in the first group of tokens and the average value of the first token value features of each token in the second group of tokens, to obtain the group of similarity values.

[0009] According to a video processing method based on spatiotemporal token merging provided by the present invention, the second converter block includes a second multi-head self-attention module and a second feedforward neural network module; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, including: inputting the third group of tokens into the second multi-head self-attention module to obtain a group of second token value feature averages output by the second multi-head self-attention module, wherein the group of second token value feature averages includes the average value of the value feature of each token in the third group of tokens in all attention heads; randomly dividing the third group of tokens into a fifth group of tokens and a sixth group of tokens. group tokens; calculating the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens based on the average value of the feature of the group of second token values, and determining a group of target similarity values; selecting n target similarity values ​​with the largest values ​​from the group of target similarity values ​​in descending order, and determining the group of spatial token pairs corresponding to the n target similarity values, wherein n is a constant; merging each spatial token pair in the group of spatial token pairs to obtain the group of spatial merged tokens; the second feedforward neural network module performs an update operation on the token features of the group of spatial merged tokens and the part of the third group of tokens.

[0010] According to a video processing method based on spatiotemporal token merging provided by the present invention, the second multi-head self-attention module updates the features of the input token based on the attention mask.

[0011] According to a video processing method based on spatiotemporal token merging provided by the present invention, before obtaining the first group of tokens and the second group of tokens, the method also includes: obtaining a video to be processed; converting the video to be processed into a group of video frames to be processed; performing a preprocessing operation on the group of video frames to be processed to obtain the group of video frames, wherein the number of video frames in the group of video frames is an exponential multiple of 2.

[0012] The present invention also provides a video processing device based on spatiotemporal token merging, comprising the following modules: an acquisition module, for acquiring a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; a first execution module, for inputting the first group of tokens and the second group of tokens into a first converter block, and obtaining part of the first group of tokens, part of the second group of tokens and a group of temporal merging tokens output by the first converter block, wherein the group of temporal merging tokens is obtained by merging each temporal token pair in a group of temporal token pairs, the group of temporal token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, and the part of the first group of tokens is obtained by excluding the group of temporal token pairs in the first group of tokens. , the partial second group of tokens is composed of tokens in the second group of tokens excluding the group of temporal token pairs; a splicing module is used to splice the partial first group of tokens, the partial second group of tokens and the group of temporal merged tokens to obtain a third group of tokens; a second execution module is used to input the third group of tokens into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and partial third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and the partial third group of tokens, the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, it implements any of the above-described video processing methods based on spatiotemporal token merging.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the video processing method based on spatiotemporal token merging as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described video processing methods based on spatiotemporal token merging.

[0016] The video processing method, device, electronic device and storage medium based on spatiotemporal token merging provided by the present invention input the first group of tokens and the second group of tokens into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of temporally merged tokens output by the first converter block, that is, the token pairs with higher similarity in the first group of tokens and the second group of tokens are merged to obtain a group of temporally merged tokens, which reduces the redundant tokens between two adjacent video frames in the time dimension (that is, in two different video frames) and avoids the possibility of loss of feature information of the tokens; the third group of tokens is input into the second converter block to obtain the fourth group of tokens output by the second converter block, that is, the tokens with higher similarity in the third group of tokens are merged in pairs, further reducing the number of tokens in the spatial dimension (that is, in the same video frame), and improving the computational efficiency of the model while maintaining the accuracy of the model from being greatly reduced, thereby solving the technical problems of the video processing method based on spatiotemporal token merging in the related art, that is, the video information is easily lost and the processing performance is limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a flow chart of the video processing method based on spatiotemporal token merging provided by the present invention.

[0019] Figure 2 This is one of the schematic diagrams of the video processing method based on spatiotemporal token merging provided by the present invention.

[0020] Figure 3 This is the second schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention.

[0021] Figure 4 This is the third schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention.

[0022] Figure 5 It is a structural block diagram of a video processing device based on spatiotemporal token merging provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0025] It should be noted that, in the description of the present invention, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element specified by the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element. Terms such as "upper" and "lower" indicate positions or relationships based on those shown in the accompanying drawings and are intended solely to facilitate description and simplify the present invention. They are not intended to indicate or imply that the device or element referred to must have a specific orientation, be constructed, or operate in a specific orientation, and are therefore not to be construed as limiting the present invention. Unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be broadly construed, for example, to mean a fixed connection, a removable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediary; or internal communication between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] The terms "first," "second," and so forth, used herein are used to distinguish similar objects, not to describe a specific order or precedence. It should be understood that such terms are interchangeable where appropriate, allowing embodiments of the present invention to be implemented in an order other than that illustrated or described herein. Furthermore, the terms "first," "second," and so forth generally distinguish objects of a single type, and do not limit the number of objects. For example, the first object may be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates an "or" relationship between the connected objects.

[0027] The goal of video text retrieval is to quickly retrieve related videos or text from a database given a text or video query. Video processing requires inputting multiple frames of images, which increases the computational effort required for inference and hinders the deployment of video text retrieval models in real-world large-scale scenarios. Due to the redundant information between adjacent frames of a video, feeding multiple frames into the visual encoder generates many similar redundant tokens. By reducing the number of these redundant tokens, the computational effort required for video inference can be effectively reduced.

[0028] The Transformer Model is a deep neural network model based on the self-attention mechanism. The Transformer Model can process a flexible number of input tokens, which facilitates the reduction of redundant tokens during the calculation process. In related technologies, when processing video data, the number of redundant tokens is reduced by discarding meaningless frames and then filtering out detailed areas from the remaining frames. However, this strategy lacks contextual modeling of continuous time information, resulting in limited performance. There are also related schemes that reduce the number of redundant tokens by discarding information that appears earlier in the sequence and only retaining tokens with semantic importance. However, its frame-by-frame calculation method is inefficient. It can be seen that the video processing method based on spatiotemporal token merging in related technologies has technical problems such as easy loss of video information and limited processing performance.

[0029] In order to at least solve some of the above problems, the following Figures 1-6 The present invention describes the video processing method, device, electronic device and storage medium based on spatiotemporal token merging. The video processing method based on spatiotemporal token merging provided by the present invention can be applied to scenarios where videos are processed to reduce redundant tokens in the videos. The video processing method based on spatiotemporal token merging provided by the present invention can be executed by a server, which runs a transformer model.

[0030] Figure 1 This is one of the flow charts of the video processing method based on spatiotemporal token merging provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0031] Step 101: Acquire a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames.

[0032] In this embodiment, the video to be processed consists of a group of video frames. A video frame may contain multiple visual elements, such as people, objects, scenes, etc. In order to convert the content of the video frame into a processable text form, feature extraction can be performed on the video frame, and then these features can be mapped to a group of tokens. This group of tokens can include words, phrases, sentences, etc., which together constitute a description of the content of the video frame; therefore, each video frame in a group of video frames represents a group of tokens, and the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames.

[0033] Step 102: Input the first group of tokens and the second group of tokens into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens output by the first converter block, wherein a group of time-series merged tokens is obtained by merging each time-series token pair in a group of time-series token pairs, a group of time-series token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except for a group of time-series token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for a group of time-series token pairs.

[0034] The server that executes the video processing method based on spatiotemporal token merging of the present invention runs a transformer model, and the server executes the video processing method based on spatiotemporal token merging based on the transformer model; the transformer model is composed of multiple transformer blocks (Transformer blocks), and the transformer blocks are used to update the features of the tokens to enhance the representation ability of a group of tokens.

[0035] In this embodiment, the first group of tokens and the second group of tokens are input into the first converter block, and the first converter block outputs part of the first group of tokens, part of the second group of tokens and a group of time-merged tokens; it should be noted that the first converter block will update the characteristics of the input tokens, so compared with the first group of tokens and the second group of tokens, the characteristics of the tokens of part of the first group of tokens, part of the second group of tokens and a group of time-merged tokens are updated.

[0036] A group of time-series token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, and each time-series token pair in a group of time-series token pairs is composed of one token in the first group of tokens and one token in the second group of tokens; optionally, the similarity between each token in the first group of tokens and each token in the second group of tokens is calculated, and token pairs with higher similarity are selected to form a group of time-series token pairs.

[0037] The set of time sequence merged tokens is obtained by merging each time sequence token pair in the set of time sequence token pairs, i.e., merging each time sequence token pair in the set of time sequence token pairs twice to obtain the set of time sequence merged tokens; the partial first group of tokens is composed of tokens in the first group of tokens except the set of time sequence token pairs, and the partial second group of tokens is composed of tokens in the second group of tokens except the set of time sequence token pairs.

[0038] In step 103, the partial first group of tokens, the partial second group of tokens, and the set of time sequence merged tokens are spliced to obtain a third group of tokens.

[0039] The partial first group of tokens, the partial second group of tokens, and the set of time sequence merged tokens are spliced to obtain the third group of tokens, i.e., the first group of tokens and the second group of tokens are merged to obtain the third group of tokens, and here the reduction of the number of video frames is realized.

[0040] It should be noted that in the embodiment, the merging of the time sequence tokens refers to the merging between different groups of tokens, i.e., the merging between different video frames.

[0041] It can be understood that such a splicing operation simplifies the output format of the time sequence token merging (i.e., from three output features to one), and avoids the token fragmentation problem that may be caused by multiple applications of the time sequence token merging

[0042] In step 104, the third group of tokens is input into the second transformer block to obtain a fourth group of tokens output by the second transformer block, wherein the third group of tokens includes a set of spatial token pairs and a partial third group of tokens, and the fourth group of tokens includes a set of spatial merged tokens and a partial third group of tokens, the set of spatial merged tokens is obtained by merging each spatial token pair in the set of spatial token pairs, and the set of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0043] The set of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens, i.e., a plurality of tokens with the highest similarity between each other in the third group of tokens can be selected as the set of spatial token pairs, and the third group of tokens includes the set of spatial token pairs and the partial third group of tokens; each spatial token pair in the set of spatial token pairs is merged to obtain a set of spatial merged tokens, and the fourth group of tokens includes the set of spatial merged tokens and the partial third group of tokens.

[0044] It should be noted that the second transformer block updates the input token features, and the token features of the fourth group of tokens are updated compared with the third group of tokens; the merging of the spatial tokens refers to the merging of the tokens in the same group of tokens.

[0045] In this embodiment, the first group of tokens and the second group of tokens are input into the first converter block, and the output results of the first converter block are spliced ​​and input into the second converter block to obtain the fourth group of tokens, thereby achieving a reduction in time and space of redundant tokens of the first group of tokens and the second group of tokens; optionally, the method of this embodiment can be cyclically executed on all adjacent video frames in a group of video frames. For example, when the number of video frames in a group of video frames (each video frame corresponds to a group of tokens) is x, the method of this embodiment is executed on every two adjacent video frames in a group of video frames to obtain x / 2 groups of tokens, and the method of this embodiment is executed again on the x / 2 groups of tokens to obtain x / 4 groups of tokens, and so on, to finally obtain a group of tokens.

[0046] Through the embodiment provided by the present application, a first group of tokens and a second group of tokens are obtained, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; the first group of tokens and the second group of tokens are input into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of timing merge tokens output by the first converter block, wherein a group of timing merge tokens is obtained by merging each timing token pair in a group of timing token pairs, a group of timing token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except for a group of timing token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for a group of timing token pairs. The invention relates to a method for processing video signals of a video stream, wherein the video signals are transmitted to a plurality of spatial tokens, and the plurality of spatial tokens are transmitted to the video processing unit. The method comprises the following steps: first, a plurality of spatial tokens are transmitted to the video processing unit, and a plurality of spatial tokens are transmitted to the video processing unit. The plurality of spatial tokens are transmitted to the video processing unit, and a plurality of spatial tokens are transmitted to the video processing unit. The plurality of spatial tokens are transmitted to the video processing unit, and the plurality of spatial tokens are transmitted to the video processing unit. The plurality of spatial tokens are transmitted to the video processing unit, and the plurality of spatial tokens are transmitted to the video processing unit. The plurality of spatial tokens are transmitted to the video processing unit, and the plurality of spatial tokens are transmitted to the video processing unit.

[0047] As an optional solution, the first transformer block includes a first multi-head self-attention module and a first feedforward neural network module;

[0048] Inputting the first group of tokens and the second group of tokens into the first converter block, obtaining a portion of the first group of tokens, a portion of the second group of tokens, and a group of time-series merged tokens output by the first converter block, including:

[0049] S11, inputting the first group of tokens and the second group of tokens into the first multi-head self-attention module, obtaining a set of first token value feature averages output by the first multi-head self-attention module, wherein the set of first token value feature averages includes the average value feature of each token in the first group of tokens and the second group of tokens in all attention heads;

[0050] S12, calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on an average value of a set of first token value features, and determining a set of similarity values;

[0051] S13, selecting m similarity values ​​with the largest values ​​from a set of similarity values ​​in descending order, and determining a set of time sequence token pairs corresponding to the m similarity values, where m is a constant;

[0052] S14, merging each time sequence token pair in a set of time sequence token pairs to obtain a set of time sequence merge tokens;

[0053] S15, the first feedforward neural network module performs an update operation on the token features of part of the first group of tokens, part of the second group of tokens, and a group of time-series merged tokens.

[0054] Figure 2 This is one of the schematic diagrams of the video processing method based on spatiotemporal token merging provided by the present invention, such as Figure 2 As shown, Figure 2 The (a) Transformer block in corresponds to the first transformer block in this embodiment.

[0055] The first set of tokens (corresponding to Figure 2 The first group of tokens corresponds to the video frame at time t1) and the second group of tokens corresponds to the video frame at time t1. Figure 2 A group of tokens corresponding to t in the multi-head self-attention layer, which means the video frame corresponding to time t2, is input into the first multi-head self-attention module to obtain a group of first token value feature averages output by the first multi-head self-attention module, wherein a group of first token value feature averages includes the average value features of each token in the first group of tokens and the second group of tokens in all attention heads; that is, the value features of the tokens in the multi-head self-attention layer are used to calculate the average value features of each token in the first group of tokens and the second group of tokens in all attention heads to obtain a group of first token value feature averages.

[0056] Optionally, calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on an average value of a set of first token value features to determine a set of similarity values ​​includes:

[0057] The cosine similarity between the average value of the first token value feature of each token in the first group of tokens and the average value of the first token value feature of each token in the second group of tokens is calculated to obtain a set of similarity values.

[0058] Specifically, for two tokens, the cosine similarity between the average values ​​of the first token value features of the two tokens can be calculated. The closer the obtained cosine similarity is to 1, the more similar the two tokens are. Therefore, based on the cosine similarity between the average value of the first token value features of each token in the first group of tokens and the average value of the first token value features of each token in the second group of tokens, the obtained set of similarity values ​​can reflect the similarity between each token in the first group of tokens and each token in the second group of tokens.

[0059] After determining the token similarity calculation method, it is necessary to quickly determine the matching relationship based on the similarity between the first group of tokens and the second group of tokens. In order to prevent the matching process from introducing too much calculation and reducing the model calculation efficiency, a fast approximation method can be used here, that is, for each token in the second group of tokens, the token in the first group of tokens with the greatest similarity is taken as the matching pair.

[0060] Select m, which has the largest value, from a set of similarity values ​​in descending order (m in this embodiment corresponds to Figure 2 In the algorithm, we select k similarity values ​​and determine a set of time-series token pairs corresponding to m similarity values, where m is a constant. After completing token matching, if the token pair similarity is high, it means that the two tokens contain a high degree of information duplication, and the information loss after token merging is small. Therefore, for all token pairs, we select the m token pairs with the greatest similarity and merge them to obtain a set of time-series merged tokens.

[0061] Furthermore, the first feedforward neural network module performs an update operation on token features of a portion of the first group of tokens, a portion of the second group of tokens, and a group of temporally merged tokens.

[0062] Optionally, after obtaining the partial first group of tokens, partial second group of tokens and a group of time-series merged tokens output by the first feedforward neural network module, the partial first group of tokens, partial second group of tokens and a group of time-series merged tokens are input into the third transformer block (corresponding to Figure 2 (b) Transformer block in

[15] .

[0063] Specifically, after passing through the first transformer block equipped with a temporal token merging module, the features are converted into three parts: the token features of each of the two adjacent frames (part of the first group of tokens, part of the second group of tokens) and the token features shared by the two adjacent frames (a group of temporal merged tokens). These features are fed into the subsequent Transformer block (the third transformer block). When performing multi-head self-attention calculations, the shared token features need to be included in the update process of the token features of the adjacent frames to prevent large information loss within the video frame. See Figure 2 In part (d), the self-attention of each feature is calculated as follows:

[0064] Partial first group of tokens: The features of the partial first group of tokens are used as query features in the self-attention calculation, and the features of the partial first group of tokens and a group of temporally merged tokens are used as key and value features. That is, the partial first group of tokens needs to pay attention to the features of the tokens contained in itself and the features of a group of temporally merged tokens to complete the update. It can be understood that when the features of a group of temporally merged tokens are exactly the same as the features of the tokens merged into the first group of tokens, the output features of the partial first group of tokens calculated in this way are exactly the same as the output features of the first group of tokens.

[0065] Partial second group of tokens: The features of the partial second group of tokens are used as query features in the self-attention calculation, and the features of the partial second group of tokens and a group of temporally merged tokens are used as key and value features. That is, the partial second group of tokens needs to pay attention to the features of the tokens contained in themselves and the features of a group of temporally merged tokens to complete the update. It can be understood that when the features of a group of temporally merged tokens are exactly the same as the features of the tokens merged into the second group of tokens, the output features of the partial second group of tokens calculated in this way are exactly the same as the output features of the second group of tokens.

[0066] Features of a group of time-merged tokens: The features of a group of time-merged tokens are used as query features in the self-attention calculation, and the features of part of the first group of tokens, part of the second group of tokens, and a group of time-merged tokens are used as key and value features. That is, the features of a group of time-merged tokens need to pay attention to the token features contained in themselves and the token features contained in part of the first group of tokens and part of the second group of tokens to complete the update. It can be understood that the features of a group of time-merged tokens can aggregate the information of two adjacent frames and play a certain role in time series modeling.

[0067] Through this embodiment, the number of redundant tokens in the first group of tokens and the second group of tokens can be effectively reduced, while information loss is alleviated and processing efficiency is improved.

[0068] As an optional solution, the second transformer block includes a second multi-head self-attention module and a second feedforward neural network module;

[0069] Inputting the third set of tokens into the second transformer block, obtaining a fourth set of tokens output by the second transformer block, including:

[0070] S21, inputting the third group of tokens into the second multi-head self-attention module to obtain a set of second token value feature averages output by the second multi-head self-attention module, wherein the set of second token value feature averages includes the average value feature of each token in the third group of tokens in all attention heads;

[0071] S22, randomly dividing the third group of tokens into a fifth group of tokens and a sixth group of tokens;

[0072] S23, calculating the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens based on an average value of a set of second token value features, and determining a set of target similarity values;

[0073] S24, selecting n target similarity values ​​with the largest values ​​from the set of target similarity values ​​in descending order, and determining a set of spatial token pairs corresponding to the n target similarity values, where n is a constant;

[0074] S25, merging each spatial token pair in the set of spatial token pairs to obtain a set of spatial merged tokens;

[0075] S26, the second feedforward neural network module performs an update operation on the token features of a group of spatially merged tokens and a portion of the third group of tokens.

[0076] Figure 3 This is a second schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention, such as Figure 3 As shown, the steps of this embodiment correspond to Figure 3 The spatial token merging step in .

[0077] Figure 3 The Transformer block including the spatial token merging step in stage 1 corresponds to the second transformer block in this embodiment, which includes a second multi-head self-attention module and a second feedforward neural network module.

[0078] The third group of tokens is input into the second multi-head self-attention module to obtain a set of second token value feature averages output by the second multi-head self-attention module, wherein the set of second token value feature averages includes the average value feature of each token in the third group of tokens in all attention heads; the third group of tokens corresponds to Figure 3 t in i The corresponding set of tokens.

[0079] The third group of tokens is randomly divided into the fifth group of tokens and the sixth group of tokens, corresponding to Figure 3The third group of tokens is randomly divided into two groups A and B; the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens is calculated based on a set of second token value feature averages, and a set of target similarity values ​​is determined. Here, the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens can be determined based on the cosine similarity between the second token value feature average of each token in the fifth group of tokens and the second token value feature average of each token in the sixth group of tokens.

[0080] Select the n with the largest value from a set of target similarity values ​​in descending order (n in this embodiment corresponds to Figure 3 and determining a set of spatial token pairs corresponding to n target similarity values, where n is a constant; merging each spatial token pair in the set of spatial token pairs to obtain a set of spatial merged tokens; and performing an update operation on the token features of the set of spatial merged tokens and part of the third set of tokens by a second feedforward neural network module.

[0081] Through this embodiment, the number of redundant tokens in the third group of tokens can be effectively reduced, while information loss is alleviated and processing efficiency is improved.

[0082] As an optional solution, the second multi-head self-attention module updates the features of the input token based on the attention mask.

[0083] Figure 4 This is a third schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention, as shown in FIG. Figure 4 As shown, in this embodiment, the attention mask method is introduced into the second multi-head self-attention module of the second transformer block.

[0084] Specifically, in order to ensure that the token features of the third group of tokens can be reasonably updated when passing through the second multi-head self-attention module (that is, only focusing on the features of this frame (token features of part of the first group of tokens and part of the second group of tokens) and the token features shared with this frame (token features of a group of time-series merged tokens)), the scheme of this embodiment introduces the attention mask method.

[0085] In the second multi-head self-attention module, the self-attention matrix can reflect the correlation between tokens and is used to achieve feature interaction between tokens. Each row in the self-attention matrix represents the correlation score of a token with other tokens. As long as the correlation scores with irrelevant tokens are masked (i.e., set to negative infinity), the token will not have feature interaction with irrelevant tokens. Therefore, when performing self-attention calculations, it is only necessary to mask the tokens of other frames and irrelevant shared tokens to achieve the token sharing effect; see Figure 4, that is, the correlation scores of the self-attention matrix corresponding to the t1 frame token-t2 frame token and the t2 frame token-t1 frame token are masked.

[0086] It should be noted that Figure 4 The t1 frame token in the embodiment represents part of the first group of tokens. Figure 4 The t2 frame token in represents part of the second group of tokens in this embodiment, and the shared token represents a group of timing merging tokens in this embodiment.

[0087] Optional, see Figure 4 , after getting the fourth set of tokens (corresponding to Figure 4 in ), the video processing method based on spatiotemporal token merging of the present invention can be performed on other adjacent video frames in a group of video frames to obtain A corresponding set of tokens, and based on 、 The corresponding two groups of tokens cyclically execute the video processing method based on spatiotemporal token merging.

[0088] Through this embodiment, the second multi-head self-attention module updates the features of the input tokens based on the attention mask, which can ensure that the token features of the merged third group of tokens can be reasonably updated when passing through the second multi-head self-attention module.

[0089] As an optional solution, before obtaining the first set of tokens and the second set of tokens, the method further includes:

[0090] S31, obtaining the video to be processed;

[0091] S32, converting the video to be processed into a group of video frames to be processed;

[0092] S33 , performing a preprocessing operation on a group of to-be-processed video frames to obtain a group of video frames, wherein the number of video frames in the group of video frames is an exponential multiple of 2.

[0093] Optional, attend Figure 3 The server includes a visual embedding module, which converts the video to be processed into a group of video frames to be processed and performs a preprocessing operation on the video to be processed so that the number of video frames in the group of video frames is an exponential multiple of 2.

[0094] It should be noted that Figure 3 Where T is the total number of frames of the video to be processed, H is the height of each frame of the video to be processed, W is the width of each frame of the video to be processed, M1 and M2 respectively represent the number of repetitions of the corresponding modules in the figure, C is the feature dimension of a token, and N is the number of tokens corresponding to each frame of the video.

[0095] It can be understood that the video processing method based on spatiotemporal token merging of the present invention represents Figure 3 The execution of a scheme in one stage, based on the input video loop, executes the video processing method based on spatiotemporal token merging. After each stage, the number of video frames can be reduced by 1 / 2. Therefore, the number of video frames in a group of video frames is adjusted to an exponential multiple of 2. Figure 3 After the loop execution of the video processing method based on spatiotemporal token merging, a final set of tokens can be obtained.

[0096] Optional, Figure 3 Each stage shown in FIG includes three converter blocks. The first converter block (e.g., Figure 3 The first transformer block in stage 1 is used to perform a sequential token merging step based on the updated token features, converting two groups of adjacent tokens into three groups of updated tokens (the tokens of the two adjacent groups of tokens and the tokens shared by the two adjacent groups of tokens), and inputting the three groups of updated tokens into the second transformer block (for example, Figure 2 (b) Transformer block in the figure), the second transformer block updates the features of the three updated tokens, merges the three updated tokens into a group of merged tokens, and then inputs this group of merged tokens into the third transformer block (e.g., Figure 3 The second transformer block in stage 1 in the figure above) performs steps such as attention masking and spatial token merging on a set of merged tokens to reduce the redundant tokens in the set of merged tokens. It can be understood that after each stage, the number of T video frames (T groups of tokens) will be reduced by half to T / 2 frames. After multiple stages, a set of tokens can be finally obtained.

[0097] Through this embodiment, the solution of this embodiment can achieve a significant improvement in computing efficiency at the expense of a small amount of precision.

[0098] Figure 5 This is a structural block diagram of the video processing based on spatiotemporal token merging provided by the present invention, such as Figure 5 As shown, including but not limited to the following modules:

[0099] An acquisition module 501 is configured to acquire a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames;

[0100] The first execution module 502 is configured to input the first group of tokens and the second group of tokens into a first transformer block to obtain part of the first group of tokens, part of the second group of tokens and a group of time sequence merged tokens output by the first transformer block, wherein the group of time sequence merged tokens is obtained by merging each time sequence token pair in a group of time sequence token pairs, the group of time sequence token pairs are determined based on similarity between the first group of tokens and the second group of tokens, the part of the first group of tokens is composed of tokens in the first group of tokens except the group of time sequence token pairs, and the part of the second group of tokens is composed of tokens in the second group of tokens except the group of time sequence token pairs.

[0101] The splicing module 503 is configured to splice the part of the first group of tokens, the part of the second group of tokens and the group of time sequence merged tokens to obtain a third group of tokens.

[0102] The second execution module 504 is configured to input the third group of tokens into a second transformer block to obtain a fourth group of tokens output by the second transformer block, wherein the third group of tokens include a group of space token pairs and part of the third group of tokens, the fourth group of tokens include a group of space merged tokens and the part of the third group of tokens, the group of space merged tokens is obtained by merging each space token pair in the group of space token pairs, and the group of space token pairs are determined based on similarity between each two tokens in the third group of tokens.

[0103] Through the embodiments of the present application, a first group of tokens and a second group of tokens are obtained, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; the first group of tokens and the second group of tokens are input into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of timing merge tokens output by the first converter block, wherein a group of timing merge tokens is obtained by merging each timing token pair in a group of timing token pairs, a group of timing token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except for a group of timing token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for a group of timing token pairs. ; Part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens are input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, a group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens, which solves the video processing method based on spatiotemporal token merging in the related art, which has the technical problems that video information is easily lost and processing performance is limited, thereby improving the efficiency of video processing.

[0104] It should be noted that the video processing device based on spatiotemporal token merging provided by the present invention can execute the video processing method based on spatiotemporal token merging of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0105] Figure 6 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device can include a processor 610, a communications interface 620, a memory 630, and a communications bus 640, wherein the processor 610, the communications interface 620, and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logic instruction in the memory 630 to execute a video processing method based on spatiotemporal token merging, which includes: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first transformer block to obtain a partial first group of tokens, a partial second group of tokens, and a group of temporal merged tokens output by the first transformer block, wherein the group of temporal merged tokens is obtained by merging each temporal token pair in a group of temporal token pairs, the group of temporal token pairs is determined based on a similarity between the first group of tokens and the second group of tokens, the partial first group of tokens is composed of tokens other than the group of temporal token pairs in the first group of tokens, and the partial second group of tokens is composed of tokens other than the group of temporal token pairs in the second group of tokens; splicing the partial first group of tokens, the partial second group of tokens, and the group of temporal merged tokens to obtain a third group of tokens; inputting the third group of tokens into a second transformer block to obtain a fourth group of tokens output by the second transformer block, wherein the third group of tokens includes a group of spatial token pairs and a partial third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and the partial third group of tokens, the group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and the group of spatial token pairs is determined based on a similarity between each two tokens in the third group of tokens.

[0106] In addition, the logic instruction in the memory 630 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the present application that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the video processing method based on spatiotemporal token merging provided by the above-mentioned embodiments, the method including: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of time-series merging tokens output by the first converter block, wherein a group of time-series merging tokens is obtained by merging each time-series token pair in a group of time-series token pairs, and a group of time-series merging tokens. The card pairs are determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens other than a group of temporal token pairs in the first group of tokens, and part of the second group of tokens is composed of tokens other than a group of temporal token pairs in the second group of tokens; part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and a group of spatial token pairs is determined based on the similarity between every two tokens in the third group of tokens.

[0108] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the video processing method based on spatiotemporal token merging provided in the above-mentioned embodiments, the method comprising: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of time-series merging tokens output by the first converter block, wherein a group of time-series merging tokens is obtained by merging each time-series token pair in a group of time-series token pairs, and a group of time-series token pairs is based on the first group of tokens and the second group of tokens. The similarity between the two groups of tokens is determined by the following method: part of the first group of tokens is composed of tokens other than a group of temporal token pairs in the first group of tokens, and part of the second group of tokens is composed of tokens other than a group of temporal token pairs in the second group of tokens; part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and a group of spatial token pairs is determined based on the similarity between every two tokens in the third group of tokens.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0110] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0111] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method based on spatiotemporal token merging, characterized in that: include: Obtain a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; Inputting the first group of tokens and the second group of tokens into a first converter block, obtaining a portion of the first group of tokens, a portion of the second group of tokens, and a group of time-series merged tokens output by the first converter block, wherein the group of time-series merged tokens is obtained by merging each time-series token pair in a group of time-series token pairs, the group of time-series token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, the portion of the first group of tokens is composed of tokens in the first group of tokens excluding the group of time-series token pairs, and the portion of the second group of tokens is composed of tokens in the second group of tokens excluding the group of time-series token pairs; Concatenate the portion of the first group of tokens, the portion of the second group of tokens, and the group of time-series merged tokens to obtain a third group of tokens; The third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, and the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

2. The video processing method based on spatiotemporal token merging according to claim 1, characterized in that: The first transformer block includes a first multi-head self-attention module and a first feedforward neural network module; The step of inputting the first group of tokens and the second group of tokens into a first converter block to obtain a portion of the first group of tokens, a portion of the second group of tokens, and a group of time-series merged tokens output by the first converter block includes: Inputting the first group of tokens and the second group of tokens into the first multi-head self-attention module, obtaining a set of first token value feature averages output by the first multi-head self-attention module, wherein the set of first token value feature averages includes the average value feature of each token in the first group of tokens and the second group of tokens in all attention heads; calculating a similarity between each token in the first set of tokens and each token in the second set of tokens based on an average value of the set of first token value features, and determining a set of similarity values; Selecting m similarity values ​​with the largest values ​​from the set of similarity values ​​in descending order, and determining the set of time sequence token pairs corresponding to the m similarity values, where m is a constant; Merging each of the timing token pairs in the set of timing token pairs to obtain the set of timing merge tokens; The first feedforward neural network module performs an update operation on token features of the portion of the first group of tokens, the portion of the second group of tokens, and the group of temporally merged tokens.

3. The video processing method based on spatiotemporal token merging according to claim 2, characterized in that: The calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on the average value of the first token value feature to determine a set of similarity values ​​includes: The cosine similarity between the average value of the first token value feature of each token in the first group of tokens and the average value of the first token value feature of each token in the second group of tokens is calculated to obtain the set of similarity values.

4. The video processing method based on spatiotemporal token merging according to claim 1, characterized in that: The second transformer block includes a second multi-head self-attention module and a second feed-forward neural network module; Inputting the third set of tokens into the second converter block to obtain a fourth set of tokens output by the second converter block includes: Inputting the third group of tokens into the second multi-head self-attention module to obtain a set of second token value feature averages output by the second multi-head self-attention module, wherein the set of second token value feature averages includes the average value feature of each token in the third group of tokens in all attention heads; randomly dividing the third group of tokens into a fifth group of tokens and a sixth group of tokens; calculating a similarity between each token in the fifth group of tokens and each token in the sixth group of tokens based on an average value of the set of second token value features, and determining a set of target similarity values; Selecting n target similarity values ​​with the largest values ​​from the set of target similarity values ​​in descending order, and determining the set of spatial token pairs corresponding to the n target similarity values, where n is a constant; Merging each spatial token pair in the set of spatial token pairs to obtain the set of spatial merge tokens; The second feedforward neural network module performs an update operation on the token features of the set of spatially merged tokens and the portion of the third set of tokens.

5. The video processing method based on spatiotemporal token merging according to claim 4, characterized in that: The second multi-head self-attention module updates the features of the input token based on the attention mask.

6. The video processing method based on spatiotemporal token merging according to any one of claims 1 to 5, characterized in that: Before obtaining the first set of tokens and the second set of tokens, the method further includes: Get the video to be processed; Converting the video to be processed into a set of video frames to be processed; A preprocessing operation is performed on the group of to-be-processed video frames to obtain the group of video frames, wherein the number of video frames in the group of video frames is an exponential multiple of 2.

7. A video processing device based on spatiotemporal token merging, characterized in that: include: An acquisition module, configured to acquire a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; a first execution module, configured to input the first group of tokens and the second group of tokens into a first converter block, and obtain, output by the first converter block, a portion of the first group of tokens, a portion of the second group of tokens, and a group of time-series merged tokens, wherein the group of time-series merged tokens is obtained by merging each time-series token pair in a group of time-series token pairs, the group of time-series token pairs being determined based on a similarity between the first group of tokens and the second group of tokens, the portion of the first group of tokens consisting of tokens in the first group of tokens other than the group of time-series token pairs, and the portion of the second group of tokens consisting of tokens in the second group of tokens other than the group of time-series token pairs; a splicing module, configured to splice part of the first group of tokens, part of the second group of tokens, and the group of time-series merged tokens to obtain a third group of tokens; The second execution module is used to input the third group of tokens into the second converter block to obtain the fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and the part of the third group of tokens, and the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video processing method based on spatiotemporal token merging according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video processing method based on spatiotemporal token merging according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method based on spatiotemporal token merging according to any one of claims 1 to 6 is implemented.

Images