Video processing method and device based on space-time token merging, and electronic equipment

Abstract

The invention provides a video processing method and device based on space-time token merging, electronic equipment and a storage medium, and belongs to the technical field of video processing, and the method comprises the steps: obtaining a first group of tokens and a second group of tokens, the first group of tokens and the second group of tokens being two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into the first converter block to obtain a part of the first group of tokens, a part of the second group of tokens and a group of time sequence combination tokens output by the first converter block; splicing a part of the first group of tokens, a part of the second group of tokens and the group of time sequence combination tokens to obtain a third group of tokens; and inputting the third group of tokens into the second converter block to obtain a fourth group of tokens output by the second converter block. According to the video processing method based on time-space token combination, the technical problems that video information is easy to lose and the processing performance is limited in the video processing method based on time-space token combination in the related technology are solved.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video processing method, device and electronic equipment based on spatiotemporal token merging. Background Art

[0002] The goal of video text retrieval is to quickly retrieve related videos or texts from a database given a text or video as a query; processing videos requires input of multiple frames of images, which requires more computational effort during inference, hindering the deployment of video text retrieval models in real-world large-scale scenarios; since there is information redundancy between adjacent frames of a video, feeding multiple frames of images into a visual encoder will generate many similar redundant tokens. By reducing the number of these redundant tokens, the computational effort required for video inference can be effectively reduced.

[0003] In the related art, when processing video data, the number of redundant tokens is reduced by discarding meaningless frames and then filtering out detailed areas from the remaining frames. However, this strategy lacks contextual modeling of continuous time information, resulting in limited performance. There are also related schemes that reduce the number of redundant tokens by discarding information that appears earlier in the sequence and only retaining tokens with semantic importance. However, its frame-by-frame calculation method is inefficient.

[0004] It can be seen that the video processing method based on spatiotemporal token merging in the related art has technical problems that video information is easily lost and the processing performance is limited. Summary of the invention

[0005] The present invention provides a video processing method, device and electronic device based on time-space token merging, which are used to solve the technical problems of video processing methods based on time-space token merging in related technologies, such as easy loss of video information and limited processing performance.

[0006] The present invention provides a video processing method based on spatiotemporal token merging, comprising the following steps: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block, obtaining part of the first group of tokens, part of the second group of tokens and a group of temporal merging tokens output by the first converter block, wherein the group of temporal merging tokens is obtained by merging each temporal token pair in a group of temporal token pairs, the group of temporal token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, and the part of the first group of tokens is composed of the first group of tokens except the group of temporal token pairs and the second group of tokens. The part of the second group of tokens is composed of tokens in the second group of tokens except the group of timing token pairs; the part of the first group of tokens, the part of the second group of tokens and the group of timing merge tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merge tokens and the part of the third group of tokens, the group of spatial merge tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0007] According to a video processing method based on spatiotemporal token merging provided by the present invention, the first converter block includes a first multi-head self-attention module and a first feedforward neural network module; the first group of tokens and the second group of tokens are input into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens output by the first converter block, including: the first group of tokens and the second group of tokens are input into the first multi-head self-attention module to obtain a group of first token value feature averages output by the first multi-head self-attention module, wherein the group of first token value feature averages includes each token in the first group of tokens and the second group of tokens in all attention The method comprises the following steps: calculating the average value feature of the value in the force head; calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on the average value of the value feature of the group of first tokens, and determining a group of similarity values; selecting m similarity values ​​with the largest values ​​from the group of similarity values ​​in order from large to small, and determining the group of time series token pairs corresponding to the m similarity values, wherein m is a constant; merging each time series token pair in the group of time series token pairs to obtain the group of time series merged tokens; and performing an update operation on the token features of the part of the first group of tokens, the part of the second group of tokens and the group of time series merged tokens by the first feedforward neural network module.

[0008] According to a video processing method based on spatiotemporal token merging provided by the present invention, the similarity between each token in the first group of tokens and each token in the second group of tokens is calculated based on the average value of the first token value features of the group, and a group of similarity values ​​is determined, including: calculating the cosine similarity between the average value of the first token value features of each token in the first group of tokens and the average value of the first token value features of each token in the second group of tokens, to obtain the group of similarity values.

[0009] According to a video processing method based on spatiotemporal token merging provided by the present invention, the second transformer block includes a second multi-head self-attention module and a second feedforward neural network module; the third group of tokens are input into the second transformer block to obtain a fourth group of tokens output by the second transformer block, including: inputting the third group of tokens into the second multi-head self-attention module to obtain a group of second token value feature averages output by the second multi-head self-attention module, wherein the group of second token value feature averages includes the average value of the value feature of each token in the third group of tokens in all attention heads; randomly dividing the third group of tokens into a fifth group of tokens and a sixth group of tokens; group tokens; calculate the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens based on the average value of the feature of the group of second token values, and determine a group of target similarity values; select n target similarity values ​​with the largest values ​​from the group of target similarity values ​​in order from large to small, and determine the group of spatial token pairs corresponding to the n target similarity values, wherein n is a constant; merge each spatial token pair in the group of spatial token pairs to obtain the group of spatial merged tokens; the second feedforward neural network module performs an update operation on the token features of the group of spatial merged tokens and the part of the third group of tokens.

[0010] According to a video processing method based on spatiotemporal token merging provided by the present invention, the second multi-head self-attention module updates the features of the input token based on an attention mask.

[0011] According to a video processing method based on spatiotemporal token merging provided by the present invention, before obtaining the first group of tokens and the second group of tokens, the method also includes: obtaining a video to be processed; converting the video to be processed into a group of video frames to be processed; performing a preprocessing operation on the group of video frames to be processed to obtain the group of video frames, wherein the number of video frames in the group of video frames is an exponential multiple of 2.

[0012] The present invention also provides a video processing device based on spatiotemporal token merging, comprising the following modules: an acquisition module, used to acquire a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; a first execution module, used to input the first group of tokens and the second group of tokens into a first converter block, to obtain part of the first group of tokens, part of the second group of tokens and a group of temporal merging tokens output by the first converter block, wherein the group of temporal merging tokens is obtained by merging each temporal token pair in a group of temporal token pairs, the group of temporal token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, and the part of the first group of tokens is composed of the first group of tokens except the group of temporal token pairs. The second group of tokens is composed of tokens other than the group of timing token pairs, and the part of the second group of tokens is composed of tokens in the second group of tokens except the group of timing token pairs; a splicing module is used to splice the part of the first group of tokens, the part of the second group of tokens and the group of timing merge tokens to obtain a third group of tokens; a second execution module is used to input the third group of tokens into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and the part of the third group of tokens, the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the video processing method based on spatiotemporal token merging as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video processing method based on spatiotemporal token merging as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the video processing method based on spatiotemporal token merging as described above is implemented.

[0016] The video processing method, device, electronic device and storage medium based on spatiotemporal token merging provided by the present invention input the first group of tokens and the second group of tokens into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens output by the first converter block, that is, the token pairs with higher similarity in the first group of tokens and the second group of tokens are merged to obtain a group of temporal merged tokens, which reduces the redundant tokens between two adjacent video frames in the time dimension (that is, in two different video frames) and avoids the possibility of loss of feature information of the tokens; the third group of tokens is input into the second converter block to obtain the fourth group of tokens output by the second converter block, that is, the tokens with higher similarity in the third group of tokens are merged in pairs, and the number of tokens is further reduced in the spatial dimension (that is, in the same video frame), while maintaining the model accuracy without a significant decrease, the calculation efficiency of the model is improved, thereby solving the technical problems of the video processing method based on spatiotemporal token merging in the related art, that is, the video information is easily lost and the processing performance is limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a video processing method based on spatiotemporal token merging provided by the present invention.

[0019] Figure 2 It is one of the schematic diagrams of the video processing method based on spatiotemporal token merging provided by the present invention.

[0020] Figure 3 This is the second schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention.

[0021] Figure 4 This is the third schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention.

[0022] Figure 5 It is a structural block diagram of a video processing device based on spatiotemporal token merging provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that in the description of the present invention, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0027] The goal of video text retrieval is to quickly retrieve related videos or texts from a database given a text or video as a query; processing videos requires input of multiple frames of images, which requires more computational effort during inference, hindering the deployment of video text retrieval models in real-world large-scale scenarios; since there is information redundancy between adjacent frames of a video, feeding multiple frames of images into a visual encoder will generate many similar redundant tokens. By reducing the number of these redundant tokens, the computational effort required for video inference can be effectively reduced.

[0028] The Transformer Model is a deep neural network model based on the self-attention mechanism. The Transformer Model can process a flexible number of input tokens, which facilitates the reduction of redundant tokens during the calculation process. In the related art, when processing video data, the number of redundant tokens is reduced by discarding meaningless frames and then filtering out detailed areas from the remaining frames. However, this strategy lacks context modeling of continuous time information, resulting in limited performance. There are also related schemes that reduce the number of redundant tokens by discarding information that appears earlier in the sequence and only retaining tokens with semantic importance. However, its frame-by-frame calculation method is inefficient. It can be seen that the video processing method based on spatiotemporal token merging in the related art has technical problems such as easy loss of video information and limited processing performance.

[0029] In order to at least solve some of the above problems, the following Figure 1-Figure 6 The video processing method, device, electronic device and storage medium based on spatiotemporal token merging provided by the present invention are described; the video processing method based on spatiotemporal token merging provided by the present invention can be applied to the scene of processing videos to reduce redundant tokens in the videos; the video processing method based on spatiotemporal token merging provided by the present invention can be executed by a server, and the server runs a transformer model.

[0030] Figure 1 is one of the flow charts of the video processing method based on spatiotemporal token merging provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps: Step 101: Obtain a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames.

[0031] In this embodiment, the video to be processed is composed of a group of video frames. A video frame may contain multiple visual elements, such as characters, objects, scenes, etc. In order to convert the content of the video frame into a processable text form, features of the video frame can be extracted, and then these features can be mapped to a group of tokens. This group of tokens may include words, phrases, sentences, etc., which together constitute a description of the content of the video frame; therefore, each video frame in a group of video frames represents a group of tokens, and the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames.

[0032] Step 102, input the first group of tokens and the second group of tokens into the first converter block, and obtain part of the first group of tokens, part of the second group of tokens and a group of timing merge tokens output by the first converter block, wherein a group of timing merge tokens is obtained by merging each timing token pair in a group of timing token pairs, a group of timing token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except for a group of timing token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for a group of timing token pairs.

[0033] The server that executes the video processing method based on spatiotemporal token merging of the present invention runs a transformer model, and the server executes the video processing method based on spatiotemporal token merging based on the transformer model; the transformer model is composed of multiple transformer blocks (Transformer blocks), and the transformer blocks are used to update the features of tokens to enhance the representation ability of a group of tokens.

[0034] In this embodiment, the first group of tokens and the second group of tokens are input into the first converter block, and the first converter block outputs part of the first group of tokens, part of the second group of tokens and a group of time-merged tokens; it should be noted that the first converter block will update the characteristics of the input tokens, and therefore, compared with the first group of tokens and the second group of tokens, the characteristics of the tokens of part of the first group of tokens, part of the second group of tokens and a group of time-merged tokens are updated.

[0035] A group of timing token pairs is determined based on the similarity between a first group of tokens and a second group of tokens, and each timing token pair in a group of timing token pairs is composed of a token in the first group of tokens and a token in the second group of tokens; optionally, the similarity between each token in the first group of tokens and each token in the second group of tokens is calculated, and token pairs with higher similarity are selected to form a group of timing token pairs.

[0036] A group of time-series merged tokens is obtained by merging each time-series token pair in a group of time-series token pairs, that is, each time-series token pair in a group of time-series token pairs is merged into a group of time-series merged tokens; part of the first group of tokens is composed of tokens in the first group of tokens except for one group of time-series token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for one group of time-series token pairs.

[0037] Step 103, concatenate part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens to obtain a third group of tokens.

[0038] Part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens are spliced ​​to obtain a third group of tokens, that is, the first group of tokens and the second group of tokens are merged into the third group of tokens, thereby achieving a reduction in the number of video frames.

[0039] It should be noted that, in this embodiment, the merging of timing tokens refers to the merging of different groups of tokens, that is, the merging of different video frames.

[0040] Understandably, such concatenation operation simplifies the output format of sequential token merging (i.e., from three output features to one), avoiding the token fragmentation problem that may be caused by multiple applications of sequential token merging.

[0041] Step 104, input the third group of tokens into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and a group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0042] A group of spatial token pairs is determined based on the similarity between every two tokens in the third group of tokens, that is, multiple tokens with the highest similarity between each other in the third group of tokens can be selected as a group of spatial token pairs, and the third group of tokens includes a group of spatial token pairs and part of the third group of tokens; each spatial token pair in a group of spatial token pairs is merged to obtain a group of spatial merged tokens, and the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens.

[0043] It should be noted that the second transformer block will update the input token features, and the features of the tokens of the fourth group of tokens are updated compared to the third group of tokens; the merging of spatial tokens refers to the merging of tokens within the same group of tokens.

[0044] In this embodiment, the first group of tokens and the second group of tokens are input into the first converter block, and the output results of the first converter block are spliced ​​and input into the second converter block to obtain a fourth group of tokens, thereby achieving a reduction in time and space of redundant tokens of the first group of tokens and the second group of tokens; optionally, the method of this embodiment can be cyclically executed on all adjacent video frames in a group of video frames. For example, when the number of video frames (each video frame corresponds to a group of tokens) in a group of video frames is x, the method of this embodiment is executed on every two adjacent video frames in a group of video frames to obtain x / 2 groups of tokens, and the method of this embodiment is executed again on the x / 2 groups of tokens to obtain x / 4 groups of tokens, and so on, to finally obtain a group of tokens.

[0045] Through the embodiments provided by the present application, a first group of tokens and a second group of tokens are obtained, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; the first group of tokens and the second group of tokens are input into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of timing merge tokens output by the first converter block, wherein a group of timing merge tokens is obtained by merging each timing token pair in a group of timing token pairs, a group of timing token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except for a group of timing token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for a group of timing token pairs. The invention relates to a method for producing a video processing method of a video converter comprising: forming a video processing method of a video converter comprising a first group of spatial tokens, a second group of tokens and a group of temporally merged tokens, so as to obtain a third group of tokens; inputting the third group of tokens into the second converter block, so as to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens comprises a group of spatial token pairs and a part of the third group of tokens, the fourth group of tokens comprises a group of spatially merged tokens and a part of the third group of tokens, a group of spatially merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, a group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens, the method solves the technical problems that the video information is easily lost and the processing performance is limited in the video processing method based on the merging of spatiotemporal tokens in the related art, and the efficiency of video processing is improved.

[0046] As an optional solution, the first transformer block includes a first multi-head self-attention module and a first feedforward neural network module; Inputting the first group of tokens and the second group of tokens into the first converter block, obtaining part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens output by the first converter block, including: S11, inputting the first group of tokens and the second group of tokens into the first multi-head self-attention module, and obtaining a set of first token value feature averages output by the first multi-head self-attention module, wherein the set of first token value feature averages includes the average value of the value features of each token in the first group of tokens and the second group of tokens in all attention heads; S12, calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on an average value of a set of first token value features, and determining a set of similarity values; S13, selecting m similarity values ​​with the largest values ​​from a group of similarity values ​​in descending order, and determining a group of time sequence token pairs corresponding to the m similarity values, where m is a constant; S14, merging each timing token pair in a group of timing token pairs to obtain a group of timing merged tokens; S15, the first feedforward neural network module performs an update operation on the token features of part of the first group of tokens, part of the second group of tokens, and a group of time-series merged tokens.

[0047] Figure 2 is one of the schematic diagrams of the video processing method based on spatiotemporal token merging provided by the present invention, such as Figure 2 As shown, Figure 2 The (a) Transformer block in FIG. 5 corresponds to the first transformer block in this embodiment.

[0048] The first set of tokens (corresponding to Figure 2 The first group of tokens corresponds to t1 in the video frame, which means the video frame corresponding to time t1) and the second group of tokens (corresponding to Figure 2 A group of tokens corresponding to t in (which means the video frame corresponding to time t2) is input into the first multi-head self-attention module to obtain a group of first token value feature averages output by the first multi-head self-attention module, wherein a group of first token value feature averages includes the average value features of each token in the first group of tokens and the second group of tokens in all attention heads; that is, using the value features of the tokens in the multi-head self-attention layer, the average value features of each token in the first group of tokens and the second group of tokens in all attention heads are calculated to obtain a group of first token value feature averages.

[0049] Optionally, calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on an average value of a set of first token value features to determine a set of similarity values ​​includes: The cosine similarity between the average value of the first token value feature of each token in the first group of tokens and the average value of the first token value feature of each token in the second group of tokens is calculated to obtain a set of similarity values.

[0050] Specifically, for two tokens, the cosine similarity between the average values ​​of the first token value features of the two tokens can be calculated. The closer the obtained cosine similarity is to 1, the more similar the two tokens are. Therefore, based on the cosine similarity between the average value of the first token value features of each token in the first group of tokens and the average value of the first token value features of each token in the second group of tokens, the obtained set of similarity values ​​can reflect the similarity between each token in the first group of tokens and each token in the second group of tokens.

[0051] After determining the token similarity calculation method, it is necessary to quickly determine the matching relationship based on the similarity between the first group of tokens and the second group of tokens. In order to prevent the matching process from introducing too much calculation and reducing the model calculation efficiency, a fast approximation method can be used here. That is, for each token in the second group of tokens, the token in the first group of tokens with the greatest similarity is taken as the matching pair.

[0052] Select m with the largest value from a set of similarity values ​​in descending order (m in this embodiment corresponds to Figure 2 In the above method, k similarity values ​​are selected and a set of time-series token pairs corresponding to m similarity values ​​are determined, where m is a constant. After completing the token matching, a high similarity of the token pairs indicates that the two tokens contain a high degree of information duplication, and the information loss after the tokens are merged is small. Therefore, for all the token pairs, the m token pairs with the largest similarity are selected and merged to obtain a set of time-series merged tokens.

[0053] Furthermore, the first feedforward neural network module performs an update operation on token features of a portion of the first group of tokens, a portion of the second group of tokens, and a group of temporally merged tokens.

[0054] Optionally, after obtaining part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens output by the first feedforward neural network module, the part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens are input into a third transformer block (corresponding to Figure 2 (b) Transformer block in ).

[0055] Specifically, after passing through the first transformer block equipped with a temporal token merging module, the features are converted into three parts: the token features of each of the two adjacent frames (part of the first group of tokens, part of the second group of tokens) and the token features shared by the two adjacent frames (a group of temporal merged tokens). These features are sent to the subsequent Transformer block (the third transformer block). When performing multi-head self-attention calculations, the shared token features need to be allowed to participate in the update process of the token features of the adjacent frames at the same time to prevent large information loss within the video frame, see Figure 2 In part (d), the self-attention of each feature is calculated as follows: Partial first group of tokens: The features of the partial first group of tokens are used as query features in the self-attention calculation, and the features of the partial first group of tokens and a group of temporally merged tokens are used as key and value features. That is, the partial first group of tokens needs to pay attention to the features of the tokens contained in itself and the features of a group of temporally merged tokens to complete the update. It can be understood that when the features of a group of temporally merged tokens are exactly the same as the features of the tokens merged by the first group of tokens, the output features of the partial first group of tokens calculated in this way are exactly the same as the output features of the first group of tokens.

[0056] Partial second group of tokens: The features of the partial second group of tokens are used as query features in the self-attention calculation, and the features of the partial second group of tokens and a group of temporally merged tokens are used as key and value features. That is, the partial second group of tokens need to pay attention to the features of the tokens contained in themselves and the features of a group of temporally merged tokens to complete the update. It can be understood that when the features of a group of temporally merged tokens are exactly the same as the features of the tokens merged by the second group of tokens, the output features of the partial second group of tokens calculated in this way are exactly the same as the output features of the second group of tokens.

[0057] Features of a group of time-merged tokens: The features of a group of time-merged tokens are used as query features in self-attention calculations, and the features of part of the first group of tokens, part of the second group of tokens, and a group of time-merged tokens are used as key and value features. That is, the features of a group of time-merged tokens need to pay attention to the token features contained in themselves and the token features contained in part of the first group of tokens and part of the second group of tokens to complete the update. It can be understood that the features of a group of time-merged tokens can aggregate the information of two adjacent frames and play a certain time series modeling function.

[0058] Through this embodiment, the number of redundant tokens in the first group of tokens and the second group of tokens can be effectively reduced, while information loss can be alleviated and processing efficiency can be improved.

[0059] As an optional solution, the second transformer block includes a second multi-head self-attention module and a second feedforward neural network module; Inputting the third group of tokens into the second transformer block, obtaining a fourth group of tokens output by the second transformer block, comprising: S21, inputting the third group of tokens into the second multi-head self-attention module, obtaining a set of second token value feature averages output by the second multi-head self-attention module, wherein the set of second token value feature averages includes the average value of the value feature of each token in the third group of tokens in all attention heads; S22, randomly dividing the third group of tokens into a fifth group of tokens and a sixth group of tokens; S23, calculating the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens based on a set of average values ​​of second token value features, and determining a set of target similarity values; S24, selecting n target similarity values ​​with the largest values ​​from a set of target similarity values ​​in descending order, and determining a set of spatial token pairs corresponding to the n target similarity values, where n is a constant; S25, merging each spatial token pair in a set of spatial token pairs to obtain a set of spatial merged tokens; S26, the second feedforward neural network module performs an update operation on the token features of a group of spatially merged tokens and a portion of the third group of tokens.

[0060] Figure 3 Schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention is the second one. Figure 3 As shown, the steps of this embodiment correspond to Figure 3 The spatial token merging step in .

[0061] Figure 3 The Transformer block including the spatial token merging step in stage 1 corresponds to the second transformer block in this embodiment, and the second transformer block includes a second multi-head self-attention module and a second feedforward neural network module.

[0062] The third group of tokens is input into the second multi-head self-attention module to obtain a set of second token value feature averages output by the second multi-head self-attention module, wherein the set of second token value feature averages includes the average value of the value feature of each token in the third group of tokens in all attention heads; the third group of tokens corresponds to Figure 3 t in i The corresponding set of tokens.

[0063] The third group of tokens is randomly divided into the fifth group of tokens and the sixth group of tokens, corresponding to Figure 3 The third group of tokens is randomly divided into two groups A and B; the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens is calculated based on a set of second token value feature averages, and a set of target similarity values ​​is determined. Here, the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens can be determined based on the cosine similarity between the second token value feature average of each token in the fifth group of tokens and the second token value feature average of each token in the sixth group of tokens.

[0064] Select the n with the largest value from a set of target similarity values ​​in descending order (n in this embodiment corresponds to Figure 3The method comprises the following steps: first, determining k) target similarity values ​​in the set of spatial token pairs, and determining a set of spatial token pairs corresponding to n target similarity values, wherein n is a constant; merging each spatial token pair in the set of spatial token pairs to obtain a set of spatial merged tokens; and performing an update operation on the token features of the set of spatial merged tokens and part of the third set of tokens by the second feedforward neural network module.

[0065] Through this embodiment, the number of redundant tokens in the third group of tokens can be effectively reduced, while information loss is alleviated and processing efficiency is improved.

[0066] As an optional solution, the second multi-head self-attention module updates the features of the input token based on the attention mask.

[0067] Figure 4 Schematic diagram of the video processing method based on spatiotemporal token merging provided by the present invention is shown in FIG. Figure 4 As shown, in this embodiment, the attention mask method is introduced into the second multi-head self-attention module of the second transformer block.

[0068] Specifically, in order to ensure that the token features of the third group of tokens can be reasonably updated when passing through the second multi-head self-attention module (that is, only focusing on the features of this frame (token features of part of the first group of tokens and part of the second group of tokens) and the token features shared with this frame (token features of a group of time-series merged tokens)), the scheme of this embodiment introduces an attention mask method.

[0069] In the second multi-head self-attention module, the self-attention matrix can reflect the correlation between tokens and is used to achieve feature interaction between tokens. Each row in the self-attention matrix represents the correlation score between a certain token and other tokens. As long as the correlation score with irrelevant tokens is masked (that is, set to negative infinity), the token will not have feature interaction with irrelevant tokens; therefore, when performing self-attention calculations, it is only necessary to mask the tokens of other frames and irrelevant shared tokens to achieve the token sharing effect; see Figure 4 , that is, the correlation scores of the self-attention matrices corresponding to the t1 frame token-t2 frame token and the t2 frame token-t1 frame token are masked.

[0070] It should be noted that Figure 4 The t1 frame token in the example represents part of the first group of tokens in this embodiment. Figure 4 The t2 frame token in represents part of the second group of tokens in this embodiment, and the shared token represents a group of timing merging tokens in this embodiment.

[0071] Optional, see Figure 4 , after getting the fourth set of tokens (corresponding to Figure 4 In ), the video processing method based on spatiotemporal token merging of the present invention can be performed on other adjacent video frames in a group of video frames to obtain A corresponding set of tokens, based on , The corresponding two groups of tokens cyclically execute the video processing method based on spatiotemporal token merging.

[0072] Through this embodiment, the second multi-head self-attention module updates the features of the input tokens based on the attention mask, which can ensure that the token features of the merged third group of tokens can be reasonably updated when passing through the second multi-head self-attention module.

[0073] As an optional solution, before obtaining the first set of tokens and the second set of tokens, the method further includes: S31, obtaining the video to be processed; S32, converting the video to be processed into a group of video frames to be processed; S33, performing a preprocessing operation on a group of video frames to be processed to obtain a group of video frames, wherein the number of video frames in the group of video frames is an exponential multiple of 2.

[0074] Optional, attend Figure 3 The server includes a visual embedding module, which converts the video to be processed into a group of video frames to be processed and performs a preprocessing operation on the video to be processed so that the number of video frames in the group of video frames is an exponential multiple of 2.

[0075] It should be noted that Figure 3 Where T is the total number of frames of the video to be processed, H is the height of each frame of the video to be processed, W is the width of each frame of the video to be processed, M1 and M2 respectively represent the number of repetitions of the corresponding modules in the figure, C is the feature dimension of a token, and N is the number of tokens corresponding to each frame of the video.

[0076] It can be understood that the video processing method based on spatiotemporal token merging of the present invention represents Figure 3 The execution of the scheme of one stage in the process is based on the input video loop to execute the video processing method based on spatiotemporal token merging. After each stage, the number of video frames can be reduced by 1 / 2. Therefore, the number of video frames in a group of video frames is adjusted to an exponential multiple of 2. Figure 3 After the video processing method based on spatiotemporal token merging is executed cyclically, a final set of tokens can be obtained.

[0077] Optional, Figure 3 Each stage shown in FIG. 1 includes three converter blocks. The first converter block (e.g., Figure 3The first transformer block in stage 1 in the process is used to perform a sequential token merging step based on the updated token features, converting two groups of adjacent tokens into three groups of updated tokens (the tokens of the two adjacent groups of tokens and the tokens shared by the two adjacent groups of tokens), and input the three groups of updated tokens into the second transformer block (for example, Figure 2 (b) Transformer block in the figure), the second transformer block updates the features of the three updated tokens, merges the three updated tokens into a group of merged tokens, and then inputs this group of merged tokens into the third transformer block (for example, Figure 3 The second transformer block in stage 1 in the figure), the third transformer block performs steps such as attention mask and spatial token merging on a group of merged tokens to reduce the redundant tokens in a group of merged tokens; it can be understood that after each stage, the T-frame video frames (T groups of tokens) will be reduced by half to T / 2 frames of video frames. After multiple stages, a group of tokens can be finally obtained.

[0078] Through this embodiment, the solution of this embodiment can achieve a significant improvement in computing efficiency at the cost of a small amount of precision.

[0079] Figure 5 It is a structural block diagram of the video processing based on spatiotemporal token merging provided by the present invention, such as Figure 5 As shown, including but not limited to the following modules: An acquisition module 501 is used to acquire a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; A first execution module 502 is used to input the first group of tokens and the second group of tokens into a first converter block, and obtain a part of the first group of tokens, a part of the second group of tokens, and a group of sequential merge tokens output by the first converter block, wherein the group of sequential merge tokens is obtained by merging each sequential token pair in a group of sequential token pairs, the group of sequential token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, the part of the first group of tokens is composed of tokens in the first group of tokens except the group of sequential token pairs, and the part of the second group of tokens is composed of tokens in the second group of tokens except the group of sequential token pairs; A splicing module 503, configured to splice the part of the first group of tokens, the part of the second group of tokens and the group of time-series merged tokens to obtain a third group of tokens; The second execution module 504 is used to input the third group of tokens into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and the part of the third group of tokens, and the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

[0080] Through the embodiments of the present application, a first group of tokens and a second group of tokens are obtained, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; the first group of tokens and the second group of tokens are input into the first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of timing merge tokens output by the first converter block, wherein a group of timing merge tokens is obtained by merging each timing token pair in a group of timing token pairs, a group of timing token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except for a group of timing token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except for a group of timing token pairs. ; Splice part of the first group of tokens, part of the second group of tokens and a group of temporal merged tokens to obtain a third group of tokens; input the third group of tokens into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, a group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens, which solves the technical problems of easy loss of video information and limited processing performance in the video processing method based on spatiotemporal token merging in the related art, thereby improving the efficiency of video processing.

[0081] It should be noted that the video processing device based on spatiotemporal token merging provided by the present invention can execute the video processing method based on spatiotemporal token merging of any of the above-mentioned embodiments during specific operation, which will not be elaborated in this embodiment.

[0082] Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a video processing method based on spatiotemporal token merging, the method comprising: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens output by the first converter block, wherein a group of time-series merged tokens is obtained by merging each time-series token pair in a group of time-series token pairs, a group of time-series token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, and part of the first group of tokens is determined by the first group of tokens. The tokens are composed of tokens other than a group of time-sequential token pairs in the tokens, and part of the second group of tokens is composed of tokens other than a group of time-sequential token pairs in the second group of tokens; part of the first group of tokens, part of the second group of tokens and a group of time-sequential merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and a group of spatial token pairs is determined based on the similarity between every two tokens in the third group of tokens.

[0083] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0084] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the video processing method based on spatiotemporal token merging provided in the above-mentioned embodiments, and the method includes: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of time-series merging tokens output by the first converter block, wherein a group of time-series merging tokens is obtained by merging each time-series token pair in a group of time-series token pairs, and a group of time-series merging tokens. The card pairs are determined based on the similarity between the first group of tokens and the second group of tokens, part of the first group of tokens is composed of tokens in the first group of tokens except a group of time-sequential token pairs, and part of the second group of tokens is composed of tokens in the second group of tokens except a group of time-sequential token pairs; part of the first group of tokens, part of the second group of tokens and a group of time-sequential merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and a group of spatial token pairs is determined based on the similarity between every two tokens in the third group of tokens.

[0085] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the video processing method based on spatiotemporal token merging provided in the above-mentioned embodiments, the method comprising: obtaining a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; inputting the first group of tokens and the second group of tokens into a first converter block to obtain part of the first group of tokens, part of the second group of tokens and a group of time-series merging tokens output by the first converter block, wherein a group of time-series merging tokens is obtained by merging each time-series token pair in a group of time-series token pairs, and a group of time-series token pairs is based on the first group of tokens and the second group of tokens. The first group of tokens is determined by the similarity between the first group of tokens, part of the first group of tokens is composed of tokens other than a group of timing token pairs in the first group of tokens, and part of the second group of tokens is composed of tokens other than a group of timing token pairs in the second group of tokens; part of the first group of tokens, part of the second group of tokens and a group of timing merged tokens are spliced ​​to obtain a third group of tokens; the third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, a group of spatial merged tokens is obtained by merging each spatial token pair in a group of spatial token pairs, and a group of spatial token pairs is determined based on the similarity between every two tokens in the third group of tokens.

[0086] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0087] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video processing method based on spatiotemporal token merging, characterized in that: include: Obtain a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; Inputting the first group of tokens and the second group of tokens into a first converter block, obtaining part of the first group of tokens, part of the second group of tokens and a group of time-series merged tokens output by the first converter block, wherein the group of time-series merged tokens is obtained by merging each time-series token pair in a group of time-series token pairs, the group of time-series token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, the part of the first group of tokens is composed of tokens in the first group of tokens other than the group of time-series token pairs, and the part of the second group of tokens is composed of tokens in the second group of tokens other than the group of time-series token pairs; Concatenate the part of the first group of tokens, the part of the second group of tokens and the group of time-series merged tokens to obtain a third group of tokens; The third group of tokens is input into the second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, and the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

2. The video processing method based on spatiotemporal token merging according to claim 1, characterized in that: The first transformer block includes a first multi-head self-attention module and a first feed-forward neural network module; The step of inputting the first group of tokens and the second group of tokens into a first converter block to obtain a portion of the first group of tokens, a portion of the second group of tokens, and a group of time-series merged tokens output by the first converter block includes: Inputting the first group of tokens and the second group of tokens into the first multi-head self-attention module, obtaining a set of first token value feature averages output by the first multi-head self-attention module, wherein the set of first token value feature averages includes the average value of the value features of each token in the first group of tokens and the second group of tokens in all attention heads; Calculate the similarity between each token in the first group of tokens and each token in the second group of tokens based on the average value of the first token value feature, and determine a set of similarity values; Selecting m similarity values ​​with the largest values ​​from the group of similarity values ​​in descending order, and determining the group of time sequence token pairs corresponding to the m similarity values, where m is a constant; Merging each timing token pair in the set of timing token pairs to obtain the set of timing merge tokens; The first feedforward neural network module performs an update operation on token features of the portion of the first group of tokens, the portion of the second group of tokens, and the group of temporally merged tokens.

3. The video processing method based on spatiotemporal token merging according to claim 2, characterized in that: The calculating the similarity between each token in the first group of tokens and each token in the second group of tokens based on the average value of the first token value feature to determine a group of similarity values ​​includes: The cosine similarity between the average value of the first token value feature of each token in the first group of tokens and the average value of the first token value feature of each token in the second group of tokens is calculated to obtain the set of similarity values.

4. The video processing method based on spatiotemporal token merging according to claim 1, characterized in that: The second transformer block includes a second multi-head self-attention module and a second feed-forward neural network module; The step of inputting the third group of tokens into a second converter block to obtain a fourth group of tokens output by the second converter block comprises: Input the third group of tokens into the second multi-head self-attention module to obtain a set of second token value feature averages output by the second multi-head self-attention module, wherein the set of second token value feature averages includes the average value of the value feature of each token in the third group of tokens in all attention heads; randomly dividing the third group of tokens into a fifth group of tokens and a sixth group of tokens; Calculate the similarity between each token in the fifth group of tokens and each token in the sixth group of tokens based on the average value of the set of second token value features, and determine a set of target similarity values; Selecting n target similarity values ​​with the largest values ​​from the set of target similarity values ​​in descending order, and determining the set of spatial token pairs corresponding to the n target similarity values, wherein n is a constant; Merging each spatial token pair in the set of spatial token pairs to obtain the set of spatial merged tokens; The second feedforward neural network module performs an update operation on the token features of the set of spatially merged tokens and the portion of the third set of tokens.

5. The video processing method based on spatiotemporal token merging according to claim 4, characterized in that: The second multi-head self-attention module updates the features of the input token based on the attention mask.

6. The video processing method based on spatiotemporal token merging according to any one of claims 1 to 5, characterized in that: Before obtaining the first set of tokens and the second set of tokens, the method further includes: Get the video to be processed; Converting the video to be processed into a set of video frames to be processed; A preprocessing operation is performed on the group of video frames to be processed to obtain the group of video frames, wherein the number of video frames in the group of video frames is an exponential multiple of 2.

7. A video processing device based on spatiotemporal token merging, characterized in that: include: An acquisition module, used for acquiring a first group of tokens and a second group of tokens, wherein the first group of tokens and the second group of tokens are two groups of tokens corresponding to two adjacent video frames in a group of video frames; A first execution module, configured to input the first group of tokens and the second group of tokens into a first converter block, and obtain a portion of the first group of tokens, a portion of the second group of tokens, and a group of sequential merged tokens output by the first converter block, wherein the group of sequential merged tokens is obtained by merging each sequential token pair in a group of sequential token pairs, the group of sequential token pairs is determined based on the similarity between the first group of tokens and the second group of tokens, the portion of the first group of tokens is composed of tokens in the first group of tokens other than the group of sequential token pairs, and the portion of the second group of tokens is composed of tokens in the second group of tokens other than the group of sequential token pairs; A splicing module, used for splicing the part of the first group of tokens, the part of the second group of tokens and the group of time-series combined tokens to obtain a third group of tokens; A second execution module is used to input the third group of tokens into a second converter block to obtain a fourth group of tokens output by the second converter block, wherein the third group of tokens includes a group of spatial token pairs and part of the third group of tokens, and the fourth group of tokens includes a group of spatial merged tokens and part of the third group of tokens, and the group of spatial merged tokens is obtained by merging each spatial token pair in the group of spatial token pairs, and the group of spatial token pairs is determined based on the similarity between each two tokens in the third group of tokens.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the video processing method based on spatiotemporal token merging as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video processing method based on spatiotemporal token merging as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method based on spatiotemporal token merging as claimed in any one of claims 1 to 6 is implemented.

Images