Video processing method and video processing device

Abstract

The embodiment of the invention provides a video processing method and a video processing device. The method comprises the following steps: decomposing a video into multiple frames of video images; dividing each frame of video image in the plurality of frames of video images into a plurality of data units; determining the similarity between the central data unit in the first sliding window in the second frame of video image and each data unit in the second sliding window in the first frame of video image to obtain a first result; determining the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window to obtain a second result; and obtaining a characteristic value of each data unit in the first sliding window based on the first result and the second result. According to the method and the device provided by the embodiment of the invention, the calculation amount in the video processing task can be reduced, and the real-time processing capability of the system can be improved.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a video processing method and a video processing device. Background Art

[0002] In the field of computer vision and deep learning, especially in video processing tasks, computational cost and resource requirements are an important challenge. As video resolution and frame rate increase, calculating each data unit in the video image frame by frame consumes a lot of computing resources, resulting in slower processing speed and increased energy consumption. This calculation method is obviously not efficient enough, so it is necessary to find an optimization method. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a video processing method and a video processing device, which are beneficial to reducing the amount of calculation in the video processing task, and further beneficial to improving the real-time processing capability of the system.

[0004] In a first aspect, a method for video processing is provided, the method comprising: decomposing a video into multiple video frames, the multiple video frames comprising adjacent first and second video frames, and the second frame image data being located after the first frame image data; dividing each video frame in the multiple video frames into multiple data units; determining the similarity between a central data unit in a first sliding window in the second video frame and each data unit in a second sliding window in the first video frame to obtain a first result, the second sliding window being at the same position as the first sliding window, the first sliding window and the second sliding window having the same size and both being (2n+1)*(2n+1), n ​​being a positive integer, and the number of data units included in the sliding window being less than the number of data units included in one video frame; determining the similarity between a central data unit in the first sliding window and surrounding data units in the first sliding window to obtain a second result; and obtaining a feature value of each data unit in the second sliding window based on the first result and the second result.

[0005] In one possible implementation, based on the first result and the second result, a characteristic value of each data unit in the second sliding window is obtained, including: if the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, replacing the characteristic value of the central data unit with the characteristic value of any data unit in the at least one data unit; and / or if the central data unit in the first sliding window has similarity with at least one surrounding data unit in the first sliding window, replacing the characteristic value of at least one surrounding data unit with the characteristic value of the central data unit.

[0006] In one possible implementation, if the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, the characteristic value of the central data unit is replaced with the characteristic value of any data unit in the at least one data unit, including: if the central data unit in the first sliding window has similarity with multiple data units in the second sliding window, the characteristic value of the central data unit is replaced with the characteristic value of the first data unit among the multiple data units, the first data unit being the data unit with the highest similarity to the central data unit.

[0007] In one possible implementation, based on the first result and the second result, the characteristic value of each data unit in the second sliding window is obtained, including: if the central data unit in the first sliding window has similarity with all surrounding data units in the first sliding window and there is no data unit in the second sliding window that has similarity with the central data unit in the first sliding window, calculate the characteristic value of any data unit in the first sliding window, and replace the characteristic values ​​of other data units in the first sliding window with the characteristic value of any data unit.

[0008] In one possible implementation, the method is applied to a Transformer model, the Transformer model includes a query matrix Q, a key matrix K and a value matrix V, and the data unit is a Token. The method also includes: when there is at least one Token in the second video image that has similarity with the Token in the first video image, the value corresponding to the at least one Token in the query matrix Q, the key matrix K or the value matrix V of the second video image is replaced with the value of the Token that has similarity with the at least one Token in the stored query matrix Q, the key matrix K or the value matrix V of the first video image.

[0009] In one possible implementation, the method is applied to a Transformer model, the Transformer model includes an attention matrix, and the data unit is a Token. The method also includes: when there are at least two Tokens in the second video image that have similarity with the Tokens in the first video image, the correlation values ​​of any two Tokens in the attention matrix of the second video image are replaced with the correlation values ​​of two Tokens that have similarity with the any two Tokens in the attention matrix of the stored first video image.

[0010] In a possible implementation, a method for calculating the similarity between data units includes at least one of the following methods: L1 norm, L2 norm and cosine similarity.

[0011] In a possible implementation, n=1.

[0012] In a second aspect, a video processing device is provided, including: a decomposition unit, used to decompose a video into multiple frames of video images, the multiple frames of video images include adjacent first frames of video images and second frames of video images, and the second frame of image data is located after the first frame of image data; a division unit, used to divide each frame of video images in the multiple frames of video images into multiple data units; a determination unit, used to determine the similarity of each data unit in a second sliding window of a central data unit in a first sliding window in the second frame of video image to obtain a first result, the second sliding window is at the same position as the first sliding window in the first frame of video image, the first sliding window and the second sliding window have the same size and are both (2n+1)*(2n+1), n ​​is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in one frame of video image; and determine the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window to obtain a second result; an acquisition unit, used to acquire a feature value of each data unit in the first sliding window based on the first result and the second result.

[0013] In a third aspect, a video processing device is provided, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the video processing device executes the method provided by the first aspect and any possible implementation thereof.

[0014] Based on the above technical solution, a first result is obtained by determining the similarity between a central data unit in a first sliding window in a subsequent frame of video image and each data unit in a second sliding window at the same position in a previous frame of video image, and a second result is obtained by determining the similarity between a central data unit in a first sliding window in a subsequent frame of video image and surrounding data units in the first sliding window. Finally, based on the first result and the second result, a characteristic value of each data unit in the first sliding window of the subsequent frame of video image is obtained, which is beneficial to significantly reduce the amount of calculation by identifying and reusing the characteristic values ​​of similar data units. In addition, in high frame rate and high resolution video processing, it is also beneficial to improve the real-time processing capability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without paying creative work.

[0016] Figure 1 A schematic block diagram of a video processing method according to an embodiment of the present application is shown.

[0017] Figure 2 A comparison diagram of similarity in a spatial dimension of an embodiment of the present application is shown.

[0018] Figure 3 Another comparison diagram of similarity in spatial dimension according to an embodiment of the present application is shown.

[0019] Figure 4 A schematic block diagram of a video processing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] Deep learning has been widely used in video processing tasks due to its advantages such as strong learning ability, wide coverage, good portability, and automatic feature extraction, especially the Transformer model. But no matter which deep learning model is used, extracting the feature value of each data unit in the video image is an essential step. However, as the video resolution and frame rate increase, calculating each data unit in each frame of the video image frame by frame will consume a lot of computing resources, resulting in slower processing speed and increased energy consumption. This calculation method is obviously not efficient enough, so it is necessary to find an optimization method.

[0022] In view of this, an embodiment of the present application provides a method for video processing, which obtains a first result by determining the similarity between a central data unit in a first sliding window in a subsequent frame of video image and each data unit in a second sliding window at the same position in a previous frame of video image, and obtains a second result by determining the similarity between a central data unit in a first sliding window in a subsequent frame of video image and surrounding data units in the first sliding window, and finally obtains a characteristic value of each data unit in the first sliding window of the subsequent frame of video image based on the first result and the second result, which is beneficial to significantly reduce the amount of calculation by identifying and reusing the characteristic values ​​of similar data units; in addition, in high frame rate and high resolution video processing, it is also beneficial to improve the real-time processing capability of the system.

[0023] Figure 1 FIG. 1 is a schematic block diagram of a video processing method provided by an embodiment of the present application. Figure 1 As shown, the method 100 includes part or all of the following contents.

[0024] S110, decomposing a video sequence into a plurality of video image frames, wherein the plurality of video image frames include a first video image frame and a second video image frame that are adjacent to each other, and the image data of the second video image frame is located after the image data of the first video image frame;

[0025] S120, dividing each frame of the multiple frames of video images into a plurality of data units;

[0026] S130, determining the similarity between the central data unit in the first sliding window in the second frame of the video image and each data unit in the second sliding window in the first frame of the video image, and obtaining a first result, wherein the second sliding window is at the same position as the first sliding window, the first sliding window and the second sliding window have the same size, and are both (2n+1)*(2n+1), where n is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in one frame of the video image;

[0027] S140, determining the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window to obtain a second result;

[0028] S150: Based on the first result and the second result, obtain a characteristic value of each data unit in the first sliding window.

[0029] Optionally, the video processing method of the embodiment of the present application may be executed by a processor, for example, a general-purpose processor or a hardware accelerator in the Transformer model. The general-purpose processor may be, for example, a CPU or a GPU, and the hardware accelerator may be, for example, an FPGA or an ASIC. The embodiment of the present application does not limit the execution subject of the video processing method.

[0030] It should be noted that method 100 may also include only one of steps S130 and S140, that is, S130 and S140 do not have to be executed at the same time, and the characteristic value of each data unit in the first sliding window can be obtained only based on the first result obtained by S130, or the characteristic value of each data unit in the first sliding window can be obtained only based on the second result obtained by S140. The embodiment of the present application is not limited to this.

[0031] In step S110, the video is usually a dynamic image sequence composed of continuously played frames. Through the picture acquisition technology, each moment of the picture can be accurately extracted to make it an independent frame. Independent frames can be more conveniently processed and analyzed separately. Each frame can be regarded as a static image, that is, each video can be decomposed into multiple frames of video images using the picture acquisition technology.

[0032] In step S120, by dividing each frame of video image into multiple representative data units, the accuracy and efficiency of subsequent image processing and analysis can be improved. The data unit here can be, for example, a Token in the Transformer model, or a basic data unit in other models. The embodiment of the present application does not limit the size of the data unit, and it is mainly enough to refine the video image.

[0033] Assume that each frame of video image is divided into x data units, usually x=y*y, y is a positive integer greater than 1, for two consecutive frames of video image, for example, the i-th frame of video image and the (i+1)-th frame of video image, a fixed-size sliding window is selected at the same position, and the number of data units included in the sliding window should be less than the number of data units included in one frame of video image. In the embodiment of the present application, the size of the sliding window is (2n+1)*(2n+1), and n is a positive integer. Then there will be a central data unit in the sliding window in the (i+1)-th frame of video image, and the other data units in the sliding window except the central data unit can be recorded as surrounding data units.

[0034] In step 130, the central data unit in the sliding window in the (i+1)th frame of video image and each data unit in the sliding window in the i-th frame of video image can be compared for similarity, and the result is recorded as the first result.

[0035] It should be noted that step 130 is a loop step, that is, after obtaining a first result, the sliding window can be moved, and the solution of step 130 can be used again to obtain the next first result. And for the same frame of video image, the data units in two adjacent sliding windows need to overlap partially until the first result corresponding to each data unit in the (i+1)th frame of video image is obtained.

[0036] The following will be combined Figure 2 and Figure 3 Step S130 is described in detail. In the figure below, the data unit is a Token in the Transformer model.

[0037] like Figure 2 and Figure 3 As shown in the figure, the i-th frame video image and the (i+1)-th frame video image are divided into 6*6=36 Tokens, which are respectively denoted as Token i1 ,......,Token i36 And Token (i+1)1 ,......,Token (i+1)36,,A sliding window includes 3*3=9 Tokens. For the first step S130, the specific position of the sliding window is as follows Figure 2 As shown, in this Figure 2 In the (i+1)th frame video image, the center Token in the sliding window is Token (i+1)8 , and compare the similarity between it and each Token in the sliding window of the i-th frame video image. (i+1)8 Respectively with Token i1 Token i2 Token i3 Token i7 Token i8 Token i9 Token i13 Token i14 Token i15 Compare similarities and get Token (i+1)8 The corresponding first result is shown in Table 1.

[0038] <![CDATA[Token i1 ]]> <![CDATA[Token i2 ]]> <![CDATA[Token i3 ]]> <![CDATA[Token i7 ]]> <![CDATA[Token i8 ]]> <![CDATA[Token i9 ]]> <![CDATA[Token i13 ]]> <![CDATA[Token i14 ]]> <![CDATA[Token i15 ]]> <![CDATA[Token (i+1)8 ]]> 0 0 1 1 0 0 1 0 0

[0039] Next, move the sliding window two columns to the right and left, so that the sliding window after the move overlaps with the sliding window before the move by at least one column of tokens. This not only saves the number of calculations, but also ensures that the two sliding windows before and after the move have overlapping parts, so that the calculations of the two sliding windows are not separated.

[0040] For the second step S130, the specific position of the sliding window after the movement is as follows: Figure 3 As shown. Figure 3 In the (i+1)th frame video image, the center Token in the sliding window is Token (i+1)10 , and compare the similarity with each Token in the sliding window of the i-th frame video image, that is, Token (i+1)10 Respectively with Token i3 Token i4 Token i5 Token i9 Token i10 Token i11 Token i15 Token i16 Token i17 Compare similarities and get Token (i+1)10 The corresponding first result is shown in Table 2.

[0041] <![CDATA[Token i3 ]]> <![CDATA[Token i4 ]]> <![CDATA[Token i5 ]]> <![CDATA[Token i9 ]]> <![CDATA[Token i10 ]]> <![CDATA[Token i11 ]]> <![CDATA[Token i15 ]]> <![CDATA[Token i16 ]]> <![CDATA[Token i17 ]]> <![CDATA[Token (i+1)10 ]]> 1 1 0 0 1 1 0 0 0

[0042] It should be noted that the sliding window can also be moved downward, and step S130 is repeatedly performed until each Token in the (i+1)th frame of the video image is used as the center Token to obtain a first result. For the Token at the edge position of the (i+1)th frame of the video image, the similarity of the Token at the edge position and the Token falling within the extended sliding window of the i-th frame of the video image can be compared by extending the sliding window, and the first result corresponding to the Token at the edge position can be obtained.

[0043] In practical applications, the size and shape of the sliding window are adjusted according to the requirements of the specific task to balance the computational complexity and the adequacy of information capture.

[0044] In step S140, the central data unit in the sliding window of the (i+1)th frame of video image is compared with all surrounding data units in the sliding window of the (i+1)th frame of video image for similarity, and the result is recorded as the second result.

[0045] It should be noted that step 140 may also be a loop step, that is, after obtaining a second result, the sliding window may be moved, and the solution of step 140 may be used again to obtain the next second result. In other words, the number of executions of step 140 is related to the number of movements of the sliding window, and step 140 is executed once each time the sliding window is moved.

[0046] Specifically, Figure 2 As shown, for the first step S140, the center Token in the sliding window in the (i+1)th frame of the video image is Token (i+1)8 , compare the similarity between it and other surrounding Tokens in the sliding window in the (i+1)th frame of the video image. (i+1)8 Respectively with Token (i+1)1 Token (i+1)2 Token (i+1)3 Token (i+1)7 Token (i+1)9 Token (i+1)13 Token (i+1)14 Token (i+1)15 Compare similarities and get Token (i+1)8 The corresponding second result is shown in Table 3.

[0047] <![CDATA[Token (i+1)1 ]]> <![CDATA[Token (i+1)2 ]]> <![CDATA[Token (i+1)3 ]]> <![CDATA[Token (i+1)7 ]]> <![CDATA[Token (i+1)9 ]]> <![CDATA[Token (i+1)13 ]]> <![CDATA[Token (i+1)14 ]]> <![CDATA[Token (i+1)15 ]]> <![CDATA[Token (i+1)8 ]]> 1 1 1 1 1 1 1 1

[0048] Next, if Figure 3As shown, for the second step S140, the center Token in the sliding window in the (i+1)th frame of the video image is Token (i+1)10 , compare the similarity between it and other surrounding Tokens in the sliding window in the (i+1)th frame of the video image. (i+1)10 Respectively with Token (i+1)3 Token (i+1)4 Token (i+1)5 Token (i+1)9 Token (i+1)11 Token (i+1)15 Token (i+1)16 Token (i+1)17 Compare similarities and get Token (i+1)10 The corresponding second result is shown in Table 4.

[0049] <![CDATA[Token (i+1)3 ]]> <![CDATA[Token (i+1)4 ]]> <![CDATA[Token (i+1)5 ]]> <![CDATA[Token (i+1)9 ]]> <![CDATA[Token (i+1)11 ]]> <![CDATA[Token (i+1)15 ]]> <![CDATA[Token (i+1)16 ]]> <![CDATA[Token (i+1)17 ]]> <![CDATA[Token (i+1)10 ]]> 1 1 1 1 1 1 1 1

[0050] Optionally, in the embodiment of the present application, step 130 can be understood as a process of obtaining the similarity between data units in the time dimension, and step 140 can be understood as a process of obtaining the similarity between data units in the space dimension.

[0051] Optionally, both step 130 and step 140 need to be performed, and based on the first result obtained in step 130 and the second result obtained in step 140, the feature values ​​of all Tokens in the sliding window in the (i+1)th frame of the video image are determined.

[0052] Optionally, only one of step 130 and step 140 needs to be executed, and based on the first result obtained in step 130 or the second result obtained in step 140, the characteristic values ​​of all Tokens in the sliding window in the (i+1)th frame of the video image are determined.

[0053] In an embodiment of the present application, a variety of difference evaluation functions can be used to measure the similarity between data units, for example, the L2 norm (i.e., Euclidean distance), which calculates the straight-line distance of the feature vectors between data units; another example, the L1 norm (also known as Manhattan distance or absolute difference), which evaluates the difference by accumulating the absolute value of the difference in each dimension; another example, cosine similarity, which measures their directional similarity by calculating the cosine value of the angle between the feature vectors of two data units, without considering the size of the vector. The selection of these difference evaluation functions depends on the needs of the specific application scenario and the consideration of data characteristics to ensure that similar data units between frames can be efficiently identified. If the result calculated by any selected similarity evaluation indicator (such as L2 norm, L1 norm or cosine similarity) is lower than its preset threshold, it can be considered that the two data units currently under investigation have similarity under this indicator.

[0054] In one embodiment, step S150, i.e., obtaining the characteristic value of each data unit in the first sliding window based on the first result and the second result, includes: if the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, replacing the characteristic value of the central data unit with the characteristic value of any data unit in the at least one data unit; and / or, if the central data unit in the first sliding window has similarity with at least one surrounding data unit in the first sliding window, replacing the characteristic value of at least one surrounding data unit with the characteristic value of the central data unit.

[0055] In the time dimension, the central Token in the sliding window of the (i+1)th frame of video image is used as the reference target, and a sliding window is constructed at the corresponding position of the i-th frame of video image. In the sliding window of the i-th frame of video image, tokens similar to the reference Token in the sliding window of the (i+1)th frame of video image are searched one by one. If a similar token is found, its feature value can be reused directly, because the feature value of the similar token has been calculated in the i-th frame of video image. If no similar token is found, its feature value needs to be recalculated.

[0056] In the spatial dimension, the center Token in the sliding window of the (i+1)th frame of the video image is used as the reference target, and a Token similar to the center Token is searched in the sliding window of the (i+1)th frame of the video image. If a similar Token is found, the feature values ​​of the center Token and the similar Token can be obtained at the same time as long as the feature value of one of the center Token and the similar Token is obtained.

[0057] For example, if a token similar to the center token is found in the sliding window of the (i+1)th frame of the video image, and the center token has established a corresponding similarity relationship in the time dimension, the feature value of the token similar to the center token in the sliding window of the i-th frame of the video image can be directly reused. If two tokens similar in the spatial dimension do not establish a similarity relationship in the time dimension, the feature value of the token in the sliding window of the i-th frame of the video image cannot be reused, but only the feature value of one of the tokens needs to be calculated.

[0058] In one embodiment, if the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, the characteristic value of the central data unit is replaced with the characteristic value of any data unit in the at least one data unit, including: if the central data unit in the first sliding window has similarity with multiple data units in the second sliding window, the characteristic value of the central data unit is replaced with the characteristic value of the first data unit among the multiple data units, and the first data unit is the data unit in the second sliding window with the highest similarity to the central data unit.

[0059] That is, when the similarity indexes between multiple tokens in the sliding window of the i-th video image and the central token in the sliding window of the (i+1)-th video image simultaneously meet the condition of being lower than the threshold, the token pair with the closest value is selected. When the calculation results of the similarity indexes between all tokens in the sliding window of the i-th video image and the central token in the sliding window of the (i+1)-th video image are higher than the threshold, we can judge that in the sliding window of the i-th video image, no token similar to the reference token in the sliding window of the (i+1)-th frame video image is found in the spatial and temporal dimensions, which indicates that the video content or features at this position have changed significantly between frames.

[0060] In another embodiment, step S150, i.e., obtaining the characteristic value of each data unit in the first sliding window based on the first result and the second result, includes: if the central data unit in the first sliding window has similarity with all surrounding data units in the first sliding window and there is no data unit in the second sliding window that has similarity with the central data unit in the first sliding window, calculating the characteristic value of any data unit in the first sliding window, and replacing the characteristic values ​​of other data units in the first sliding window with the characteristic value of any data unit.

[0061] by Figure 2 For example, assume that the center Token in the sliding window in the (i+1)th frame of the video image is (i+1)8It has no similarity with all the tokens in the sliding window in the i-th frame of the video image, but the center token ( i +1)8 and other Tokens in the sliding window in the (i+1)th frame of the video image, i.e., Token ( i +1)1 Token (i+1)2 Token (i+1)3 Token (i+1)7 Token (i+1)9 Token (i+1)13 Token (i+1)14 Token (i+1)15 If all of them have similarity, then the feature value of any Token in the sliding window in the (i+1)th frame of video image can be calculated, and all other Tokens in the sliding window in the (i+1)th frame of video image can reuse the feature value of the calculated Token. That is, the feature values ​​of all Tokens in the sliding window in the (i+1)th frame of video image are the same.

[0062] Again Figure 2 For example, assume that the center Token in the sliding window in the (i+1)th frame of the video image is (i+1)8 The Token in the sliding window in the i-th frame video image i5 is the most similar Token pair, and the central Token (i+1)8 and other Tokens in the sliding window in the (i+1)th frame of the video image, i.e., Token (i+1)1 Token (i+1)2 Token (i+1)3 Token (i+1)7 Token (i+1)9 Token (i+1)13 Token (i+1)14 Token(i +1)15 If they are similar, then all the tokens in the sliding window in the (i+1)th frame of the video image can reuse the tokens in the sliding window in the i-th frame of the video image. i5 The characteristic value of .

[0063] As mentioned above, the video processing method can be applied to a Transformer model, which includes a query matrix Q, a key matrix K and a value matrix V, and the data unit is a Token. The method may also include: when there is at least one Token in the second video image that has similarity with a Token in the first video image, replacing the value corresponding to at least one Token in the query matrix Q, key matrix K or value matrix V of the second video image with the value of the Token that has similarity with at least one Token in the stored query matrix Q, key matrix K or value matrix V of the first video image.

[0064] Specifically, for each frame of video image, it is necessary to obtain the corresponding query matrix Q, key matrix K and value matrix V. The query matrix Q, key matrix K and value matrix V are obtained in the following way:

[0065] (1) Get the given input sequence X = [x1, x2, ..., x n ], where each x i Represents the feature value of a Token in a video image.

[0066] (2) Perform three sets of linear transformations on the input sequence X to obtain the query matrix Q, the key matrix K or the value matrix V. Q ,W K ,W V They are trainable weight matrices used to calculate the query matrix Q, key matrix K or value matrix V, respectively. The calculation formula is as follows:

[0067] Q=XW Q

[0068] K=XW K

[0069] V=XW V

[0070] That is to say, the query matrix Q, key matrix K or value matrix V are respectively linear transformations of the input sequence, that is, when the eigenvalue of each Token in a frame of video image is obtained, the query matrix Q, key matrix K or value matrix V corresponding to the video image can be obtained through the above calculation formula.

[0071] After obtaining the characteristic value of each Token in each frame of video image, it can be stored. At the same time, when the corresponding query matrix Q, key matrix K and value matrix V are obtained, they can also be stored. Once it is determined in the time dimension that a Token of the current frame of video image is similar to a Token of the previous frame of video image, the value related to the Token in the query matrix Q, key matrix K or value matrix V of the current frame of video image can reuse the value corresponding to the Token with similarity in the query matrix Q, key matrix K or value matrix V of the previous frame of video image. This is feature reuse.

[0072] For example, for the query matrix Q of the i-th frame video image, we have: [q i1 ,q i2 ,...,q in ]=[x i1 ,x i2 ,...,x in ]W Q , where x ij Represents the feature value of the jth Token in the i-th frame video image, q ij It represents the value of the jth Token in the query matrix Q of the i-th frame of the video image. i1 ,q i2 ,...,q in ] is stored. If it is determined that the mth Token in the (i+1)th frame of the video image is similar to the (m-1)th Token in the i-th frame of the video image, then q can be directly m-1 The q value corresponding to the mth Token of the query matrix Q input to the (i+1)th frame of the video image.

[0073] Similarly, the key matrix K and the value matrix V are obtained in a similar way, which will not be described in detail here.

[0074] In an embodiment of the present application, the Transformer model also includes an attention matrix, and the method also includes: when there are at least two tokens in the second video image that have similarity with the tokens in the first video image, replacing the correlation values ​​of any two tokens of the at least two tokens in the attention matrix of the second video image with the correlation values ​​of two tokens that have similarity with the any two tokens in the stored attention matrix of the first video image.

[0075] After obtaining the query matrix Q and the key matrix K, we can also calculate the attention score and get the attention matrix through the dot product. Specifically, the attention matrix can be obtained by the following method, where d k is the dimension of the K vector, used to scale the dot product to avoid large values.

[0076]

[0077] Similar to the above embodiment, once the attention matrix of a frame of video image is obtained, it is immediately stored.

[0078] That is to say, once it is determined in the time dimension that two Tokens of the current frame video image are similar to two Tokens of the previous frame video image, the product of the q value and the k value corresponding to the two Tokens in the attention matrix of the current frame video image can reuse the product of the q value and the k value corresponding to the two similar Tokens in the attention matrix V of the previous frame video image. This is relational reuse.

[0079] For example, for the current frame video image, the i-th Token and the j-th Token are similar to the i'-th Token and the j'-th Token of the previous frame video image respectively. When calculating the attention matrix of the current frame video image, the product of the q value of the i-th Token and the k value of the j-th Token can be replaced by the q value of the i'-th Token and the k value of the j'-th Token, or the product of the k value of the i-th Token and the q value of the j-th Token can be replaced by the product of the k value of the i'-th Token and the q value of the j'-th Token.

[0080] Based on the same idea, the embodiment of the present application also provides a video processing device. Figure 4 As shown, the video processing device 200 includes part or all of the following contents.

[0081] A decomposition unit 210, configured to decompose a video into multiple frames of video images, wherein the multiple frames of video images include adjacent first and second frames of video images, and the second frame of image data is located after the first frame of image data;

[0082] A division unit 220, configured to divide each frame of a plurality of frames of video images into a plurality of data units;

[0083] The determining unit 230 is used to determine the similarity between the central data unit in the first sliding window in the second frame of the video image and each data unit in the second sliding window in the first frame of the video image to obtain a first result, the second sliding window is in the same position as the first sliding window, the first sliding window and the second sliding window have the same size and are both (2n+1)*(2n+1), n ​​is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in one frame of the video image; and determine the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window to obtain a second result;

[0084] The acquisition unit 240 is used to acquire the characteristic value of each data unit in the first sliding window based on the first result and the second result.

[0085] In a possible embodiment, the acquisition unit 240 is specifically used to: if the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, replace the characteristic value of the central data unit with the characteristic value of any data unit in the at least one data unit; and / or if the central data unit in the first sliding window has similarity with at least one surrounding data unit in the first sliding window, replace the characteristic value of at least one surrounding data unit with the characteristic value of the central data unit.

[0086] In a possible embodiment, the acquisition unit 240 is specifically used to: if the central data unit in the first sliding window has similarity with multiple data units in the second sliding window, replace the characteristic value of the central data unit with the characteristic value of the first data unit among the multiple data units, and the first data unit is the data unit with the highest similarity to the central data unit in the second sliding window.

[0087] In a possible embodiment, the acquisition unit 240 is specifically used to: if the central data unit in the first sliding window has similarity with all surrounding data units in the first sliding window and there is no data unit in the second sliding window that has similarity with the central data unit in the first sliding window, calculate the characteristic value of any data unit in the first sliding window, and replace the characteristic values ​​of other data units in the first sliding window with the characteristic value of any data unit.

[0088] In a possible embodiment, the video processing device is applied to a Transformer model, the Transformer model includes a query matrix Q, a key matrix K and a value matrix V, the data unit is a Token, and the device also includes: a replacement unit, which is used to replace the value corresponding to at least one Token in the query matrix Q, key matrix K or value matrix V of the second video image with the value of the Token that has similarity with at least one Token in the stored query matrix Q, key matrix K or value matrix V of the first video image when there is at least one Token in the second video image that has similarity with the Token in the first video image.

[0089] In a possible embodiment, the video processing device is applied to a Transformer model, the Transformer model includes an attention matrix, the data unit is a Token, and the device also includes: a replacement unit, which is used to replace the correlation values ​​of any two Tokens in the attention matrix of the second video image with the correlation values ​​of two Tokens that have similarity with any two Tokens in the attention matrix of the stored first video image when there are at least two Tokens in the second video image that have similarity with the Tokens in the first video image.

[0090] In a possible embodiment, the similarity calculation method between data units includes at least one of the following methods: L1 norm, L2 norm and cosine similarity.

[0091] In a possible embodiment, n=1.

[0092] Based on the same idea, the embodiment of the present application also provides another video processing device. The video processing device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the video processing device executes the above-mentioned various method embodiments.

[0093] It should be noted that the detailed contents of the device side embodiment can be found in the method side embodiment, and for the sake of brevity, they will not be repeated here.

[0094] Although the present application has been described with reference to preferred embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, the various technical features mentioned in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A video processing method, characterized in that: include: Decomposing a video into multiple video frames, wherein the multiple video frames include a first video frame and a second video frame that are adjacent to each other, and the second video frame is located after the first video frame; Dividing each frame of the multiple frames of video images into a plurality of data units; Determine the similarity between a central data unit in a first sliding window in the second frame of video image and each data unit in a second sliding window in the first frame of video image, and obtain a first result, wherein the second sliding window is at the same position as the first sliding window, the first sliding window and the second sliding window have the same size and are both (2n+1)*(2n+1), where n is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in one frame of the video image; Determine the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window to obtain a second result; Based on the first result and the second result, a feature value of each data unit in the first sliding window is obtained.

2. The method according to claim 1, characterized in that The acquiring, based on the first result and the second result, a feature value of each data unit in the first sliding window includes: If the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, replacing the characteristic value of the central data unit with the characteristic value of any data unit in the at least one data unit; and / or If the central data unit in the first sliding window has similarity with at least one surrounding data unit in the first sliding window, the feature value of the at least one surrounding data unit is replaced with the feature value of the central data unit.

3. The method according to claim 2, characterized in that If the central data unit in the first sliding window has similarity with at least one data unit in the second sliding window, replacing the characteristic value of the central data unit with the characteristic value of any data unit in the at least one data unit comprises: If the central data unit in the first sliding window has similarity with multiple data units in the second sliding window, the characteristic value of the central data unit is replaced with the characteristic value of the first data unit among the multiple data units, and the first data unit is the data unit in the second sliding window with the highest similarity to the central data unit.

4. The method according to claim 1, characterized in that: The acquiring, based on the first result and the second result, a feature value of each data unit in the first sliding window includes: If the central data unit in the first sliding window has similarity with all surrounding data units in the first sliding window and there is no data unit in the second sliding window that has similarity with the central data unit in the first sliding window, calculate the characteristic value of any data unit in the first sliding window, and replace the characteristic values ​​of other data units in the first sliding window with the characteristic value of any data unit.

5. The method according to claim 1, characterized in that The method is applied to a Transformer model, the Transformer model includes a query matrix Q, a key matrix K and a value matrix V, the data unit is a Token, and the method further includes: When there is at least one Token in the second video image that has similarity with a Token in the first video image, the value corresponding to the at least one Token in the query matrix Q, the key matrix K or the value matrix V of the second video image is replaced with the value of the Token that has similarity with the at least one Token in the stored query matrix Q, the key matrix K or the value matrix V of the first video image.

6. The method according to claim 1, characterized in that The method is applied to a Transformer model, the Transformer model includes an attention matrix, the data unit is a Token, and the method further includes: When there are at least two tokens in the second video image that have similarity with the tokens in the first video image, the correlation values ​​of the attention matrix of the second video image with any two tokens of the at least two tokens are replaced with the correlation values ​​of two tokens that have similarity with the any two tokens in the attention matrix of the first video image stored.

7. The method according to claim 1, characterized in that The similarity calculation method between the data units includes at least one of the following methods: L1 norm, L2 norm and cosine similarity.

8. The method according to claim 1, characterized in that n＝1。 9. A video processing device, characterized in that: include: A decomposition unit, used to decompose a video into multiple frames of video images, wherein the multiple frames of video images include a first frame of video image and a second frame of video image that are adjacent to each other, and the second frame of video image is located after the first frame of video image; A division unit, used for dividing each frame of the multiple frames of video images into multiple data units; a determining unit, configured to determine a similarity between a central data unit in a first sliding window in the second frame of video image and each data unit in a second sliding window in the first frame of video image, to obtain a first result, wherein the second sliding window is at the same position as the first sliding window, the first sliding window and the second sliding window have the same size and are both (2n+1)*(2n+1), n ​​is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in one frame of the video image; and determine a similarity between a central data unit in the first sliding window and surrounding data units in the first sliding window, to obtain a second result; An acquiring unit is used to acquire a characteristic value of each data unit in the first sliding window based on the first result and the second result.

10. A video processing device, characterized in that: The video processing device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the video processing device executes the method according to any one of claims 1 to 8.

Images