Video processing methods and video processing devices

Abstract

This application provides a video processing method and a video processing apparatus. The method includes: decomposing a video segment into multiple video frames; dividing each video frame into multiple data units; determining the similarity between a central data unit within a first sliding window in a second video frame and each data unit within a second sliding window in the first video frame, obtaining a first result; determining the similarity between the central data unit within the first sliding window and surrounding data units within the first sliding window, obtaining a second result; and based on the first and second results, obtaining the feature value of each data unit within the first sliding window. The method and apparatus of this application are beneficial for reducing the computational load in video processing tasks, thereby improving the real-time processing capability of the system.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning, and in particular to a method and apparatus for video processing. Background Technology

[0002] In computer vision and deep learning, especially in video processing tasks, computational cost and resource requirements are significant challenges. As video resolution and frame rate increase, computing each data unit in a video image frame by frame consumes substantial computational resources, leading to slower processing speeds and increased energy consumption. This computational method is clearly inefficient, necessitating the search for optimization methods. Summary of the Invention

[0003] In view of this, the embodiments of this application provide a video processing method and a video processing apparatus, which are beneficial to reducing the amount of computation in video processing tasks and thus improving the real-time processing capability of the system.

[0004] In a first aspect, a video processing method is provided, comprising: decomposing a video segment into multiple video frames, the multiple video frames including adjacent first and second video frames, wherein the second frame data is located after the first frame data; dividing each video frame in the multiple video frames into multiple data units; determining the similarity between the center data unit in a first sliding window in the second video frame and each data unit in a second sliding window in the first video frame, to obtain a first result, wherein the second sliding window is located at the same position as the first sliding window, the first and second sliding windows have the same size, and both are (2n+1)*(2n+1), where n is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in a single video frame; determining the similarity between the center data unit in the first sliding window and the surrounding data units in the first sliding window, to obtain a second result; and obtaining the feature value of each data unit in the second sliding window based on the first and second results.

[0005] In one possible implementation, based on the first result and the second result, obtaining the feature value of each data unit within the second sliding window includes: if the central data unit within the first sliding window is similar to at least one data unit within the second sliding window, replacing the feature value of the central data unit with the feature value of any data unit among the at least one data unit; and / or if the central data unit within the first sliding window is similar to at least one surrounding data unit within the first sliding window, replacing the feature value of at least one surrounding data unit with the feature value of the central data unit.

[0006] In one possible implementation, if the central data unit in the first sliding window is similar to at least one data unit in the second sliding window, the feature value of the central data unit is replaced with the feature value of any data unit among the at least one data unit. This includes: if the central data unit in the first sliding window is similar to multiple data units in the second sliding window, the feature value of the central data unit is replaced with the feature value of the first data unit among the multiple data units, where the first data unit is the data unit with the highest similarity to the central data unit.

[0007] In one possible implementation, based on the first result and the second result, the feature value of each data unit in the second sliding window is obtained, including: if the central data unit in the first sliding window is similar to all the surrounding data units in the first sliding window and there are no data units in the second sliding window that are similar to the central data unit in the first sliding window, the feature value of any data unit in the first sliding window is calculated, and the feature values ​​of other data units in the first sliding window are replaced with the feature value of any data unit.

[0008] In one possible implementation, the method is applied to a Transformer model, which includes a query matrix Q, a key matrix K, and a value matrix V, with data units being Tokens. The method further includes: if at least one Token in the second frame video image is similar to a Token in the first frame video image, replacing the value corresponding to at least one Token in the query matrix Q, key matrix K, or value matrix V of the second frame video image with the value of the Token that is similar to at least one Token in the stored query matrix Q, key matrix K, or value matrix V of the first frame video image.

[0009] In one possible implementation, the method is applied to a Transformer model, which includes an attention matrix and data units of tokens. The method further includes: if at least two tokens in the second frame video image are similar to tokens in the first frame video image, replacing the correlation values ​​in the attention matrix of the second frame video image with any two of the at least two tokens with the correlation values ​​in the stored attention matrix of the first frame video image with any two tokens that are similar to the two tokens.

[0010] In one possible implementation, the similarity calculation method between data units includes at least one of the following methods: L1 norm, L2 norm, and cosine similarity.

[0011] In one possible implementation, n=1.

[0012] In a second aspect, a video processing apparatus is provided, comprising: a decomposition unit for decomposing a video segment into multiple video frames, the multiple video frames including adjacent first and second video frames, wherein the second frame data is located after the first frame data; a segmentation unit for segmenting each video frame in the multiple video frames into multiple data units; a determination unit for determining the similarity of each data unit in a second sliding window of a central data unit in a first sliding window of a second video frame, obtaining a first result, wherein the second sliding window is located at the same position as the first sliding window in the first video frame, the first and second sliding windows have the same size, and both are (2n+1)*(2n+1), where n is a positive integer, and the number of data units included in the sliding window is less than the number of data units included in a video frame; and determining the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window, obtaining a second result; and an acquisition unit for acquiring the feature value of each data unit in the first sliding window based on the first and second results.

[0013] Thirdly, a video processing apparatus is provided, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the video processing apparatus performs the method provided in the first aspect and any possible implementation thereof.

[0014] Based on the above technical solution, a first result is obtained by determining the similarity between the central data unit in the first sliding window of the subsequent video image and each data unit in the second sliding window at the same position in the previous video image. A second result is obtained by determining the similarity between the central data unit in the first sliding window of the subsequent video image and the surrounding data units in the first sliding window. Finally, based on the first and second results, the feature value of each data unit in the first sliding window of the subsequent video image is obtained. This is beneficial for significantly reducing the amount of computation by identifying and reusing the feature values ​​of similar data units. In addition, it is also beneficial for improving the real-time processing capability of the system in high frame rate and high resolution video processing. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0016] Figure 1 A schematic block diagram illustrating a video processing method according to an embodiment of this application is shown.

[0017] Figure 2 This document illustrates a spatial similarity comparison diagram based on an embodiment of this application.

[0018] Figure 3 This illustration shows a similarity comparison diagram in another spatial dimension of an embodiment of this application.

[0019] Figure 4 A schematic block diagram of a video processing apparatus according to an embodiment of this application is shown. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Deep learning, especially the Transformer model, has been widely used in video processing tasks due to its advantages such as strong learning ability, wide coverage, good portability, and automatic feature extraction. However, regardless of the deep learning model, extracting the feature value of each data unit in the video image is an essential step. However, as video resolution and frame rate increase, calculating each data unit in each frame of the video image consumes a large amount of computational resources, leading to slower processing speed and increased energy consumption. This computational method is clearly inefficient, thus requiring the search for optimization methods.

[0022] In view of this, embodiments of this application provide a video processing method. A first result is obtained by determining the similarity between the central data unit within a first sliding window in a subsequent video frame and each data unit within a second sliding window at the same position in a previous video frame. A second result is obtained by determining the similarity between the central data unit within the first sliding window in the subsequent video frame and the surrounding data units within the first sliding window. Finally, based on the first and second results, the feature value of each data unit within the first sliding window of the subsequent video frame is obtained. This method is beneficial for significantly reducing computational load by identifying and reusing the feature values ​​of similar data units. Furthermore, in high frame rate and high resolution video processing, it also helps improve the real-time processing capability of the system.

[0023] Figure 1 A schematic block diagram illustrating a video processing method provided in an embodiment of this application is shown. Figure 1 As shown, the method 100 includes some or all of the following contents.

[0024] S110, decompose the video sequence into multiple video images, the multiple video images include adjacent first and second video images, and the second frame image data is located after the first frame image data;

[0025] S120, divides each frame of a multi-frame video image into multiple data units;

[0026] S130, determine the similarity between the center data unit in the first sliding window in the second frame video image and each data unit in the second sliding window in the first frame video image, and obtain the first result. The second sliding window is in the same position as the first sliding window. The size of the first sliding window and the second sliding window is the same, and both are (2n+1)*(2n+1), where n is a positive integer. The number of data units included in the sliding window is less than the number of data units included in a frame video image.

[0027] S140, determine the similarity between the central data unit in the first sliding window and the surrounding data units in the first sliding window, and obtain the second result;

[0028] S150, based on the first result and the second result, obtain the feature value of each data unit within the first sliding window.

[0029] Optionally, the video processing method in this application embodiment can be executed by a processor, such as a general-purpose processor in the Transformer model or a hardware accelerator. A general-purpose processor can be, for example, a CPU or a GPU, while a hardware accelerator can be, for example, an FPGA or an ASIC. This application embodiment does not limit the executing entity of the video processing method.

[0030] It should be noted that method 100 may also include only one step of S130 and S140. That is, S130 and S140 do not need to be executed. The feature value of each data unit in the first sliding window may be obtained based only on the first result obtained by S130, or the feature value of each data unit in the first sliding window may be obtained based only on the second result obtained by S140. This application embodiment does not limit this.

[0031] In step S110, video is typically a sequence of continuously played frames, a dynamic image sequence. Image acquisition technology can be used to precisely extract each instantaneous scene, making it an independent frame. Independent frames allow for easier individual processing and analysis. Each frame can be considered a static image; that is, each video segment can be decomposed into multiple video frames using image acquisition technology.

[0032] In step S120, dividing each frame of video image into multiple representative data units can improve the accuracy and efficiency of subsequent image processing and analysis. These data units can be, for example, a token in a Transformer model or a basic data unit in other models. This embodiment does not limit the size of the data units; the main requirement is that they can refine the video image.

[0033] Assuming each video frame is divided into x data units, typically x = y * y, where y is a positive integer greater than 1, for two consecutive video frames, such as the i-th and (i+1)-th frames, a fixed-size sliding window is selected at the same position. The number of data units included in this sliding window should be less than the number of data units in a single video frame. In this embodiment, the size of the sliding window is (2n+1) * (2n+1), where n is a positive integer. Therefore, the sliding window in the (i+1)-th frame will contain a central data unit, and the other data units within the sliding window besides the central data unit can be referred to as surrounding data units.

[0034] In step 130, the center data unit in the sliding window of the (i+1)th frame video image can be compared with each data unit in the sliding window of the ith frame video image, and the result is recorded as the first result.

[0035] It should be noted that step 130 is a cyclical step. That is, after obtaining a first result, the sliding window can be moved, and the scheme of step 130 can be used again to obtain the next first result. Furthermore, for the same frame of video image, the data units in adjacent sliding windows need to overlap until the first result corresponding to each data unit in the (i+1)th frame of video image is obtained.

[0036] The following will combine Figure 2 and Figure 3 Step S130 is described in detail. In the diagram below, the data unit is the Token in the Transformer model.

[0037] like Figure 2 and Figure 3 As shown, the i-th frame and the (i+1)-th frame of video image are divided into 6*6=36 tokens, denoted as Token and Token respectively. i1 , ......, Token i36 and Token (i+1)1 , ......, Token (i+1)36，,A sliding window contains 3*3=9 tokens. For the first step S130, the specific position of the sliding window is as follows: Figure 2 As shown, in this Figure 2 In the (i+1)th frame of the video image, the center Token within the sliding window is the Token. (i+1)8 It compares the similarity of the token with each token in the sliding window of the i-th frame of the video image, that is, the token... (i+1)8 Each with Token i1 Token i2 Token i3 Token i7 Token i8 Token i9 Token i13 Token i14 Token i15 Perform a similarity comparison to obtain the Token. (i+1)8 The corresponding first results are shown in Table 1.

[0038]

[0039] Next, move the sliding window two columns to the right and left, so that the moved sliding window overlaps with the original sliding window by at least one column of tokens. This ensures that the number of calculations is reduced while also ensuring that the two sliding windows have overlapping portions, so that the calculations of the two sliding windows are not interrupted.

[0040] For the second step S130, the specific position of the moved sliding window is as follows: Figure 3 As shown. In this Figure 3 In the (i+1)th frame of the video image, the center Token within the sliding window is the Token. (i+1)10 It compares the similarity of the token with each token within the sliding window in the i-th frame of the video image, i.e., the token... (i+1)10 Each with Token i3 Token i4 Token i5 Token i9 Token i10 Token i11 Token i15 Token i16 Token i17 Perform a similarity comparison to obtain the Token. (i+1)10 The corresponding first results are shown in Table 2.

[0041]

[0042] It should be noted that the sliding window can also move downwards, repeating step S130 until each token in the (i+1)th frame of the video image is used as the center token to obtain a first result. For tokens located at the edge of the (i+1)th frame of the video image, the first result corresponding to the token at the edge position can be obtained by expanding the sliding window and comparing the similarity between the token at the edge position and the tokens falling within the expanded sliding window of the i-th frame of the video image.

[0043] In practical applications, the size and shape of the sliding window are adjusted according to the needs of the specific task to balance computational complexity and the sufficiency of information capture.

[0044] In step S140, the center data unit in the sliding window of the (i+1)th frame video image is compared with all the surrounding data units in the sliding window of the (i+1)th frame video image, and the result is recorded as the second result.

[0045] It should be noted that step 140 can also be a cyclical step. That is, after obtaining a second result, the sliding window can be moved, and the scheme of step 140 can be used again to obtain the next second result. In other words, the number of times step 140 is executed is related to the number of times the sliding window is moved; step 140 is executed once for each movement of the sliding window.

[0046] Specifically, such as Figure 2 As shown, for the first step S140, the center Token within the sliding window of the (i+1)th frame video image is Token. (i+1)8 It compares the similarity of the token with other surrounding tokens within the sliding window in the (i+1)th frame of the video image, that is, the token... (i+1)8 Each with Token (i+1)1 Token (i+1)2 Token (i+1)3 Token (i+1)7 Token (i+1)9 Token (i+1)13 Token (i+1)14 Token (i+1)15 Perform a similarity comparison to obtain the Token. (i+1)8 The corresponding second results are shown in Table 3.

[0047]

[0048] Next, as Figure 3As shown, for the second step S140, the center Token within the sliding window of the (i+1)th frame video image is Token. (i+1)10 It compares the similarity of the token with other surrounding tokens within the sliding window in the (i+1)th frame of the video image, that is, the token... (i+1)10 Each with Token (i+1)3 Token (i+1)4 Token (i+1)5 Token (i+1)9 Token (i+1)11 Token (i+1)15 Token (i+1)16 Token (i+1)17 Perform a similarity comparison to obtain the Token. (i+1)10 The corresponding second results are shown in Table 4.

[0049]

[0050] Optionally, in the embodiments of this application, step 130 can be understood as the process of obtaining the similarity between data units in the time dimension, while step 140 can be understood as the process of obtaining the similarity between data units in the spatial dimension.

[0051] Optionally, both steps 130 and 140 need to be performed, and based on the first result obtained in step 130 and the second result obtained in step 140, the feature values ​​of all tokens within the sliding window in the (i+1)th frame video image are determined.

[0052] Optionally, only one of steps 130 and 140 needs to be performed, and based on the first result obtained in step 130 or the second result obtained in step 140, the feature values ​​of all tokens within the sliding window in the (i+1)th frame video image are determined.

[0053] In this embodiment, various difference evaluation functions can be used to measure the similarity between data units. For example, the L2 norm (Euclidean distance) calculates the straight-line distance between the feature vectors of data units; the L1 norm (also known as Manhattan distance or absolute difference) evaluates the difference by summing the absolute values ​​of the differences in each dimension; and cosine similarity measures the directional similarity of two data units by calculating the cosine of the angle between their feature vectors, without considering the magnitude of the vectors. The selection of these difference evaluation functions depends on the specific application scenario and the consideration of data characteristics to ensure efficient identification of similar data units between frames. If the result calculated by any selected similarity evaluation metric (such as L2 norm, L1 norm, or cosine similarity) is lower than its preset threshold, then the two data units currently under consideration can be considered similar under that metric.

[0054] In one embodiment, step S150, namely, obtaining the feature value of each data unit within the first sliding window based on the first result and the second result, includes: if the central data unit within the first sliding window is similar to at least one data unit within the second sliding window, replacing the feature value of the central data unit with the feature value of any data unit among the at least one data unit; and / or, if the central data unit within the first sliding window is similar to at least one surrounding data unit within the first sliding window, replacing the feature value of at least one surrounding data unit with the feature value of the central data unit.

[0055] In the time dimension, using the center token within the sliding window of the (i+1)th frame of the video image as a reference target, a sliding window is constructed at the corresponding position in the i-th frame of the video image. Within the sliding window of the i-th frame, tokens similar to the reference token within the sliding window of the (i+1)th frame are searched one by one. If a similar token is found, its feature value can be directly reused, as the feature value of the similar token has already been calculated in the i-th frame. If no similar token is found, its feature value needs to be recalculated.

[0056] In the spatial dimension, taking the center token within the sliding window of the (i+1)th frame of the video image as the reference target, we search for tokens similar to the center token within the sliding window of the (i+1)th frame of the video image. If a similar token is found, we can obtain the feature values ​​of both the center token and the similar token by obtaining the feature value of either the center token or the similar token.

[0057] For example, if a token similar to the center token is found within the sliding window of the (i+1)th frame of the video image, and a similarity relationship has been established between the center token and the center token in the temporal dimension, then the feature value of the token similar to the center token within the sliding window of the i-th frame of the video image can be directly reused. If two tokens similar in the spatial dimension do not have a similarity relationship in the temporal dimension, then the feature value of the token within the sliding window of the i-th frame of the video image cannot be reused, but only the feature value of one of the tokens needs to be calculated.

[0058] In one embodiment, if the central data unit in the first sliding window is similar to at least one data unit in the second sliding window, the feature value of the central data unit is replaced with the feature value of any one of the at least one data units. This includes: if the central data unit in the first sliding window is similar to multiple data units in the second sliding window, the feature value of the central data unit is replaced with the feature value of the first data unit among the multiple data units, where the first data unit is the data unit in the second sliding window with the highest similarity to the central data unit.

[0059] In other words, when the similarity index between multiple tokens within the sliding window of the i-th video image and the central token within the sliding window of the (i+1)-th video image simultaneously meets the condition of being below a threshold, the token pair that is numerically closest is selected. Conversely, when the calculated similarity index between all tokens within the sliding window of the i-th video image and the central token within the sliding window of the (i+1)-th video image is above the threshold, we can determine that no token within the sliding window of the i-th video image is similar in the spatiotemporal dimension to the reference token within the sliding window of the (i+1)-th video image. This indicates that the video content or features at that location have undergone significant changes between frames.

[0060] In another embodiment, step S150, which is to obtain the feature value of each data unit in the first sliding window based on the first result and the second result, includes: if the central data unit in the first sliding window is similar to all the surrounding data units in the first sliding window and there are no data units in the second sliding window that are similar to the central data unit in the first sliding window, calculate the feature value of any data unit in the first sliding window, and replace the feature values ​​of other data units in the first sliding window with the feature value of any data unit.

[0061] by Figure 2 For example, suppose the center token within the sliding window of the (i+1)th frame of the video image... (i+1)8It is not similar to any of the tokens within the sliding window in the i-th frame of the video image, but the center token... (i+1)8 Compared with other Tokens within the sliding window of the (i+1)th frame video image, i.e., Token (i+1)1 Token (i+1)2 Token (i+1)3 Token (i+1)7 Token (i+1)9 Token (i+1)13 Token (i+1)14 Token (i+1)15 Since all tokens share similarities, the feature value of any token within the sliding window of the (i+1)th frame of the video image can be calculated, and the feature values ​​of all other tokens within the sliding window of the (i+1)th frame of the video image can be reused. That is, the feature values ​​of all tokens within the sliding window of the (i+1)th frame of the video image are the same.

[0062] Again Figure 2 For example, suppose the center token within the sliding window of the (i+1)th frame of the video image... (i+1)8 Token within the sliding window of the i-th frame of the video image i5 The most similar token pair, and the central token (i+1)8 Compared with other Tokens within the sliding window of the (i+1)th frame video image, i.e., Token (i+1)1 Token (i+1)2 Token (i+1)3 Token (i+1)7 Token (i+1)9 Token (i+1)13 Token (i+1)14 Token (i+1)15 If all tokens are similar, then all tokens within the sliding window of the (i+1)th frame of the video image can reuse the tokens within the sliding window of the i-th frame of the video image. i5 eigenvalues.

[0063] As mentioned above, this video processing method can be applied to the Transformer model, which includes a query matrix Q, a key matrix K, and a value matrix V, with data units being tokens. The method may further include: if at least one token in the second frame of the video image is similar to a token in the first frame of the video image, then replacing the value corresponding to at least one token in the query matrix Q, key matrix K, or value matrix V of the second frame of the video image with the value of the token in the stored query matrix Q, key matrix K, or value matrix V of the first frame of the video image that is similar to at least one token.

[0064] Specifically, for each frame of video image, it is necessary to obtain the corresponding query matrix Q, key matrix K, and value matrix V. The query matrix Q, key matrix K, and value matrix V are obtained as follows:

[0065] (1) Obtain the given input sequence Each of them This represents the feature value of a token in a video image.

[0066] (2) Perform three sets of linear transformations on the input sequence X to obtain the query matrix Q, the key matrix K, or the value matrix V. These are the trainable weight matrices used to calculate the query matrix Q, the key matrix K, or the value matrix V, respectively, and the calculation formulas are as follows:

[0067]

[0068] In other words, the query matrix Q, key matrix K, or value matrix V are linear transformations of the input sequence. That is, when the feature value of each token in a certain frame of video image is obtained, the query matrix Q, key matrix K, or value matrix V corresponding to the video image can be obtained through the above calculation formula.

[0069] After obtaining the feature value of each token in each frame of video image, it can be stored. At the same time, the corresponding query matrix Q, key matrix K, and value matrix V can also be stored. Once it is determined in the time dimension that a certain token in the current frame of video image is similar to a certain token in the previous frame of video image, the value related to that token in the query matrix Q, key matrix K, or value matrix V of the current frame of video image can reuse the value corresponding to the similar token in the query matrix Q, key matrix K, or value matrix V of the previous frame of video image. This is feature reuse.

[0070] For example, for the query matrix Q of the i-th video image, we have: ,in, This represents the feature value of the j-th token in the i-th frame of the video image. This represents the value of the j-th token in the query matrix Q of the i-th video frame. If it is determined that the m-th token in the (i+1)-th video frame is similar to the (m-1)-th token in the i-th video frame, then q can be directly stored. m-1 The q value corresponding to the m-th Token in the query matrix Q input to the (i+1)-th frame of the video image.

[0071] Similarly, the key matrix K and value matrix V are obtained in a similar way, which will not be elaborated on here.

[0072] In this embodiment of the application, the Transformer model further includes an attention matrix, and the method further includes: when there are at least two tokens in the second frame video image that are similar to the tokens in the first frame video image, replacing the correlation values ​​of any two tokens in the attention matrix of the second frame video image with the correlation values ​​of two tokens in the stored attention matrix of the first frame video image that are similar to the two tokens.

[0073] After obtaining the query matrix Q and the key matrix K, attention scores can be calculated, and the attention matrix can be obtained through a dot product. Specifically, the attention matrix can be obtained as follows: It is the dimension of the K vector, used to scale the dot product to avoid the value becoming too large.

[0074]

[0075]

[0076] Similar to the above embodiments, once the attention matrix of a certain frame of video image is obtained, it is stored immediately.

[0077] In other words, once it is determined in the time dimension that two tokens in the current frame of the video image are similar to two tokens in the previous frame of the video image, the product of the q-values ​​and k-values ​​corresponding to the two tokens in the attention matrix of the current frame of the video image can reuse the product of the q-values ​​and k-values ​​corresponding to the two similar tokens in the attention matrix V of the previous frame of the video image. This is called relation reuse.

[0078] For example, for the current frame of video image, the i-th token and the j-th token are similar to the i'-th token and the j'-th token of the previous frame of video image, respectively. When calculating the attention matrix of the current frame of video image, the product of the q-value of the i-th token and the k-value of the j-th token can be replaced with the q-value of the i'-th token and the k-value of the j'-th token, or the product of the k-value of the i-th token and the q-value of the j-th token can be replaced with the product of the k-value of the i'-th token and the q-value of the j'-th token.

[0079] Based on the same idea, embodiments of this application also provide a video processing apparatus. For example... Figure 4 As shown, the video processing apparatus 200 includes some or all of the following components.

[0080] The decomposition unit 210 is used to decompose a video segment into multiple video frames, the multiple video frames including adjacent first and second video frames, and the second frame image data is located after the first frame image data.

[0081] The partitioning unit 220 is used to divide each frame of a multi-frame video image into multiple data units.

[0082] The determining unit 230 is used to determine the similarity between the center data unit in the first sliding window of the second frame video image and each data unit in the second sliding window of the first frame video image, to obtain a first result. The second sliding window is located at the same position as the first sliding window. The first and second sliding windows have the same size, and both are (2n+1)*(2n+1), where n is a positive integer. The number of data units included in the sliding window is less than the number of data units included in a frame video image. The determining unit 230 is also used to determine the similarity between the center data unit in the first sliding window and the surrounding data units in the first sliding window, to obtain a second result.

[0083] The acquisition unit 240 is used to acquire the feature value of each data unit within the first sliding window based on the first result and the second result.

[0084] In one possible embodiment, the acquisition unit 240 is specifically configured to: if the central data unit in the first sliding window is similar to at least one data unit in the second sliding window, replace the feature value of the central data unit with the feature value of any one of the at least one data units; and / or if the central data unit in the first sliding window is similar to at least one surrounding data unit in the first sliding window, replace the feature value of at least one surrounding data unit with the feature value of the central data unit.

[0085] In one possible embodiment, the acquisition unit 240 is specifically used to: if the central data unit in the first sliding window is similar to multiple data units in the second sliding window, replace the feature value of the central data unit with the feature value of the first data unit among the multiple data units, wherein the first data unit is the data unit in the second sliding window that has the highest similarity to the central data unit.

[0086] In one possible embodiment, the acquisition unit 240 is specifically used to: if the central data unit in the first sliding window is similar to all the surrounding data units in the first sliding window and there are no data units in the second sliding window that are similar to the central data unit in the first sliding window, calculate the feature value of any data unit in the first sliding window, and replace the feature values ​​of other data units in the first sliding window with the feature value of any data unit.

[0087] In one possible embodiment, the video processing apparatus is applied to a Transformer model, which includes a query matrix Q, a key matrix K, and a value matrix V, with data units being Tokens. The apparatus further includes a replacement unit, configured to, when at least one Token in the second frame video image is similar to a Token in the first frame video image, replace the value corresponding to at least one Token in the query matrix Q, key matrix K, or value matrix V of the second frame video image with the value of a Token in the stored query matrix Q, key matrix K, or value matrix V of the first frame video image that is similar to at least one Token.

[0088] In one possible embodiment, the video processing apparatus is applied to a Transformer model, which includes an attention matrix and data units of tokens. The apparatus further includes a replacement unit, configured to replace the correlation values ​​of any two tokens in the attention matrix of the second frame video image with the correlation values ​​of two tokens in the attention matrix of the first frame video image that are similar to any two tokens when at least two tokens in the second frame video image are similar to the tokens in the first frame video image.

[0089] In one possible embodiment, the similarity calculation method between data units includes at least one of the following methods: L1 norm, L2 norm, and cosine similarity.

[0090] In one possible implementation, n=1.

[0091] Based on the same idea, this application also provides another video processing apparatus. This video processing apparatus includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the video processing apparatus performs the various method embodiments described above.

[0092] It should be noted that the details of the device-side embodiment can be found in the method-side embodiment, and for the sake of brevity, they will not be repeated here.

[0093] Although this application has been described with reference to preferred embodiments, various modifications can be made thereto and components can be replaced with equivalents without departing from the scope of this application. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no structural conflict. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A video processing method, characterized in that, include: A video segment is decomposed into multiple video frames, which include adjacent first and second video frames, with the second video frame located after the first video frame. Each frame of the multi-frame video image is divided into multiple data units; The similarity between the center data unit in the first sliding window of the second frame video image and each data unit in the second sliding window of the first frame video image is determined to obtain a first result. The second sliding window is in the same position as the first sliding window. The first sliding window and the second sliding window have the same size, and both are (2n+1)*(2n+1), where n is a positive integer. The number of data units included in the sliding window is less than the number of data units included in a frame of the video image. The similarity between the central data unit within the first sliding window and the surrounding data units within the first sliding window is determined to obtain a second result; Based on the first result and the second result, the feature value of each data unit in the first sliding window is obtained; The step of obtaining the feature value of each data unit within the first sliding window based on the first result and the second result includes: If the central data unit within the first sliding window is similar to all surrounding data units within the first sliding window, and there are no data units within the second sliding window that are similar to the central data unit within the first sliding window, calculate the feature value of any data unit within the first sliding window, and replace the feature values ​​of the other data units within the first sliding window with the feature value of that data unit. The method is applied to a Transformer model, which includes a query matrix Q, a key matrix K, and a value matrix V, wherein the data unit is a Token. The method further includes: If at least one token in the second frame video image is similar to a token in the first frame video image, the value corresponding to the at least one token in the query matrix Q, the key matrix K, or the value matrix V of the second frame video image is replaced with the value of the token that is similar to the at least one token in the query matrix Q, the key matrix K, or the value matrix V of the first frame video image.

2. The method according to claim 1, characterized in that, The step of obtaining the feature value of each data unit within the first sliding window based on the first result and the second result includes: If the central data unit in the first sliding window is similar to at least one data unit in the second sliding window, the feature value of the central data unit is replaced with the feature value of any one of the at least one data units; and / or If the central data unit in the first sliding window is similar to at least one surrounding data unit in the first sliding window, the feature value of the at least one surrounding data unit is replaced with the feature value of the central data unit.

3. The method according to claim 2, characterized in that, If the central data unit in the first sliding window is similar to at least one data unit in the second sliding window, the feature value of the central data unit is replaced with the feature value of any one of the at least one data units, including: If the central data unit in the first sliding window is similar to multiple data units in the second sliding window, the feature value of the central data unit is replaced with the feature value of the first data unit among the multiple data units, where the first data unit is the data unit in the second sliding window with the highest similarity to the central data unit.

4. The method according to claim 1, characterized in that, The Transformer model includes an attention matrix, and the method further includes: If at least two tokens in the second frame of the video image are similar to the tokens in the first frame of the video image, the correlation values ​​in the attention matrix of the second frame of the video image that are similar to any two of the at least two tokens are replaced with the correlation values ​​in the attention matrix of the first frame of the video image that are similar to the two tokens.

5. The method according to claim 1, characterized in that, The similarity calculation method between the data units includes at least one of the following methods: L1 norm, L2 norm, and cosine similarity.

6. The method according to claim 1, characterized in that, n=1。 7. A video processing apparatus, characterized in that, include: A decomposition unit is used to decompose a video segment into multiple video frames, wherein the multiple video frames include adjacent first and second video frames, and the second video frame is located after the first video frame. A partitioning unit is used to divide each frame of the multi-frame video image into multiple data units; A determining unit is configured to determine the similarity between the center data unit within a first sliding window in the second frame video image and each data unit within a second sliding window in the first frame video image, to obtain a first result. The second sliding window and the first sliding window are located at the same position. The first sliding window and the second sliding window have the same size, and both are (2n+1)*(2n+1), where n is a positive integer. The number of data units included in the sliding window is less than the number of data units included in a single frame of the video image. The unit also determines the similarity between the center data unit within the first sliding window and the surrounding data units within the first sliding window, to obtain a second result. The acquisition unit is configured to acquire the feature value of each data unit within the first sliding window based on the first result and the second result. The acquisition unit is specifically used for: If the central data unit within the first sliding window is similar to all surrounding data units within the first sliding window, and there are no data units within the second sliding window that are similar to the central data unit within the first sliding window, calculate the feature value of any data unit within the first sliding window, and replace the feature values ​​of the other data units within the first sliding window with the feature value of that data unit. The video processing device is applied to a Transformer model, which includes a query matrix Q, a key matrix K, and a value matrix V. The data unit is a Token. The video processing device also includes: The replacement unit is configured to, when there is at least one token in the second frame video image that is similar to a token in the first frame video image, replace the value corresponding to the at least one token in the query matrix Q, the key matrix K, or the value matrix V of the second frame video image with the value of a token in the stored query matrix Q, the key matrix K, or the value matrix V of the first frame video image that is similar to the at least one token.

8. A video processing apparatus, characterized in that, It includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the video processing apparatus causes the video processing apparatus to perform the method of any one of claims 1 to 6.

Images