Video block processing method and device, neural network training method and storage medium

Abstract

A frequency block processing method comprises: (S110) obtaining an input video block, the input video block comprising a plurality of video frames arranged in time sequence; (S120) obtaining initial feature video blocks of N levels arranged from high to low in resolution based on the input video block, N being a positive integer and N>2; (S130) performing cyclic scaling processing on the initial feature video block of the first level based on the initial feature video blocks of the second to Nth levels to obtain an intermediate feature video block, the resolution of the intermediate feature video block being the same as that of the input video block; and (S140) performing synthesis processing on the intermediate feature video block to obtain an output video block, the resolution of the output video block being the same as that of the input video block. In addition, a neural network training method, a video block processing device (470), a computer device (500) and a storage medium (600) are also disclosed.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a video block processing method, a neural network training method, a neural network processor, a video block processing device, a computer device, and a storage medium. Background Technology

[0002] In related technologies, deep learning techniques based on artificial neural networks have made significant progress in fields such as image classification, image capture and search, facial recognition, age and speech recognition. Neural networks can enhance digital images and generate realistic images by processing multi-scale information. However, processing each frame as a single image independently for consecutive video frames leads to quality degradation and flickering artifacts. Summary of the Invention

[0003] This application discloses a video block processing method, a neural network training method, a neural network processor, a video block processing device, a computer device, and a storage medium.

[0004] The video block processing method provided in this application includes: acquiring an input video block, the input video block comprising multiple video frames arranged in chronological order; based on the input video block, obtaining N initial feature video blocks arranged in descending order of resolution, where N is a positive integer and N>2; based on the initial feature video blocks of the 2nd to Nth levels, performing cyclic scaling processing on the initial feature video block of the 1st level to obtain intermediate feature video blocks, the resolution of the intermediate feature video blocks being the same as the resolution of the input video block; and performing synthesis processing on the intermediate feature video blocks to obtain an output video block, the resolution of the output video block being the same as the resolution of the input video block; wherein, the cyclic scaling processing includes: nested scaling processing of N-1 levels, each level of scaling processing including downsampling processing, concatenation processing, upsampling processing, and residual concatenation addition processing; The downsampling process of level i is based on the input of the scaling process of level i to obtain the downsampled output of level i. The concatenation process of level i is based on the downsampled output of level i and the initial feature video block of level i+1 to obtain the joint output of level i. The upsampling process of level i is based on the joint output of level i to obtain the upsampled output of level i. The residual concatenation and addition process of level i adds the input of the scaling process of level i and the upsampled output of level i to obtain the output of the scaling process of level i. Here, i = 1, 2, ..., N-1. The scaling process of level j+1 is nested between the downsampling process of level j and the concatenation process of level j. The output of the downsampling process of level j is used as the input of the scaling process of level j+1. Here, j = 1, 2, ..., N-2.

[0005] In some implementations, each of the downsampling processes and each of the upsampling processes uses different parameters.

[0006] In some implementations, the concatenation process of the i-th level is based on concatenating the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level, including: using the downsampled output of the i-th level as the input of the scaling process of the (i+1)-th level to obtain the output of the scaling process of the (i+1)-th level; and concatenating the output of the scaling process of the (i+1)-th level with the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level.

[0007] In some implementations, at least one level of scaling is executed multiple times consecutively, with the output of the previous scaling process serving as the input for the next scaling process.

[0008] In some implementations, the initial feature video block of the first level has the highest resolution, and the resolution of the initial feature video block of the first level is the same as the resolution of the input video block.

[0009] In some implementations, the resolution of the initial feature video block of the preceding level is an integer multiple of the resolution of the initial feature video block of the following level.

[0010] In some implementations, obtaining N levels of initial feature video blocks arranged from high to low resolution based on the input video block includes: concatenating the input video block with a random noise video block to obtain a joint input video block; and performing N different levels of analysis processing on the joint input video block to obtain the N levels of initial feature video blocks arranged from high to low resolution respectively.

[0011] In some embodiments, obtaining the input video block includes: obtaining an original input video block with a first resolution; and performing a resolution conversion process on the original input video block to obtain the input video block with a second resolution, wherein the second resolution is greater than the first resolution.

[0012] In some implementations, the resolution conversion process is performed using one of the bicubic interpolation algorithm, the bilinear interpolation algorithm, and the Lanczos interpolation algorithm.

[0013] In some implementations, the video block method includes: cropping the input video block to obtain a plurality of sub-input video blocks having overlapping regions;

[0014] The specific steps of obtaining N initial feature video blocks of varying resolutions from high to low based on the input video blocks include: obtaining N initial feature video blocks of varying resolutions from high to low based on each sub-input video block, where N is a positive integer and N>2;

[0015] The process of performing cyclic scaling on the initial feature video block of level 1 based on the initial feature video block of levels 2 to N to obtain intermediate feature video blocks specifically includes: performing cyclic scaling on the sub-initial feature video block of level 1 based on the sub-initial feature video block of levels 2 to N to obtain sub-intermediate feature video blocks, wherein the resolution of the sub-intermediate feature video block is the same as the resolution of the sub-input video block;

[0016] The process of synthesizing the intermediate feature video blocks to obtain an output video block specifically includes: synthesizing the sub-intermediate feature video blocks to obtain corresponding sub-output video blocks, wherein the resolution of the sub-output video blocks is the same as the resolution of the sub-input video blocks; and concatenating the sub-output video blocks corresponding to the plurality of sub-input video blocks into the output video block.

[0017] In some implementations, the relative positions of the sub-output video block pixels in the output video block are the same as the relative positions of the corresponding sub-input video block pixels in the input video block. The step of concatenating the sub-output video blocks corresponding to the plurality of sub-input video blocks into the output video block includes: initializing an output video matrix and a counting matrix, wherein the resolution of both the output video matrix and the counting matrix is ​​the same as the resolution of the output video block; using a window function to add the pixel values ​​of the plurality of sub-output video blocks to their corresponding positions in the output video matrix; each time a pixel value is added to the initial output video matrix, adding a floating-point number equal to the value of the window function to the corresponding element value in the counting matrix; and processing the corresponding elements of the output video matrix and the counting matrix to generate the output video block.

[0018] The neural network training method provided in this application includes an analysis network, a cyclic scaling network, and a synthesis network. The training method includes: acquiring a first training input video block, the first training input video block comprising multiple video frames arranged in chronological order; processing the first training input video block using the analysis network to obtain N levels of initial training feature video blocks arranged from high to low resolution, where N is a positive integer and N>2; using the cyclic scaling network to perform cyclic scaling processing on the first level of the initial training feature video block based on the second to N levels of the initial training feature video block to obtain intermediate training feature video blocks, the resolution of which is the same as that of the first training input video block; and synthesizing the intermediate training feature video blocks using the synthesis network to obtain a first training output video block, the resolution of which is the same as that of the first training input video block; calculating the loss value of the neural network based on the first training output video block using a loss function; and calculating the loss value of the neural network based on the loss value of the neural network. The parameters of the neural network are corrected; wherein, the cyclic scaling process includes: N-1 nested scaling processes, each level of scaling process includes downsampling, concatenation, upsampling and residual concatenation; the downsampling process of the i-th level is based on the input of the scaling process of the i-th level to obtain the downsampled output of the i-th level, the concatenation process of the i-th level is based on the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level, and the upsampling process of the i-th level... The upsampled output of the i-th level is obtained based on the joint output of the i-th level. The residual linking and addition processing of the i-th level performs residual linking and addition of the input of the scaling processing of the i-th level and the upsampled output of the i-th level to obtain the output of the scaling processing of the i-th level, where i = 1, 2, ..., N-1; the scaling processing of the (j+1)-th level is nested between the downsampling processing of the j-th level and the linking processing of the j-th level, and the output of the downsampling processing of the j-th level is used as the input of the scaling processing of the (j+1)-th level, where j = 1, 2, ..., N-2.

[0019] In some embodiments, processing the first training input video block using the analysis network to obtain N levels of initial training feature video blocks arranged from high to low resolution includes: concatenating the first training input video block with a random noise video block to obtain a joint training input video block; and performing N different levels of analysis processing on the joint training input video block using the analysis network to obtain the N levels of initial training feature video blocks arranged from high to low resolution respectively.

[0020] In some implementations, calculating the loss value of the neural network based on the first training output video block using a loss function includes: processing the first training output video block using a discriminant network, and calculating the loss value of the neural network based on the output of the discriminant network corresponding to the first training output video block.

[0021] In some embodiments, the discriminant network includes: M-1 level downsampling subnetworks, M level discriminant branch networks, a synthesis subnetwork, and an activation layer; the M-1 level downsampling subnetworks are used to perform downsampling processing on the input of the discriminant network at different levels to obtain the output of the M-1 level downsampling subnetworks; the input of the discriminant network and the output of the M-1 level downsampling subnetworks respectively correspond to the input of the M level discriminant branch networks; each level discriminant branch network includes a brightness processing subnetwork, a first convolutional subnetwork, a second convolutional subnetwork, and a third convolutional subnetwork connected in sequence; the second layer in the t-th level discriminant branch network... The output of the convolutional subnetwork is concatenated with the output of the first convolutional subnetwork in the (t+1)th level discriminative branch network and used as the input of the second convolutional subnetwork in the (t+1)th level discriminative branch network, where t = 1, 2, ..., M-2; the output of the second convolutional subnetwork in the (M-1)th level discriminative branch network is concatenated with the output of the first convolutional subnetwork in the Mth level discriminative branch network and used as the input of the third convolutional subnetwork; the synthesis subnetwork is used to synthesize the output of the third convolutional subnetwork to obtain a discriminative output video block; the activation layer is used to process the discriminative output video block to obtain a value characterizing the quality of the input of the discriminative network.

[0022] In some implementations, the loss function is expressed as:

[0023]

[0024] Where L(Y, X) represents the loss function, and Y represents the first training output video block (including Y). n =1 and Y n =0), Y represents the generation loss function. n＝1 Y represents the first training output video block obtained when the noise amplitude of the random noise video block is not zero. n＝0 L represents the first training output video block obtained when the noise amplitude of the random video block is 0. L1 S represents the contrastive loss function. f L represents the downsampling of the bicubic difference of factor f (spatiotemporal factor 1*f*f), contextual This represents the content loss function. λ1, λ2, λ3, λ4, and λ5 represent the preset weights;

[0025] The generation loss function can be expressed as:

[0026]

[0027]

[0028] fake={Y n＝1 S2(Y n＝1 S4(Y) n＝1 ), S8(Y n＝1 )},

[0029] real={X, S2(X), S4(X), S8(X)};

[0030] The content loss function can be expressed as:

[0031]

[0032] Where S1 is a constant, F ij P represents the value at the j-th position in the first content feature block of the first training output video block extracted by the i-th convolutional kernel in content feature extraction. ij This represents the value at the j-th position in the second content feature block of the first training standard video block extracted by the i-th convolutional kernel in the content feature extraction.

[0033] In some implementations, the contrastive loss function can be:

[0034]

[0035] Where Loss(Y, y) represents the loss function, ⊙ represents the pixel product, and |||| represents the pixel loss function. or

[0036] In some embodiments, the discriminant network is trained based on the neural network; and the training process of the discriminant network and the training process of the neural network are performed alternately to obtain a trained neural network; wherein, training the discriminant network based on the neural network includes: acquiring a second training input video block; processing the second training input video block using the neural network to obtain a second training output video block; calculating a discriminant loss value based on the second training output video block using a discriminant loss function; and correcting the parameters of the discriminant network according to the discriminant loss value.

[0037] The neural network processor provided in this application includes an analysis circuit, a cyclic scaling circuit, and a synthesis circuit. The analysis circuit is configured to obtain N initial feature video blocks of varying resolutions from high to low based on the input video blocks, where N is a positive integer and N>2. The cyclic scaling circuit is configured to perform cyclic scaling on the initial feature video blocks of the first level based on the initial feature video blocks of the second to Nth levels to obtain intermediate feature video blocks, the resolution of which is the same as that of the input video blocks. The synthesis circuit is configured to synthesize the intermediate feature video blocks to obtain an output video block, the resolution of which is the same as that of the input video blocks. The cyclic scaling circuit includes N-1 nested scaling circuits, each level of which includes a downsampling circuit, a connection circuit, an upsampling circuit, and a residual chain. The circuit consists of an adder circuit; the downsampling circuit of the i-th level downsamples the input of the scaling circuit of the i-th level to obtain the downsampled output of the i-th level; the connection circuit of the i-th level connects the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level; the upsampling circuit of the i-th level obtains the upsampled output of the i-th level based on the joint output of the i-th level; and the residual link adder circuit of the i-th level performs residual linking and addition of the input of the scaling circuit of the i-th level and the upsampled output of the i-th level to obtain the output of the scaling circuit of the i-th level, where i = 1, 2, ..., N-1; the scaling circuit of the (j+1)-th level is nested between the downsampling circuit of the j-th level and the connection circuit of the j-th level, and the output of the downsampling circuit of the j-th level is used as the input of the scaling circuit of the (j+1)-th level, where j = 1, 2, ..., N-2.

[0038] The video block processing apparatus provided in this application includes an acquisition module and a processing module. The acquisition module is configured to acquire an input video block, which includes multiple video frames arranged in chronological order. The processing module is configured to obtain N initial feature video blocks of varying resolutions from high to low based on the input video block, where N is a positive integer and N>2; perform cyclic scaling on the initial feature video block of the first level based on the initial feature video blocks of the second to Nth levels to obtain intermediate feature video blocks; and perform synthesis processing on the intermediate feature video blocks to obtain an output video block, wherein the resolution of the intermediate feature video block is the same as the resolution of the input video block, and the resolution of the output video block is the same as the resolution of the input video block; wherein the cyclic scaling process includes: nested scaling processes of N-1 levels, each level of scaling process including downsampling, concatenation, and upsampling. The process involves sampling and residual linking. The downsampling process at level i is based on the input of the scaling process at level i, yielding the downsampled output. The linking process at level i concatenates the downsampled output of level i with the initial feature video block of level i+1, yielding the joint output of level i. The upsampling process at level i concatenates the joint output of level i, yielding the upsampled output of level i. The residual linking process at level i concatenates the input of the scaling process at level i with the upsampled output of level i, yielding the output of the scaling process at level i. Here, i = 1, 2, ..., N-1. The scaling process at level j+1 is nested between the downsampling process at level j and the linking process at level j, with the output of the downsampling process at level j serving as the input of the scaling process at level j+1. Here, j = 1, 2, ..., N-2.

[0039] The computer device provided in this application includes a processor and a memory. The memory stores computer-readable instructions, and the processor is used to run the computer-readable instructions. When the computer-readable instructions are run by the processor, they execute the video block processing method described in any embodiment of this application, or execute the neural network training method described in any embodiment of this application.

[0040] The storage medium provided in the embodiments of this application stores computer-readable instructions, wherein when the computer-readable instructions are executed by a computer, the video block processing method described in any embodiment of this application or the neural network training method described in any embodiment of this application can be executed. Attached Figure Description

[0041] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, wherein:

[0042] Figure 1 This is a schematic diagram of a convolutional neural network according to an embodiment of this application.

[0043] Figure 2 This is a schematic diagram of the structure of the video block according to an embodiment of this application.

[0044] Figure 3 This is a schematic diagram of the processing flow of the 3D rear projection block according to an embodiment of this application.

[0045] Figure 4 This is a flowchart illustrating the video block processing method according to an embodiment of this application.

[0046] Figure 5 This is the corresponding embodiment of the present application. Figure 4 The diagram shows a schematic flowchart of the video block processing method.

[0047] Figure 6 This is the corresponding embodiment of the present application. Figure 4 Another schematic flowchart of the video block processing method shown.

[0048] Figure 7 This is another flowchart illustrating the video block processing method according to an embodiment of this application.

[0049] Figure 8 This is another flowchart illustrating the video block processing method according to the embodiments of this application.

[0050] Figure 9 This is another schematic flowchart of the video block processing method according to the embodiments of this application.

[0051] Figure 10 This is a schematic diagram of video block trimming and splicing processing according to an embodiment of this application.

[0052] Figure 11 This is another schematic flowchart of the video block processing method according to the embodiments of this application.

[0053] Figure 12 This is a schematic diagram illustrating how multiple sub-output video blocks are stitched together into an output video block according to an embodiment of this application.

[0054] Figure 13 This is a schematic block diagram of the neural network structure according to an embodiment of this application.

[0055] Figure 14 This is a flowchart illustrating the neural network training method according to an embodiment of this application.

[0056] Figure 15 This is a schematic block diagram of the training method for training a neural network according to the embodiments of this application.

[0057] Figure 16 This is a schematic diagram of the discrimination network in an embodiment of this application.

[0058] Figure 17 This is a schematic diagram of the generative adversarial training process according to the embodiments of this application.

[0059] Figure 18 This is a schematic diagram of the training process of the discrimination network in the embodiments of this application.

[0060] Figure 19 This is a schematic block diagram of the generative adversarial training architecture of the embodiments of this application.

[0061] Figure 20 This is a schematic block diagram of the neural network training method of the present application for training a discriminant network.

[0062] Figure 21 This is a schematic block diagram of a neural network processor according to an embodiment of this application.

[0063] Figure 22 This is a schematic block diagram of a video block processing apparatus according to an embodiment of this application.

[0064] Figure 23 This is a schematic block diagram of a computer device according to an embodiment of this application.

[0065] Figure 24 This is a schematic block diagram of the storage medium according to an embodiment of this application. Detailed Implementation

[0066] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0067] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0068] The following disclosure provides many different implementations or examples for carrying out different structures of this application. To simplify the disclosure, the steps and setups of specific examples are described below. Of course, these are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or setups discussed. In addition, various specific examples of processes and materials are provided in this application, but those skilled in the art will recognize the application of other processes and / or the use of other materials.

[0069] Image enhancement is a hot research topic in the field of image processing. Due to various physical limitations during image acquisition (e.g., the small size of image sensors in mobile phone cameras and other software and hardware limitations) and interference from environmental noise, image quality can be significantly degraded. The purpose of image enhancement is to improve the grayscale histogram and contrast of an image through image enhancement techniques, thereby highlighting image details and improving the visual effect of the image.

[0070] Initially, Convolutional Neural Networks (CNNs) were primarily used to recognize two-dimensional shapes, exhibiting high invariance to image translation, scaling, tilting, or other forms of deformation. CNNs mainly simplify the complexity of neural networks and reduce the number of weights through local receptive fields and weight sharing. With the development of deep learning technology, the application scope of CNNs has expanded beyond image recognition to include face recognition, text recognition, animal classification, image processing, and other fields.

[0071] Figure 1 A schematic diagram of a convolutional neural network is shown. For example, this convolutional neural network can be used for image processing, using images as input and output, and replacing scalar weights with convolutional kernels. Figure 1 Only a convolutional neural network with a 3-layer structure is shown in the example; the embodiments of this application are not limited thereto. Figure 1 As shown, the convolutional neural network includes an input layer 101, a hidden layer 102, and an output layer 103. The input layer 101 has 4 inputs, the hidden layer 102 has 3 outputs, and the output layer 103 has 2 outputs. Finally, the convolutional neural network outputs 2 images.

[0072] In some implementations, the four inputs to the input layer 101 can be four images, or four feature images of one image. The three outputs of the hidden layer 102 can be feature images of the images input through the input layer 101.

[0073] For example, such as Figure 1As shown, the convolutional layer has weights. and bias Weight Represents the convolution kernel, bias This is a scalar value superimposed on the output of the convolutional layer, where k represents the label of input layer 101, and i and j are the labels of the units in input layer 101 and hidden layer 102, respectively. For example, the first convolutional layer 201 includes a first set of convolutional kernels ( Figure 1 In ) and the first set of biases ( Figure 1 In The second convolutional layer 202 includes a second set of convolutional kernels (). Figure 1 In ) and the second set of biases ( Figure 1 In Typically, each convolutional layer consists of dozens or hundreds of convolutional kernels. If the convolutional neural network is a deep convolutional neural network, it can include at least five convolutional layers.

[0074] Furthermore, such as Figure 1 As shown, the convolutional neural network also includes a first activation layer 203 and a second activation layer 204. The first activation layer 203 is located after the first convolutional layer 201, and the second activation layer 204 is located after the second convolutional layer 202. The activation layers (e.g., the first activation layer 203 and the second activation layer 204) include activation functions that introduce nonlinearity into the convolutional neural network, enabling it to better solve more complex problems. Activation functions may include ReLU functions, sigmoid functions, or tanh functions, etc. The ReLU function is a non-saturating nonlinear function, while the sigmoid and tanh functions are saturating nonlinear functions. For example, the activation layer can be a standalone layer of the convolutional neural network, or it can be included within a convolutional layer (e.g., the first convolutional layer 201 may include the first activation layer 203, and the second convolutional layer 202 may include the second activation layer 204).

[0075] For example, in the first convolutional layer 201, firstly, several convolutional kernels from the first set of convolutional kernels are applied to each input. and several biases in the first set of biases The output of the first convolutional layer 201 is obtained; then, the output of the first convolutional layer 201 can be processed by the first activation layer 203 to obtain the output of the first activation layer 203. In the second convolutional layer 202, firstly, several convolutional kernels from the second set of convolutional kernels are applied to the output of the first activation layer 203. Several biases in the second set of biases The output of the second convolutional layer 202 is obtained; then, the output of the second convolutional layer 202 can be processed by the second activation layer 204 to obtain the output of the second activation layer 204. For example, the output of the first convolutional layer 201 can be obtained by applying a convolution kernel to its input. Then with bias The sum of the results means that the output of the second convolutional layer 202 can be used to apply a convolution kernel to the output of the first activation layer 203. Then with bias The result of adding them together.

[0076] Before using a convolutional neural network (CNN) for image processing, the CNN needs to be trained. After training, the convolutional kernels and biases of the CNN remain unchanged during image processing. During training, each convolutional kernel and bias is adjusted using multiple sets of input / output example images and optimization algorithms to obtain an optimized CNN.

[0077] Traditional image processing CNNs operate on two-dimensional data (images) using fully convolutional systems. Figure 1 In the system, the input and output sizes of the features can be arbitrary. Figure 1 The system is fully convolutional, meaning it works with any image width (W) and image height (H) for 4 inputs and 2 outputs. This is because convolutional layers operate regardless of the data dimension, and activation layers operate at the pixel level.

[0078] For video data, please refer to [link / reference]. Figure 2 At a specific time value T of a video block, we can obtain a static image, called a video frame. A video block can include multiple video frames arranged in chronological order. Furthermore, at a specific image height H (or image width W) value, multiple consecutive video frames within a video block can also produce an image, called a video profile. Here, the video frame represents the spatial dimension, and the video profile represents the temporal dimension. It can be understood that 3D convolution can be applied to any values ​​of H, W, and T; similarly, extending it to a fully 3D convolutional network can also be applied to any values ​​of H, W, and T.

[0079] In neural networks, convolution operations process local regions of images (2D) or videos (3D), thus exhibiting boundary effects. For example, a 2D convolution with a kernel size of 3x3 will reduce the output resolution by one pixel at the top, bottom, left, and right boundaries. Similarly, a 3D convolution with a kernel size of 3x3x3 will reduce the temporal resolution by one frame at the beginning and end of the input video stream. Therefore, it is necessary to combine stride convolution and transposed convolution, one consuming boundary pixels and the other generating boundary pixels, so that the output resolution remains unchanged from the input resolution.

[0080] exist Figure 3 The illustrated embodiment shows a 3D back-projection block processing flow. The downward and upward operations in the figure represent stride convolutional layers and transposed convolutional layers. By first reducing and then increasing the resolution of the 3D video stream, the resolution of the generated and output streams can be made the same as that of the input video stream.

[0081] Please see Figure 4 The video block processing method of this application includes:

[0082] Step S110: Obtain the input video block, which includes multiple video frames arranged in chronological order;

[0083] Step S120: Based on the input video blocks, obtain N initial feature video blocks arranged from high to low resolution, where N is a positive integer and N>2;

[0084] Step S130: Based on the initial feature video blocks of levels 2 to N, perform cyclic scaling on the initial feature video blocks of level 1 to obtain intermediate feature video blocks. The resolution of the intermediate feature video blocks is the same as the resolution of the input video blocks; and

[0085] Step S140: Perform synthesis processing on the intermediate feature video blocks to obtain an output video block. The resolution of the output video block is the same as that of the input video block.

[0086] The cyclic scaling process includes: N-1 levels of nested scaling processes, each level of which includes downsampling, concatenation, upsampling and residual concatenation.

[0087] The downsampling process of level i is based on the input of the scaling process of level i to obtain the downsampled output of level i. The concatenation process of level i is based on the downsampled output of level i and the initial feature video block of level i+1 to obtain the joint output of level i. The upsampling process of level i is based on the joint output of level i to obtain the upsampled output of level i. The residual concatenation and addition process of level i adds the input of the scaling process of level i and the upsampled output of level i to obtain the output of the scaling process of level i. Where i = 1, 2, ..., N-1;

[0088] The scaling process at level (j+1) is nested between the downsampling process at level (j) and the connection process at level (j). The output of the downsampling process at level (j) is used as the input of the scaling process at level (j+1), where j = 1, 2, ..., N-2.

[0089] In the video block processing method of this application, a large number of frames are used to train the neural network by acquiring video blocks. By combining multiple video frames set in time sequence with multiple dimensions to process the video blocks, the temporal correlation can be effectively learned. Moreover, based on the input video blocks, multiple initial feature video blocks of different resolutions are obtained. By combining these initial feature video blocks of different resolutions and performing cyclic scaling processing on the initial feature video block with the highest resolution, higher video fidelity can be obtained and the quality of the output video blocks can be greatly improved.

[0090] Specifically, in step S110, the input video block can be as follows: Figure 5 and Figure 6 The input video block shown in the INPUT can include video data captured by a smartphone camera, tablet camera, personal computer camera, digital camera lens, surveillance camera or webcam, etc., which can include videos of people, animals and plants or landscapes, etc., and the embodiments of this application do not limit this.

[0091] The input video block INPUT can be a grayscale video block or a color video block. The color video block includes, but is not limited to, RGB data with three channels. It should be noted that, in the embodiments of this application, when the input video block INPUT is a grayscale video block, the output video block OUTPUT is also a grayscale video block; when the input video block INPUT is a color video block, the output video block OUTPUT is also a color video block.

[0092] In step S120, as Figure 5 As shown, the input video block INPUT can be analyzed at N different levels using the ANALYSIS network to obtain initial feature video blocks F01 to F0N at N levels arranged from high to low resolution (e.g., Figure 5 (as shown in F01~F05). In Figure 5 In the illustrated embodiment, the analysis network includes N analysis sub-networks ASN, each of which is used to perform the above-described analysis processing at different levels to obtain initial feature video blocks F01 to F0N at N levels arranged from high to low resolution (e.g., Figure 5 (As shown in F01 to F05). Each analysis sub-network ASN can be implemented as a convolutional network module, including convolutional neural networks (CNNs), residual networks (ResNets), dense networks (DenseNets), etc. It is understood that each analysis sub-network ASN may include convolutional layers, downsampling layers, normalization layers, etc., but is not limited to these.

[0093] In this embodiment, the resolution of a video block obtained by the neural network through different processing of an input video block can refer to the resolution of multiple video frames representing the spatial dimension. That is, among the N levels of initial video blocks obtained based on the input video block, arranged from highest to lowest resolution, the resolution of the video frames corresponding to the N levels of initial video blocks is arranged from highest to lowest. Furthermore, the number of video frames in a video block obtained by the neural network through different processing of the input video block can be the same as that of the input video block. In other words, the temporal resolution representing the time dimension of the video block can remain unchanged during the neural network processing, and the number of video frames in the feature video blocks at each level is the same.

[0094] It should be noted that the resolution of each video frame in a video block can be the same or different. Since the number of video frames in the feature video blocks of each level is the same, the resolution of the corresponding video frames in the N initial video blocks of the levels obtained based on the input video blocks are arranged from high to low.

[0095] exist Figure 5 and Figure 6 In this system, the order of each level is determined from top to bottom.

[0096] In some implementations, the initial feature video block of the first level has the highest resolution, and the resolution of the initial feature video block of the first level is the same as that of the input video block.

[0097] Thus, the input video block can be obtained by performing resolution conversion processing (e.g., super-resolution reconstruction processing) on ​​the original input video block. In this case, the resolution of the initial feature video block of the Nth level with the lowest resolution can be the same as the resolution of the original input video block. It should be noted that the embodiments of this application include, but are not limited to, this.

[0098] In some implementations, the resolution of the initial feature video patch of the preceding level (e.g., the i-th level) is an integer multiple of the resolution of the initial feature video patch of the following level (e.g., the (i+1)-th level), such as 2x, 3x, 4x, ..., etc.

[0099] It should be noted that, although Figure 5 and Figure 6 The examples show the case of obtaining initial feature video blocks F01 to F05 at 5 levels (i.e., N=5), but this should not be regarded as a limitation of this application. That is, the value of N can be set according to actual needs.

[0100] In step S130, the cyclic scaling process includes: N-1 levels of nested scaling processes, each level of scaling process including the sequential execution of downsampling process DOWNSCALE, concatenation process CONCATENATE, upsampling process UPSSCALE, and residual concatenation and addition processes.

[0101] Specifically, in some embodiments, the downsampling process of the i-th level is based on the input of the scaling process of the i-th level to obtain the downsampled output of the i-th level; the concatenation process of the i-th level is based on the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level; the upsampling process of the i-th level is based on the joint output of the i-th level to obtain the upsampled output of the i-th level; and the residual linking and addition process of the i-th level adds the input of the scaling process of the i-th level and the upsampled output of the i-th level through residual linking to obtain the output of the scaling process of the i-th level, where i = 1, 2, ..., N-1.

[0102] In some implementations, the concatenation process of the i-th level is based on concatenating the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level, including: using the downsampled output of the i-th level as the input of the scaling process of the (i+1)-th level to obtain the output of the scaling process of the (i+1)-th level; and concatenating the output of the scaling process of the (i+1)-th level with the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level.

[0103] Downsampling is used to reduce the size of individual video frames in a feature video block, thereby reducing the amount of data in the feature video block. This can be achieved through downsampling layers, but is not limited to these methods. In some embodiments, the downsampling layer can employ downsampling methods such as maxpooling, average pooling, strided convolution, decimation (e.g., selecting a fixed number of pixels), and demuxout (splitting the input video frame into multiple smaller video frames). In other embodiments, the downsampling layer can also employ interpolation algorithms such as interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. In one example, when using an interpolation algorithm for downsampling, only the interpolated values ​​can be retained while the original pixel values ​​are removed, thereby reducing the size of the feature map.

[0104] Upsampling is used to increase the size of each video frame in a feature video block, thereby increasing the amount of data in the feature video block. This can be achieved, for example, through an upsampling layer, but is not limited to this. In some embodiments, the upsampling layer can employ upsampling methods such as strided transposed convolution or interpolation algorithms. Interpolation algorithms can include, for example, interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation. In one example, when using an interpolation algorithm for upsampling, both the original pixel values ​​and the interpolated values ​​can be preserved, thereby increasing the size of the feature map.

[0105] Each scaling process at each level can be viewed as a residual network. The residual network can maintain its input in its output at a certain ratio through residual chaining and summing. In other words, residual chaining and summing can preserve the input of each scaling process at a certain ratio in the output of that level. For example, the input and output of residual chaining and summing can have the same size.

[0106] In some implementations, each downsampling process and each upsampling process can use different parameters.

[0107] It is understandable that during the cyclic scaling process, the N layers may include N-1 downsampling processes and N-1 upsampling processes. Each downsampling process can use different parameters, and each upsampling process can use different parameters. This makes the network structure more flexible and allows the number of features in each layer to be adjusted to balance the computational performance of each module.

[0108] The parameters for downsampling can be determined by the downsampling method used in the downsampling layer. For example, if the downsampling layer is a span convolutional layer, the downsampling parameters can be those of the span convolutional layer. Similarly, the parameters for upsampling can be determined by the upsampling method used in the upsampling layer. For example, if the upsampling layer is a span transposed convolutional layer, the upsampling parameters can be those of the span transposed convolutional layer.

[0109] In some implementations, different parameters can be used for each residual link addition process.

[0110] The parameter for residual chain addition processing can be, but is not limited to, the proportion by which each input is kept in the output.

[0111] Specifically, the parameters of downsampling processing at different levels (i.e., the parameters of the network structure corresponding to the downsampling processing) can be different; the parameters of upsampling processing at different levels (i.e., the parameters of the network structure corresponding to the upsampling processing) can be different; and the parameters of residual linking and adding processing at different levels can be different. The parameters of downsampling processing at the same level in different orders can be different; the parameters of upsampling processing at the same level in different orders can be different; and the parameters of residual linking and adding processing at the same level in different orders can be different.

[0112] Of course, in other embodiments, each downsampling process and each upsampling process can also use the same parameters, and each residual chain addition process can also use the same parameters. That is to say, the parameters of downsampling processes at different levels can also be the same; the parameters of upsampling processes at different levels can also be the same; the parameters of residual chain addition processes at different levels can also be the same. The parameters of downsampling processes at the same level in different orders can also be the same, the parameters of upsampling processes at the same level in different orders can also be the same; the parameters of residual chain addition processes at the same level in different orders can also be the same. The embodiments of this application do not limit this.

[0113] In some embodiments, to improve global features such as brightness and contrast of feature video blocks, multi-scale cyclic sampling processing may further include: performing instance normalization or layer normalization on the output of downsampling processing, the output of upsampling processing, etc. It should be noted that the output of downsampling processing, the output of upsampling processing, etc., may use the same normalization method (instance normalization or layer normalization), or they may use different normalization methods. The embodiments of this application do not limit this.

[0114] In the nested scaling process, the scaling process at level j+1 is nested between the downsampling process at level j and the concatenation process at level j, where j = 1, 2, ..., N-2. That is, the output of the downsampling process at level j serves as the input to the scaling process at level j+1, and simultaneously, the output of the scaling process at level j+1 serves as one of the inputs to the concatenation process at level j (the initial feature video block at level j+1 serves as the other input to the concatenation process at level j).

[0115] It should be noted that, in this application, "nesting" means that an object includes another object that is similar to or the same as that object, and the object includes, but is not limited to, processes or network structures.

[0116] In some implementations, scaling processes at at least one level can be executed multiple times consecutively; that is, each level may include multiple scaling processes, for example, the output of a previous scaling process is used as the input of a subsequent scaling process. For example, such as... Figure 5 and Figure 6 As shown, the scaling process at each level can be executed twice consecutively. In this case, the quality of the output video block can be improved while avoiding network structure complexity. It should be noted that the embodiments of this application do not limit the specific number of times the scaling process at each level is executed.

[0117] Specifically, in low-resolution layers, the scaling processes are more concentrated, while in high-resolution layers, scaling processes need to pass through the low-resolution layers, resulting in a more relaxed processing flow at high-resolution levels. Thus, rear-projection blocks can provide long jumpers at high resolution and short jumpers at low resolution. High-resolution networks have fewer features than low-resolution networks, allowing for efficient data processing across multiple levels, concentrating intensive processing at lower resolutions.

[0118] In some embodiments, the resolution of the intermediate feature video block is the same as the resolution of the input video block INPUT. For example... Figure 5 As shown, in the case of N=5, the initial feature video block F01 of the first level can be subjected to the above-mentioned cyclic scaling process based on the initial feature video blocks F01 to F05 of the second to fifth levels to obtain the intermediate feature video block FM.

[0119] In step S140, as Figure 5 and Figure 6 As shown, the intermediate feature video block FM can be synthesized using the SYNTHESIS network to obtain the output video block OUTPUT. In some embodiments, the SYNTHESIS network may include convolutional layers, etc. The output video block may include grayscale video data with one channel, or it may include, for example, RGB video data with three channels (i.e., color video data). It should be noted that the embodiments of this application do not limit the structure and parameters of the SYNTHESIS network, as long as it can convert the convolutional feature dimension (i.e., the intermediate feature video block FM) into the output video block OUTPUT.

[0120] Please see Figure 7 In some embodiments, step S120 includes:

[0121] Step S121: Concatenate the input video block with the random noise video block to obtain a joint input video block; and

[0122] Step S122: Perform N different levels of analysis and processing on the joint input video block to obtain N initial feature video blocks arranged from high to low resolution.

[0123] like Figure 6As shown, the input video block INPUT can be concatenated with the random noise video block to obtain a joint input video block. Then, an analysis network performs N different levels of analysis on the joint input video block to obtain N initial feature video blocks F01 to F0N arranged in descending order of resolution. For example, the concatenation process can be viewed as stacking the channel data of multiple (e.g., two or more) video blocks to be concatenated, so that the number of channels in the concatenated video block is the sum of the number of channels of the multiple video blocks to be concatenated. For example, the channel data of the joint input video block is the combination of the channel data of the input video block and the channel data of the random noise video block.

[0124] In one example, the random noise in a random noise video block may conform to a Gaussian distribution, but is not limited to this. Figure 6 The specific process and details of the analysis and processing in the illustrated embodiments can be found in [reference]. Figure 5 The descriptions of the analysis and processing in the illustrated embodiments will not be repeated here.

[0125] It should be noted that during video enhancement, the detailed features (e.g., hair, lines, etc.) in the output video block are often related to noise. When applying neural networks for video enhancement, the amplitude of the input noise is adjusted according to actual needs (whether details need to be highlighted and the degree of highlighting), so that the output video block meets the actual requirements.

[0126] In some embodiments, the noise amplitude of a random noise video block can be 0; in other embodiments, the noise amplitude of a random noise video block can be non-zero. The embodiments of this application do not impose such limitations.

[0127] Please see Figure 8 In some implementations, step S110 includes:

[0128] Step S111: Obtain the original input video block with a first resolution; and

[0129] Step S112: Perform resolution conversion processing on the original input video block to obtain an input video block with a second resolution, which is greater than the first resolution.

[0130] Thus, the input video block is obtained by acquiring an original input video block with a first resolution and performing resolution conversion processing (e.g., super-resolution reconstruction processing) on ​​the original input video block. Super-resolution reconstruction is a technique for upscaling video data to obtain a higher resolution. Super-resolution reconstruction techniques can be generated using interpolation algorithms. For example, commonly used interpolation algorithms include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, etc. Using one of the above interpolation algorithms, each video frame in the original input video block can be processed individually to generate multiple pixels from one pixel of the video frame, thereby obtaining a super-resolution video frame, and further obtaining a super-resolution input video block based on the original input video block. In other words, the video block processing method provided in the embodiments of this application can enhance super-resolution video blocks generated by conventional methods, thereby improving the quality of the super-resolution video block.

[0131] It is understandable that directly using the above video block processing method to process input video blocks with higher resolution (e.g., resolution of 4k or above) places higher demands on the hardware conditions (e.g., video memory) of the video block processing device.

[0132] To resolve the above issues, please refer to Figure 9 In some implementations, the video block processing method may include:

[0133] Step S115: The input video block is cropped to obtain multiple sub-input video blocks with overlapping areas;

[0134] Step S120 specifically includes:

[0135] Step S1200: Based on each sub-input video block, obtain N levels of sub-initial feature video blocks arranged from high to low resolution, where N is a positive integer and N>2;

[0136] Step S130 specifically includes:

[0137] Step S1300: Based on the sub-initial feature video blocks of the 2nd to Nth levels, perform cyclic scaling on the sub-initial feature video blocks of the 1st level to obtain sub-intermediate feature video blocks. The resolution of the sub-intermediate feature video blocks is the same as the resolution of the sub-input video blocks.

[0138] Step S140 specifically includes:

[0139] Step S1400: Perform synthesis processing on the sub-intermediate feature video blocks to obtain the corresponding sub-output video blocks; and stitch the sub-output video blocks corresponding to multiple sub-input video blocks into an output video block, wherein the resolution of the sub-output video block is the same as the resolution of the sub-input video block.

[0140] Specifically, please refer to Figure 10 In step S115, the input video block of size T×H×W can be cropped into multiple sub-input video blocks of size t×h×w with overlapping areas. These multiple sub-input video blocks should cover the entire input video block, and the centers of the sub-input video blocks should form a uniform and regular grid (e.g., ...). Figure 10 As shown, the centers maintain a constant distance. Pixels of each video frame are included in at least one sub-input video block. Video blocks can be batch-processed using a fixed time T on a time scale, thereby realizing pipelined processing of the video stream. That is, for a continuously input video stream, the video stream can be sequentially formed into video blocks (T×H×W) of a preset duration T according to a preset time, and then the video blocks are processed in batches in sequence.

[0141] It should be understood that the row and column positions of pixels in each video frame of the input video block correspond one-to-one with the row and column positions of pixels in each video frame of the output video block, and the row and column positions of pixels in each video frame of each sub-input video block correspond one-to-one with the row and column positions of pixels in each video frame of the corresponding sub-output video block. The relative position of pixels in a sub-output video block within the output video block is the same as the relative position of pixels in the corresponding sub-input video block within the input video block.

[0142] Please see Figure 11 and Figure 12 In some embodiments, step S1400 includes:

[0143] Step S1401: Initialize the initial output video matrix and the initial counting matrix. The resolution of the initial output video matrix and the initial counting matrix are the same as the resolution of the output video block.

[0144] Step S1402: Use window functions to add the pixel values ​​of multiple sub-output video blocks to the corresponding positions in the output video matrix to obtain the output video matrix;

[0145] Step S1403: Each time a pixel value is added to the initial output video matrix, a floating-point number equal to the value of the window function is added to the corresponding element value of the initial counting matrix to obtain the counting matrix; and

[0146] Step S1404: Process the corresponding elements of the output video matrix and the counting matrix to generate the output video block.

[0147] Specifically, the output video block can be represented in matrix form. For example, the pixel values ​​of the output video block can be represented in the corresponding elements of a three-dimensional matrix. That is, the row and column positions of the pixels of each video frame in the output video block can correspond one-to-one with the element positions in the three-dimensional matrix. Thus, the resolution of the initial output video matrix can be determined by the output video block. Similarly, the row and column positions of the pixels of each video frame in the input video block correspond one-to-one with the row and column positions of the pixels of each video frame in the output video block. Likewise, the resolution of the output video matrix can also be determined by the resolution of the input video block. In step S1401, the resolution of the initial output video matrix and the initial counting matrix is ​​the size of the matrix. When initializing the initial output video matrix, all pixel values ​​of the initial output video matrix are set to zero. It should be noted that when the output video block is grayscale video, each pixel value of the initial output video matrix can have one channel; when the output video block is RGB video data with three channels (i.e., color video block), each pixel value of the initial output video matrix can correspondingly have three channels. When initializing the initial counting matrix, all element values ​​are set to zero. This initial counting matrix has one channel.

[0148] When performing pipelined processing of video streams, the input video blocks in each batch (fixed time T) can be divided into a group (or multiple) of sub-input video blocks. Video block stitching can then be batch-processed by using a neural network to process multiple sub-output video blocks in parallel within a group. The number of sub-output video blocks processed in a batch can be arbitrary. Of course, if the equipment is capable of performing this task, the number of sub-output video blocks processed in a batch can also be fixed, except for the last batch.

[0149] In one example Figure 11 The diagram illustrates the relative positions of multiple sub-output video blocks (numbered 1-12) within the output video block during the stitching process of sub-input video blocks. It can be understood that since the resolution of the output video block is the same as that of the input video block, and the relative positions of pixels in the output video block are the same as the relative positions of pixels in the corresponding sub-input video blocks within the input video block, the relative positions of the multiple sub-output video blocks within the output video block can be determined based on the relative positions of the multiple sub-input video blocks within the input video block obtained during the cropping process of the input video block in step S115. For example, a sub-output video block can be represented by a 12*12*t matrix, and the video frames of the output video block can be represented by a 30*39*T matrix.

[0150] It should be noted that, Figure 11This diagram only illustrates the splicing of video blocks in the spatial dimension. The video configuration file in the temporal dimension can be similar, meaning that multiple sub-output video blocks may overlap in the temporal dimension, in which case t < T. Of course, in other implementations, the multiple sub-output video blocks may not overlap in the temporal dimension, as long as they cover the entire output video block. In this case, T can be equal to t, or T can be a positive integer multiple of t.

[0151] For each batch, in step S1402, the step of adding the pixel values ​​of multiple sub-output video blocks to the corresponding positions in the output video matrix using a window function can be as follows: The pixel values ​​of the generated sub-output video blocks are multiplied by the value of a window function and then added to the initial output video matrix. The position added to the initial output video matrix corresponds to the relative position of the pixels of the sub-output video blocks to the output video blocks. When overlapping pixels of multiple sub-output video blocks are added to the initial output video matrix, the multiple pixel values ​​added to the corresponding positions in the initial output video matrix are summed. Figure 11 In the example, the sub-output video block is a 12*12 sub-output block (corresponding to one of the matrices numbered 1-12, for example, matrix number 1). A 12*12 window function can be obtained, and the pixel values ​​of the sub-output block are multiplied by the values ​​at the corresponding positions in the window function to obtain a sub-output block to be added to the initial output matrix (e.g., ...). Figure 11 The initial output matrix, illustrated in the diagram, can be a 30x39 matrix (the matrix with the new number 1 in the diagram). The output video matrix is ​​obtained by adding the pixel values ​​of all sub-output blocks to their corresponding positions in the initial output video matrix. Specifically, as shown... Figure 11As shown, the two matrices, newly numbered 1 and newly numbered 2, have an overlapping region 1. Within this overlapping region 1 are the sums of the corresponding values ​​from the two matrices, which are then used as the updated values ​​for the corresponding positions in the 30×39 initial output matrix. Specifically, both matrices newly numbered 1 and newly numbered 2 contain 12×12 values. The last three columns of matrix newly numbered 1 and the first three columns of matrix newly numbered 2 overlap. For example, the data a(1, 12) in the 1st row and 12th column of matrix newly numbered 1 overlaps with the data b(1, 3) in the 1st row and 3rd column of matrix newly numbered 2. In this case, the data c(1, 12) in the 1st row and 12th column of the initial output matrix is ​​a(1, 12) + b(1, 3). The data a(1, 1) in the 1st row and 1st column of matrix newly numbered 1 does not overlap with matrix newly numbered 2. In this case, the data c(1, 1) in the 1st row and 1st column of the initial output matrix is ​​a(1, 1). This process continues until the final value of the initial output matrix is ​​obtained. As mentioned above, the data in the initial output matrix corresponds to several values ​​from the new numbers 1-12. The final data in the initial output matrix is ​​equal to the sum of these multiple values. The examples above illustrate the cases corresponding to one and two values; the cases corresponding to four values ​​are also included (e.g., ...). Figure 11 The new matrices numbered 1, 2, 5, and 6 in the middle matrix all have overlapping areas.

[0152] In step S1403, since the initial output video matrix and the initial counting matrix have the same resolution, and their elements are in relative positions, each time a pixel value is added to the initial output video matrix, a floating-point number equal to the window function value is added to the corresponding element value in the initial counting matrix. The multiple floating-point numbers added at the positions corresponding to pixels in the initial counting matrix that overlap with multiple sub-output video blocks are summed. Specifically, Figure 11 The values ​​in the overlapping region 2 are simply the sum of the values ​​corresponding to the two window functions. The counting matrix is ​​obtained by adding the floating-point values ​​of the window functions corresponding to all sub-output video blocks to their respective positions in the initial counting matrix.

[0153] The window function value can be obtained by normalizing a distance matrix determined by the distances of each pixel in a sub-output video block from the center of that sub-output video block. For example, the window function value can be inversely proportional to the distance of each pixel in a sub-output video block from the center of that sub-output video block. Correspondingly, the counting matrix can be used to record how many sub-output video blocks a particular pixel in the output video matrix is ​​formed by merging.

[0154] In one example, the window function could be the Hadamard window function. In other implementations, the window function could also be the result of matrix normalization determined by other distance metrics, which is not specifically limited here.

[0155] After obtaining the output video matrix and the counting matrix, the output video matrix is ​​divided by the corresponding value in the counting matrix, pixel by pixel. For color video blocks (such as RGB), each channel must be divided independently by the corresponding value in the counting matrix. It should be noted that all values ​​in the counting matrix are strictly positive (greater than zero).

[0156] Furthermore, the sub-output video blocks can be processed in temporal order (T), while the processing order in spatial dimensions (H and W) can be arbitrary. Processing in temporal order allows the release of memory for unnecessary video frames, and new video frames can be loaded for further processing.

[0157] Please see Figure 13 , Figure 14 and Figure 15 This application provides a method for training a neural network. The neural network 100 includes: an analysis network 110, a cyclic scaling network 120, and a synthesis network 130. The method for training the neural network includes:

[0158] Step S210: Obtain the first training input video block, which includes multiple video frames arranged in chronological order;

[0159] Step S220: The first training input video block is processed using the analysis network 110 to obtain N initial training feature video blocks arranged from high to low resolution, where N is a positive integer and N>2;

[0160] Step S230: Using the cyclic scaling network 120, based on the initial training feature video blocks of the 2nd to Nth layers, the initial training feature video blocks of the 1st layer are cyclically scaled to obtain intermediate training feature video blocks. The resolution of the intermediate training feature video blocks is the same as the resolution of the first training input video block.

[0161] Step S240: Use the synthesis network 130 to synthesize the intermediate feature video blocks during training to obtain the first training output video block. The resolution of the first training output video block is the same as the resolution of the first training input video block.

[0162] Step S250: Based on the first training output video block, calculate the loss value of neural network 100 using a loss function; and

[0163] Step S260: Adjust the parameters of the neural network 100 based on the loss value of the neural network 100;

[0164] The cyclic scaling process includes: N-1 levels of nested scaling processes, each level of which includes downsampling, concatenation, upsampling and residual concatenation.

[0165] The downsampling process of level i is based on the input of the scaling process of level i to obtain the downsampled output of level i. The concatenation process of level i is based on the downsampled output of level i and the initial feature video block of level i+1 to obtain the joint output of level i. The upsampling process of level i is based on the joint output of level i to obtain the upsampled output of level i. The residual concatenation and addition process of level i adds the input of the scaling process of level i and the upsampled output of level i to obtain the output of the scaling process of level i. Where i = 1, 2, ..., N-1;

[0166] The scaling process at level (j+1) is nested between the downsampling process at level (j) and the connection process at level (j). The output of the downsampling process at level (j) is used as the input of the scaling process at level (j+1), where j = 1, 2, ..., N-2.

[0167] The neural network 100 training method of this application is used to train the neural network 100. The neural network 100 can be trained by acquiring a large number of frames by acquiring video blocks. By combining multiple video frames set in time sequence with multiple dimensions to process the video blocks, the temporal correlation can be effectively learned. Moreover, based on the input video blocks, multiple initial feature video blocks of different resolutions are obtained. By combining these initial feature video blocks of different resolutions and performing cyclic scaling on the initial feature video block with the highest resolution, higher video fidelity can be obtained and the quality of the output video blocks can be greatly improved.

[0168] Understandable, such as Figure 13 As shown, the neural network 100 includes an analysis network 110, a cyclic scaling network 120, and a synthesis network 130. The neural network 100 of this embodiment can be used to perform the aforementioned embodiments (e.g., Figure 5 or Figure 6The video block processing method provided in the illustrated embodiment is as follows. For example, the analysis network 110 can be used to execute step S120 in the aforementioned video block processing method, that is, the analysis network 110 can process the input video block to obtain N initial feature video blocks arranged from high to low resolution, where N is a positive integer and N>2; the cyclic scaling network 120 can be used to execute step S130 in the aforementioned video block processing method, that is, the cyclic scaling network 120 can perform cyclic scaling processing on the initial feature video block of the first level based on the initial feature video blocks of the second to Nth levels to obtain intermediate feature video blocks; the synthesis network 130 can be used to execute step S140 in the aforementioned video block processing method, that is, the synthesis network 130 can perform synthesis processing on the intermediate feature video blocks to obtain the output video block. For example, the specific structures of the neural network 100, the analysis network 110, the cyclic scaling network 120, and the synthesis network 130, as well as their corresponding specific processing procedures and details, can be referred to the relevant descriptions in the aforementioned video block processing method, and will not be repeated here.

[0169] Specifically, in step S210, similar to the input video block in step S110 above, the first training input video block may include video data captured by a smartphone camera, tablet camera, personal computer camera, digital camera lens, surveillance camera or webcam, etc., which may include videos of people, animals and plants or landscapes, etc., and the embodiments of this application do not limit this.

[0170] The first training input video block can be a grayscale video block or a color video block. The color video block includes, but is not limited to, RGB data with three channels.

[0171] In some implementations, the first training input video block is obtained by acquiring the original training input video block and performing resolution transformation (e.g., super-resolution reconstruction) on it. The super-resolution video block can be generated using an interpolation algorithm. Common interpolation algorithms include nearest-neighbor interpolation, bilinear interpolation, bicubic interpolation, Lanczos interpolation, and so on. Using one of these interpolation algorithms, multiple pixels can be generated based on a single pixel in the original training input video block to obtain a first super-resolution training input video block based on the original training input video block.

[0172] In step S220, similar to the analysis network 110 in step S120, the analysis network 110 may include N analysis sub-networks. Each analysis sub-network is used to perform analysis processing at different levels to obtain N training initial feature video blocks arranged from high to low resolution. For example, each analysis sub-network may be implemented as a convolutional network module including convolutional neural networks (CNN), residual networks (ResNet), dense networks (DenseNet), etc. For example, each analysis sub-network may include convolutional layers, downsampling layers, normalization layers, etc., but is not limited to these.

[0173] In some implementations, the resolution of the highest-resolution first-level initial training feature video block can be the same as the resolution of the first training input video block. For example, in some embodiments, the first training input video block is obtained by performing resolution transformation processing (e.g., super-resolution reconstruction processing) on ​​the original training input video block. In this case, the resolution of the lowest-resolution Nth-level initial training feature video block can be the same as the resolution of the original training input video block. It should be noted that the embodiments of this application include, but are not limited to, this.

[0174] In step S230, the specific process and details of the cyclic scaling process of the cyclic scaling network 120 can be found in the description of the cyclic scaling process in step S130 above, and will not be repeated here.

[0175] In step S240, the parameters of the neural network 100 include the parameters of the analysis network 110, the parameters of the cyclic scaling network 120, and the parameters of the synthesis network 130. For example, the initial parameters of the neural network 100 can be random numbers, such as random numbers that conform to a Gaussian distribution; however, the embodiments of this application do not impose this limitation.

[0176] In some implementations, the N-level initial training feature video blocks are obtained by the analysis network 110 directly performing different levels of analysis on the first training input video block (not connected to the random noise video block) (see reference). Figure 5 (As shown).

[0177] It can be understood that for an input video block x, the neural network V can output an enhanced video block y, i.e., y = V(x). Here, the time t of the video block is greater than 1 (more than 1 output video frame).

[0178] In some implementations, the training objective of the neural network 100 is to minimize the loss value. For example, during the training process of the neural network 100, the parameters of the neural network 100 are continuously adjusted so that the first training output video block output by the neural network 100 after parameter adjustment continuously approaches the standard video block, thereby continuously reducing the loss value. It should be noted that the loss function provided in this embodiment is exemplary, and the embodiments of this application include, but are not limited to, this.

[0179] In other embodiments, the N-level initial feature video blocks for training are obtained by first concatenating the first training input video block with a random noise video block (CONCAT) to obtain a joint training input video block, and then performing N different levels of analysis processing on the joint training input video block by the analysis network 110 (see reference). Figure 6 (As shown). In this case, the training process of neural network 100 requires joint discriminant network 200.

[0180] In some implementations, the loss function of the neural network 100 can be expressed as:

[0181] L(Y, X) = L L1 (Y n＝0 ,X)+

[0182] L L1 (S2(Y n＝0 ), S2(X))+

[0183] L L1 (S4(Y n＝0 ), S4(X))+

[0184] L L1 (S8(Y n＝0 ), S8(X))+

[0185] L L1 (S 16 (Y n＝0 ), S 16 (X));

[0186] Where L(Y, X) represents the loss function, and Y represents the first training output video block (including Y). n =1 and Y n =0), X represents the first training input video block corresponding to the first training input video block, S f L represents the downsampling of the bicubic difference of factor f (spatiotemporal factor 1*f*f), L1 Indicates fidelity. In one example, L L1 (x, y) = E[|xy|].

[0187] In addition, models can be validated based on metrics:

[0188] V(Y)=L L2 (Y n＝0 (X);

[0189] Among them, L L2 (x, y) = E[(xy)] 2 [This refers to the mean square error related to the fidelity measurement.]

[0190] In some implementations, step S250 includes: processing the first training output video block using the discriminant network 200, and calculating the loss value of the neural network 100 based on the output of the discriminant network 200 corresponding to the first training output video block.

[0191] like Figure 16 As shown, the discriminant network 200 may include an M-1 level downsampling subnetwork DSN, an M level discriminant branch network, a synthesis subnetwork, and activation layers, where M is a positive integer and M > 1. For example, Figure 16 The example shown illustrates the case where M = 3, but this should not be considered a limitation of this application; that is, the value of M can be set according to actual needs. For example, in some embodiments, M = N-1. For example, in... Figure 16 In this system, the order of each level is determined from top to bottom.

[0192] In some implementations, when the discriminant network 200 processes the first training output video block, it first performs downsampling processing on the first training output video block at different levels through M-1 level downsampling sub-networks to obtain the output of the M-1 level downsampling sub-networks; then, the first training output video block and the output of the M-1 level downsampling sub-networks are respectively used as the inputs of the M level discriminant branch networks.

[0193] In some embodiments, the resolution of the output of the previous level downsampling subnetwork is higher than the resolution of the output of the next level downsampling subnetwork. For example, in some embodiments, a first training output video block is used as the input to the first level discriminative branch network, the output of the first level downsampling subnetwork is used as the input to the second level discriminative branch network, the output of the second level downsampling subnetwork is used as the input to the third level discriminative branch network, and so on, with the output of the (M-1)th level downsampling subnetwork used as the input to the Mth level discriminative branch network.

[0194] The downsampling subnetwork includes downsampling layers. For example, the downsampling subnetwork can employ downsampling methods such as maxpooling, average pooling, strided convolution, decimation (e.g., selecting a fixed number of pixels), and demuxout (splitting the input video block into multiple smaller video blocks) to achieve downsampling processing. For instance, the downsampling layer can also use interpolation algorithms such as interpolation, bilinear interpolation, bicubic interpolation, and Lanczos interpolation for downsampling processing.

[0195] In some implementations, each level of the discrimination subnetwork includes sequentially connected luminance processing subnetworks (e.g., Figure 16 (As shown in the dashed box), the first convolutional subnetwork, the second convolutional subnetwork, and the third convolutional subnetwork. For example, in some embodiments, the brightness processing subnetwork may include a brightness feature extraction subnetwork, a normalization subnetwork, and a translation correlation subnetwork.

[0196] In some implementations, the luminance feature extraction subnetwork at each level is used to extract luminance feature video blocks from the input of the discriminative branch network at that level. Since the human eye is more sensitive to luminance features than other features, extracting luminance feature video blocks from training video blocks can remove unnecessary information, thereby reducing computational load. It should be understood that the luminance feature extraction subnetwork can be used to extract luminance feature video blocks from color video blocks; that is, the luminance feature extraction subnetwork functions when the first training output video block is a color video block. However, when the input of the discriminative branch network (i.e., the first training output video block, etc.) is a grayscale video block, the luminance feature extraction subnetwork may not be necessary.

[0197] Taking the first training output video block as 3-channel RGB data (i.e., a color video block) as an example, in this case, the outputs of the M-1 level downsampling sub-networks are also 3-channel RGB data. That is, the input to the discrimination branch network at each level is 3-channel RGB data. In this case, the feature extraction sub-network can extract the brightness features using the following formula:

[0198] P = 0.299R + 0.587G + 0.114B

[0199] Where R, G, and B represent the red information (i.e., the data information of the first channel), green information (i.e., the data information of the second channel), and blue information (i.e., the data information of the third channel) in RGB format, respectively, and P represents the converted luminance information.

[0200] For example, the normalization subnetwork is used to normalize the above-mentioned brightness features to obtain normalized brightness feature video blocks. After normalization, the pixel values ​​of the normalized brightness feature video blocks can be unified within a relatively small range, preventing some pixel values ​​from being too large or too small, thus making it easier to calculate the correlation.

[0201] The translation correlation subnetwork is used to perform multiple translation processes on the above-mentioned normalized brightness feature video block to obtain multiple shifted video blocks; and based on the correlation between the normalized brightness feature video block and each shifted video block, multiple correlated video blocks are generated.

[0202] In some implementations, the first convolutional subnetwork is used to perform convolution processing on multiple related video blocks to obtain a first convolutional feature video block; that is, the first convolutional subnetwork may include convolutional layers. For example, in some embodiments, the first convolutional subnetwork may also include a normalization layer, so that the first convolutional subnetwork can also perform normalization processing. It should be noted that the embodiments of this application include, but are not limited to, these.

[0203] In some implementations, the second convolutional subnetwork may include convolutional layers and downsampling layers, thereby enabling convolutional and downsampling processing of the input to the second convolutional subnetwork. For example, as... Figure 16 As shown, the output of the first convolutional subnetwork in the first-level discriminative branch network is used as the input of the second convolutional subnetwork in the first-level discriminative branch network; the output of the second convolutional subnetwork in the t-th level discriminative branch network is concatenated with the output of the first convolutional subnetwork in the (t+1)-th level discriminative branch network and used as the input of the second convolutional subnetwork in the (t+1)-th level discriminative branch network, where t is an integer and 1≤t≤M-2.

[0204] In some implementations, the output of the second convolutional subnetwork in the discriminative branch network of the (M-1)th level is concatenated with the output of the first convolutional subnetwork in the discriminative branch network of the Mth level and used as the input of the third convolutional subnetwork.

[0205] In some embodiments, the synthesis subnetwork is connected to the third convolutional subnetwork in the discrimination branch network of the Mth level. The synthesis subnetwork is used to synthesize the output of the third convolutional subnetwork in the discrimination branch network of the Mth level to obtain a discriminative output video block. In some embodiments, the specific structure of the synthesis subnetwork and the specific process and details of its synthesis processing can be referred to the relevant description of the synthesis network 130 above, and will not be repeated here.

[0206] In some implementations, such as Figure 16As shown, the activation layer is connected to the synthesis subnetwork. In some embodiments, the activation function of this activation layer can be the sigmoid function, so that the output of the activation layer (i.e., the output of the discriminant network 200) is a value in the range [0, 1]. For example, the output of the discriminant network 200 can be used to characterize, for example, the quality of the first training output video block. For example, the larger the value of the output of the discriminant network 200, for example, close to 1, the higher the quality of the first training output video block is considered by the discriminant network 200 (e.g., closer to the quality of the first standard video block); for example, the smaller the value of the output of the discriminant network 200, for example, close to 0, the lower the quality of the first training output video block is considered by the discriminant network 200.

[0207] In some embodiments, the first standard video block has the same scene as the first training input video block, i.e., the content is the same, and the quality of the first standard video block is higher than that of the first training output video block. For example, the first standard video block Y is equivalent to the target output video block of neural network 100. For example, the quality evaluation criteria for video blocks include mean squared error (MSE), similarity (SSIM), peak signal-to-noise ratio (PSNR), etc. In some embodiments, interpolation algorithms such as bilinear interpolation, bicubic interpolation, and Lanczos interpolation can be used to downsample the first standard video block to obtain the original training input video block, and then the original training input video block is subjected to resolution conversion processing (e.g., super-resolution reconstruction processing) to obtain the first training input video block, thereby ensuring that the first standard video block and the first training input video block have the same scene. It should be noted that the embodiments of this application include, but are not limited to, these.

[0208] In some implementations, when the neural network 100 is trained using the aforementioned discriminant network 200, the loss function of the neural network 100 can be expressed as:

[0209]

[0210]

[0211] Where L(Y, X) represents the loss function, and Y represents the first training output video block (including Y). n =1 and Y n =0), Y represents the generation loss function. n＝1 Y represents the first training output video block obtained when the noise amplitude of the random noise video block is not zero. n＝0 L represents the first training output video block obtained when the noise amplitude of the random video block is 0. L1 S represents the contrastive loss function. fL represents the downsampling of the bicubic difference of factor f (spatiotemporal factor 1*f*f), contextual This represents the content loss function. λ1, λ2, λ3, λ4, and λ5 represent the preset weights.

[0212] The preset weights can be adjusted according to actual needs. In one example, λ1∶λ2∶λ3∶λ4∶λ5=0.001∶10∶0.1∶10∶10. The embodiments of this application include, but are not limited to, this.

[0213] In some embodiments, the generation loss function can be expressed as:

[0214]

[0215]

[0216] fake={Y n＝1 S2(Y n＝1 S4(Y) n＝1 ), S8(Y n＝1 )},

[0217] real={X, S2(X), S4(X), S8(X)};

[0218] During competitive training, alternating training methods can be used. Alternative To calculate the loss function.

[0219] It should be noted that the generation loss function expressed by the above formula is exemplary. In other embodiments, the generation loss function can also be expressed by other commonly used formulas, and the embodiments of this application do not limit this.

[0220] In some implementations, the content loss function can be expressed as:

[0221]

[0222] Where S1 is a constant, F ij P represents the value at the j-th position in the first content feature block of the first training output video block extracted by the i-th convolutional kernel in content feature extraction. ij This represents the value at the j-th position in the second content feature block of the first training standard video block extracted by the i-th convolutional kernel in the content feature extraction.

[0223] During training, the model is validated based on metrics:

[0224] V(Y)=E[NIQE(Y n＝1 )+NIQE(S2(Y n＝1))+NIQE(S4(Y n＝1 ))).

[0225] As a non-referenced video quality metric, the above-mentioned metric can serve as an approximation of human subjective evaluation (such as the average opinion score) and is simple to compute during training.

[0226] In some embodiments, the training method for the neural network may further include: determining whether the training of the neural network 100 meets predetermined conditions; if the predetermined conditions are not met, repeating the above training process (i.e., steps S210 to S260); if the predetermined conditions are met, stopping the above training process to obtain the trained neural network 100.

[0227] In some embodiments, the predetermined condition is that the loss value corresponding to two (or more) consecutive first training output video blocks no longer decreases significantly. In other embodiments, the predetermined condition is that the neural network 100 has reached a predetermined number of training iterations or training cycles. It should be noted that the embodiments of this application are not limited in this respect.

[0228] In one embodiment, the first training output video block y of the trained neural network 100 is close to the first standard video block Y in terms of content and quality.

[0229] In adversarial training, the output video block y is compared with the first standard video block Y. The pixel loss function can also be:

[0230] or

[0231] Where L1 and L2 represent pixel loss functions, y represents the output video block, and Y represents the first standard video block.

[0232] Furthermore, a three-dimensional Laplacian operator was introduced. Any implementation of the discretized Laplace operator is considered, where t represents the time dimension and xy represents the spatial dimension.

[0233] Furthermore, the mask can be calculated using the following conditional expression:

[0234]

[0235] Here, α represents the weight of pixels in a flat region where the Laplacian operator is zero, β represents the weight of pixels where the Laplacian operator reaches its maximum value, and ε represents adding a small number to the maximum value of the Laplacian operator to avoid the denominator being zero.

[0236] In this case, the contrastive loss function can be expressed as:

[0237]

[0238] Where Loss(Y, y) represents the contrast loss function, ⊙ represents the pixel product, and |||| represents the pixel loss function mentioned above. or

[0239] It should be noted that during the training of the joint discriminant network 200 on the neural network 100, generative adversarial training is typically required. Please refer to [link / reference]. Figure 17 Generative adversarial training includes:

[0240] Step S300: Train the discriminant network 200 based on the neural network 100;

[0241] Step S400: Train the neural network 100 based on the discriminant network 200; and,

[0242] The above training process is performed alternately to obtain a trained neural network 100.

[0243] For example, the training process of neural network 100 in step S400 can be implemented through steps S210 to S260 described above, and will not be repeated here. It should be noted that the parameters of the discriminant network 200 remain unchanged during the training of neural network 100. It should also be noted that in generative adversarial training, neural network 100 can often be referred to as a generative network.

[0244] Please see Figure 18 , Figure 19 and Figure 20 In some implementations, the training process of the discrimination network 200, i.e., step S300, includes:

[0245] Step S310: Obtain the second training input video block;

[0246] Step S320: Process the second training input video block using neural network 100 to obtain the second training output video block;

[0247] Step S330: Based on the second training output video block, calculate the discriminant loss value using the discriminant loss function;

[0248] Step S340: Adjust the parameters of the discrimination network 200 according to the discrimination loss value.

[0249] In some implementations, the training process of the discriminant network 200, i.e. step S400, may further include: determining whether the training of the discriminant network 200 meets predetermined conditions; if the predetermined conditions are not met, repeating the above-mentioned training process of the discriminant network 200; if the predetermined conditions are met, stopping the training process of the discriminant network 200 to obtain a trained discriminant network 200.

[0250] In one example, the predetermined condition is that the discrimination loss values ​​corresponding to two (or more) consecutive second training output video blocks and second standard video blocks no longer decrease significantly. In another example, the predetermined condition is that the discrimination network 200 has reached a predetermined number of training iterations or training periods. It should be noted that the embodiments of this application do not impose limitations on this.

[0251] like Figure 18 As shown, during the training of the discriminant network 200, it is necessary to train the neural network 100 in conjunction with it. It should be noted that during the training of the discriminant network 200, the parameters of the neural network 100 remain unchanged.

[0252] It should be noted that the above example is merely illustrative of the training process of the discriminant network 200. Those skilled in the art will understand that during the training phase, a large number of samples are needed to train the discriminant network 200; furthermore, the training process for each sample may include multiple iterations to refine the parameters of the discriminant network 200. For example, the training phase may also include fine-tuning the parameters of the discriminant network 200 to obtain more optimized parameters.

[0253] In some implementations, the initial parameters of the discrimination network 200 can be random numbers, for example, random numbers that conform to a Gaussian distribution. The embodiments of this application do not limit this.

[0254] In some implementations, the training process of the discriminant network 200 may also include an optimization function (not shown in the figure). The optimization function can calculate the error value of the parameters of the discriminant network 200 based on the discriminant loss value obtained from the discriminant loss function, and correct the parameters of the discriminant network 200 based on the error value. For example, the optimization function can use stochastic gradient descent (SGD) algorithm, batch gradient descent (BGD) algorithm, etc. to calculate the error value of the parameters of the discriminant network 200.

[0255] In some embodiments, the second training input video block may be the same as the first training input video block. For example, the set of second training input video blocks and the set of first training input video blocks may be the same set of video blocks. Embodiments of this application include, but are not limited to, this.

[0256] In some embodiments, the second training input video block may refer to the aforementioned description of the first training input video block, and will not be repeated here.

[0257] In some implementations, the training objective of the discriminant network 200 is to minimize the discriminant loss value. For example, during the training of the neural network 100, the parameters of the discriminant network 200 are continuously adjusted so that the discriminant network 200, after parameter adjustment, can accurately distinguish between the second training output video block and the second standard video block. That is, the discriminant network 200 determines that the deviation between the second training output video block and the second standard video block is getting larger and larger, thereby continuously reducing the discriminant loss value.

[0258] It should be noted that in this embodiment, the training of the neural network 100 and the training of the discriminant network 200 are performed alternately and iteratively. For example, for the untrained neural network 100 and discriminant network 200, the discriminant network 200 is generally trained in the first stage to improve its discrimination ability, resulting in a discriminant network 200 trained in the first stage; then, the neural network 100 is trained in the first stage based on the discriminant network 200 trained in the first stage to improve its video block enhancement processing ability, resulting in a neural network 100 trained in the first stage. Similar to the first stage of training, in the second stage of training, based on the neural network 100 trained in the first stage, the discriminant network 200 trained in the first stage is trained in the second stage to improve the discrimination ability of the discriminant network 200, resulting in the discriminant network 200 trained in the second stage. Then, based on the discriminant network 200 trained in the second stage, the neural network 100 trained in the first stage is trained in the second stage to improve the video block enhancement processing capability of the neural network 100, resulting in the neural network 100 trained in the second stage. This process is repeated, and then the discriminant network 200 and the neural network 100 are trained in the third stage, the fourth stage, and so on, until the quality of the output of the obtained neural network 100 can approach the quality of the corresponding standard video block.

[0259] This application also provides a neural network processor 50. Figure 21 This is a schematic block diagram of a neural network processor 50 provided for some embodiments of this application. For example, as... Figure 21 As shown, the neural network processor 50 includes an analysis circuit 60, a loop scaling circuit 70, and a synthesis circuit 80. For example, the neural network processor 50 can be used to execute the aforementioned video block processing method.

[0260] The analysis circuit 60 is configured to obtain N initial feature video blocks with resolution arranged from high to low based on the input video blocks, where N is a positive integer and N>2;

[0261] The cyclic scaling circuit 70 is configured to perform cyclic scaling on the initial feature video block of level 1 based on the initial feature video blocks of levels 2 to N, to obtain intermediate feature video blocks, the resolution of which is the same as the resolution of the input video block; and

[0262] The synthesis circuit 80 is configured to synthesize intermediate feature video blocks to obtain an output video block, the resolution of which is the same as that of the input video block.

[0263] The cyclic scaling circuit 70 includes N-1 levels of nested scaling circuits 75, and each level of scaling circuit 75 includes a downsampling circuit 751, a connection circuit 752, an upsampling circuit 753, and a residual link summing circuit 754.

[0264] The downsampling circuit 751 of the i-th level downsamples the input of the scaling circuit 75 of the i-th level to obtain the downsampled output of the i-th level. The connection circuit 752 of the i-th level connects the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level. The upsampling circuit 753 of the i-th level obtains the upsampled output of the i-th level based on the joint output of the i-th level. The residual linking and adding circuit 754 of the i-th level performs residual linking and adding the input of the scaling circuit 75 of the i-th level and the upsampled output of the i-th level to obtain the output of the scaling circuit 75 of the i-th level. Where i = 1, 2, ..., N-1.

[0265] The scaling circuit 75 of the (j+1)th level is nested between the downsampling circuit 751 of the jth level and the connection circuit 752 of the jth level. The output of the downsampling circuit 751 of the jth level is used as the input of the scaling circuit 75 of the (j+1)th level, where j = 1, 2, ..., N-2.

[0266] The neural network processor 50 (NPU) can be mounted as a coprocessor on the main CPU, with tasks assigned by the main CPU. The core of the NPU is the arithmetic circuitry, which, controlled by a controller, retrieves data (e.g., input matrices and weight matrices) from the internal memory 510 and performs calculations. In some embodiments, the arithmetic circuitry may include multiple processing engines (PEs). For example, in some embodiments, the arithmetic circuitry is a two-dimensional systolic array. The arithmetic circuitry can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. For example, in some embodiments, the arithmetic circuitry is a general-purpose matrix processor.

[0267] In some embodiments, the arithmetic circuit can read the corresponding data of the weight matrix from the internal memory 510 and cache it on each PE in the arithmetic circuit; in addition, the arithmetic circuit can also read the data of the input matrix from the internal memory 510 and perform matrix operations with the weight matrix, and store the partial result or the final result of the matrix in the accumulator.

[0268] The vector computation unit can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit can be used in network computation of non-convolutional / non-fully connected layers in a neural network 100, such as downsampling and normalization.

[0269] In some embodiments, the vector computation unit may store the processed output vector in the unified memory 510. For example, the vector computation unit may apply a nonlinear function to the output of the arithmetic circuit, such as a vector of accumulated values, to generate an activation value.

[0270] In some examples, the vector computation unit generates normalized values, merged values, or both. In some examples, the processed output vector can be used as activation input to computational circuits, for example, for use in subsequent layers of a neural network 100.

[0271] The video block processing method and neural network training method provided in the embodiments of this application can be executed by a computing circuit or a vector computing unit.

[0272] In some implementations, the neural network processor 50 can write input data, etc., from an external memory (not shown) to an internal memory and / or a unified memory via a memory access controller, and also store data from the unified memory into the external memory.

[0273] In some implementations, a bus interface unit is used to facilitate interaction between the main CPU, memory access controller, and instruction fetch memory via a bus. For example, the instruction fetch memory connected to the controller stores instructions used by the controller. The controller, for instance, is used to invoke instructions cached in the instruction fetch memory to control the operation of the arithmetic circuitry.

[0274] Figure 5 and / or Figure 6 The operations of each layer in the neural network 100 shown can be performed by arithmetic circuits or vector computing units.

[0275] This application also provides a video block processing apparatus 470, please refer to... Figure 22 The video block processing device 470 can be used to execute the aforementioned video block processing method, and embodiments of this application include, but are not limited to, this.

[0276] In some embodiments, the video block acquisition module 480 can be used to perform step S110 of the aforementioned video block processing method, and embodiments of this application include, but are not limited to, this. For example, the video block acquisition module 480 can be used to acquire input video blocks. The video block acquisition module 480 may include a memory 510 that stores input video blocks; or, the video block acquisition module 480 may also include one or more cameras to acquire input video blocks.

[0277] The video block processing module 490 can be used to execute steps S120-S140 of the aforementioned video block processing method, and embodiments of this application include, but are not limited to, these steps. For example, the video block processing module 490 can: based on the input video block, obtain N initial feature video blocks arranged in descending order of resolution, where N is a positive integer and N>2; based on the initial feature video blocks of the 2nd to Nth levels, perform cyclic scaling processing on the initial feature video block of the 1st level to obtain intermediate feature video blocks; and perform synthesis processing on the intermediate feature video blocks to obtain an output video block, wherein the resolution of the intermediate feature video block is the same as the resolution of the input video block, and the resolution of the output video block is the same as the resolution of the input video block. Specifically, the specific process and details of the cyclic scaling processing can be referred to the relevant description in the aforementioned video block processing method, and will not be repeated here.

[0278] In some embodiments, the video block acquisition module 480 and the video block processing module 490 can be implemented as hardware, software, firmware, and any feasible combination thereof.

[0279] This application provides a computer device 500, including a memory 510 and a processor 520. For example, the memory 510 is used to non-temporarily store computer-readable instructions 601, and the processor 520 is used to execute the computer-readable instructions 601. When the computer-readable instructions 601 are executed by the processor 520, they perform the video block processing method and / or neural network training method provided in any embodiment of this application.

[0280] Specifically, the memory 510 and the processor 520 can communicate with each other directly or indirectly. In some examples, such as Figure 23 As shown, the computer device 500 may also include a system bus 530, through which the memory 510 and the processor 520 can communicate with each other. For example, the processor 520 can access the memory 510 through the system bus 530.

[0281] In other examples, components such as memory 510 and processor 520 can communicate via a network connection. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The network can include a local area network (LAN), the Internet, a telecommunications network, an Internet of Things (IoT) network based on the Internet and / or a telecommunications network, and / or any combination of the above. Wired networks can use methods such as twisted-pair cables, coaxial cables, or fiber optic transmission for communication, while wireless networks can use methods such as 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee, or WiFi. This application does not limit the type and function of the network.

[0282] In some implementations, processor 520 can control other components in computer device 500 to perform desired functions. Processor 520 can be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), tensor processor (TPU), or graphics processing unit (GPU). The CPU can be based on x86 or ARM architectures. The GPU can be integrated directly onto the motherboard or built into the motherboard's northbridge chip. The GPU can also be integrated into the CPU.

[0283] In some embodiments, memory 510 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media 600, such as volatile memory 510 and / or non-volatile memory 510. Volatile memory 510 may include, for example, random access memory 510 (RAM) and / or cache memory 510. Non-volatile memory 510 may include, for example, read-only memory 510 (ROM), hard disk, erasable programmable read-only memory 510 (EPROM), portable compact disc read-only memory 510 (CD-ROM), USB memory 510, flash memory, etc.

[0284] The memory 510 may store one or more computer instructions, and the processor 520 may execute the computer instructions to perform various functions. Various application programs and various data may also be stored in the computer-readable storage medium 600, such as input video blocks, output video blocks, first / second training input video blocks, first / second training output video blocks, first / second training standard video blocks, and various data used and / or generated by the application programs.

[0285] In some embodiments, some computer instructions stored in memory 510, when executed by processor 520, can perform one or more steps according to the video block processing method described above. As another example, other computer instructions stored in memory 510, when executed by processor 520, can perform one or more steps according to the neural network training method described above.

[0286] Computer device 500 may also include an input interface 540 that allows external devices to communicate with it. For example, input interface 540 may be used to receive instructions from an external computer device, a user, etc. Computer device 500 may also include an output interface 550 that enables interconnection between computer device 500 and one or more external devices. For example, computer device 500 may display video through output interface 550. External devices communicating with computer device 500 via input interface 540 and output interface 550 may be included in an environment that provides any type of user interface that a user can interact with. Examples of user interface types include graphical user interfaces (GUIs), natural user interfaces (UIAs), etc. For example, a GUI may accept input from a user using input devices such as a keyboard, mouse, remote control, etc., and provide output on an output device such as a monitor. Furthermore, a UI allows a user to interact with computer device 500 in a manner unconstrained by input devices such as a keyboard, mouse, remote control, etc. In contrast, natural user interfaces can rely on voice recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, and machine intelligence, among others.

[0287] Computer equipment 500, despite Figure 23 While shown as a single system, it is understood that computer device 500 can also be a distributed system, or deployed as a cloud facility (including a public or private cloud). Thus, for example, several devices can communicate via a network connection and collaboratively perform tasks described as being performed by computer device 500.

[0288] In some implementations, a detailed description of the processing procedure of the video block processing method can be found in the relevant descriptions in the embodiments of the video block processing method described above, and a detailed description of the processing procedure of the neural network training method can be found in the relevant descriptions in the embodiments of the neural network training method described above. Repeated descriptions will not be repeated.

[0289] It should be noted that the computer device 500 provided in the embodiments of this application is exemplary and not restrictive. Depending on the actual application needs, the computer device 500 may also include other conventional components or structures. For example, in order to realize the necessary functions of the computer device 500, those skilled in the art can set other conventional components or structures according to specific application scenarios. The embodiments of this application do not limit this.

[0290] The technical effects of the computer device 500 provided in the embodiments of this application can be referred to the corresponding descriptions of the video block processing method and the neural network training method in the above embodiments, and will not be repeated here.

[0291] At least one embodiment of this application also provides a storage medium 600. Figure 24 This is a schematic diagram of a storage medium 600 provided according to an embodiment of this application. For example, such as... Figure 24 As shown, the storage medium 600 non-transitory stores computer-readable instructions 601. When the non-transitory computer-readable instructions 601 are executed by a computer (including processor 520), they can execute the video block processing method provided in any embodiment of this application or the neural network training method provided in any embodiment of this application.

[0292] For example, one or more computer instructions may be stored on the storage medium 600. Some of the computer instructions stored on the storage medium 600 may be instructions for implementing one or more steps in the video block processing method described above. Other computer instructions stored on the storage medium 600 may be instructions for implementing one or more steps in the training method of the neural network or the construction method of the merging neural network 100 described above.

[0293] For example, storage medium 600 may include storage components of a tablet computer, hard disk of a personal computer, random access memory 510 (RAM), read-only memory 510 (ROM), erasable programmable read-only memory 510 (EPROM), optical disc read-only memory 510 (CD-ROM), flash memory, or any combination of the above storage medium 600, or other suitable storage medium 600.

[0294] The technical effects of the storage medium 600 provided in the embodiments of this application can be referred to the corresponding descriptions of the video block processing method, the video block processing method of the merging neural network 100, and the training method of the neural network in the above embodiments, which will not be repeated here.

[0295] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0296] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A video block processing method, wherein, include: Obtain an input video block, wherein the input video block comprises multiple video frames arranged in chronological order; The input video block is a three-dimensional video block; Based on the input video block, N initial feature video blocks of varying resolutions from high to low are obtained, where N is a positive integer and N>2. Based on the initial feature video blocks of levels 2 to N, the initial feature video blocks of level 1 are subjected to cyclic scaling to obtain intermediate feature video blocks, the resolution of which is the same as that of the input video blocks. as well as The intermediate feature video blocks are synthesized to obtain an output video block, the resolution of which is the same as that of the input video block; The cyclic scaling process includes: N-1 levels of nested scaling processes, each level of which includes downsampling, concatenation, upsampling and residual concatenation; stride convolution and transposed convolution are combined, one consuming boundary pixels and the other generating boundary pixels, so that the resolution of the output video block remains unchanged from the resolution of the input video block. The downsampling process of level i is based on the input of the scaling process of level i to obtain the downsampled output of level i. The concatenation process of level i is based on the downsampled output of level i and the initial feature video block of level i+1 to obtain the joint output of level i. The upsampling process of level i is based on the joint output of level i to obtain the upsampled output of level i. The residual linking and addition process of level i adds the input of the scaling process of level i and the upsampled output of level i to obtain the output of the scaling process of level i. Where i = 1, 2, ..., N-1; The scaling process at level (j+1) is nested between the downsampling process at level j and the connection process at level j. The output of the downsampling process at level j is used as the input of the scaling process at level (j+1), where j = 1, 2, ..., N-2. Each of the downsampling processes and each of the upsampling processes uses different parameters; The video block processing method includes: The input video block is cropped to obtain multiple sub-input video blocks with overlapping areas; The specific steps of obtaining N levels of initial feature video blocks arranged from high to low resolution based on the input video blocks include: Based on each sub-input video block, N sub-initial feature video blocks of varying resolutions from high to low are obtained, where N is a positive integer and N>2; The process of cyclically scaling the initial feature video blocks of level 1 based on the initial feature video blocks of levels 2 to N to obtain intermediate feature video blocks specifically includes: Based on the sub-initial feature video blocks of the 2nd to Nth levels, the sub-initial feature video blocks of the 1st level are subjected to cyclic scaling to obtain sub-intermediate feature video blocks, the resolution of which is the same as that of the sub-input video blocks. The process of synthesizing the intermediate feature video blocks to obtain the output video blocks specifically includes: The sub-intermediate feature video blocks are synthesized to obtain corresponding sub-output video blocks, wherein the resolution of the sub-output video blocks is the same as the resolution of the sub-input video blocks; and The sub-output video blocks corresponding to the plurality of sub-input video blocks are concatenated to form the output video block; Wherein, the relative position of the sub-output video block pixels in the output video block is the same as the relative position of the corresponding sub-input video block pixels in the input video block, and the step of concatenating the sub-output video blocks corresponding to the plurality of sub-input video blocks into the output video block includes: Initialize the initial output video matrix and the initial counting matrix, wherein the resolution of the initial output video matrix and the initial counting matrix are the same as the resolution of the output video block; The pixel values ​​of the multiple sub-output video blocks are added to the corresponding positions in the initial output video matrix using window functions to obtain the output video matrix. Each time a pixel value is added to the initial output video matrix, a floating-point number equal to the value of the window function is added to the element value corresponding to the initial counting matrix to obtain the counting matrix; and The corresponding elements of the output video matrix and the counting matrix are processed to generate the output video block.

2. The video block processing method according to claim 1, wherein, The concatenation process of the i-th level is based on concatenating the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level, including: The downsampled output of the i-th level is used as the input to the scaling process of the (i+1)-th level to obtain the output of the scaling process of the (i+1)-th level; and The output of the scaling process at level (i+1) is concatenated with the initial feature video block at level (i+1) to obtain the joint output at level (i).

3. The video block processing method according to claim 2, wherein, At least one level of scaling is executed multiple times consecutively, with the output of the previous scaling process serving as the input for the next scaling process.

4. The video block processing method according to any one of claims 1-3, wherein, Among the N initial feature video blocks, the first-level initial feature video block has the highest resolution, and the resolution of the first-level initial feature video block is the same as that of the input video block.

5. The video block processing method according to any one of claims 1-3, wherein, The resolution of the initial feature video block of the previous level is an integer multiple of the resolution of the initial feature video block of the next level.

6. The video block processing method according to any one of claims 1-3, wherein, The process of obtaining N levels of initial feature video blocks arranged from high to low resolution based on the input video blocks includes: The input video block is concatenated with a random noise video block to obtain a joint input video block; and The joint input video block is analyzed and processed at N different levels to obtain initial feature video blocks at the N levels arranged from high to low resolution.

7. The video block processing method according to any one of claims 1-3, wherein, The acquisition of the input video block includes: Obtain the raw input video block with a first resolution; and The original input video block is subjected to resolution conversion processing to obtain the input video block with a second resolution, which is greater than the first resolution.

8. The video block processing method according to claim 7, wherein, The resolution conversion process is performed using one of the bicubic interpolation algorithm, the bilinear interpolation algorithm, and the Lanzos interpolation algorithm.

9. A method for training a neural network, wherein, The neural network includes: an analysis network, a cyclic scaling network, and a synthesis network; The training method includes: Obtain a first training input video block, which includes multiple video frames arranged in chronological order; the input video block is a three-dimensional video block. The analysis network is used to process the first training input video block to obtain N initial training feature video blocks arranged from high to low resolution, where N is a positive integer and N>2; Using the cyclic scaling network, based on the initial training feature video blocks of the 2nd to Nth layers, the initial training feature video blocks of the 1st layer are cyclically scaled to obtain intermediate training feature video blocks, the resolution of which is the same as that of the first training input video block. The synthesis network is used to synthesize the intermediate training feature video blocks to obtain a first training output video block, the resolution of which is the same as that of the first training input video block. Based on the first training output video block, the loss value of the neural network is calculated using a loss function; and The parameters of the neural network are corrected based on the loss value of the neural network; The cyclic scaling process includes: N-1 levels of nested scaling processes, each level of which includes downsampling, concatenation, upsampling and residual concatenation summation; stride convolution and transposed convolution are combined, one consumes boundary pixels and the other generates boundary pixels, so that the output resolution remains unchanged from the input resolution. The downsampling process of level i is based on the input of the scaling process of level i to obtain the downsampled output of level i. The concatenation process of level i is based on the downsampled output of level i and the initial feature video block of level i+1 to obtain the joint output of level i. The upsampling process of level i is based on the joint output of level i to obtain the upsampled output of level i. The residual linking and addition process of level i adds the input of the scaling process of level i and the upsampled output of level i to obtain the output of the scaling process of level i. Where i = 1, 2, ..., N-1; The scaling process at level (j+1) is nested between the downsampling process at level j and the connection process at level j. The output of the downsampling process at level j is used as the input of the scaling process at level (j+1), where j = 1, 2, ..., N-2. Each of the downsampling processes and each of the upsampling processes uses different parameters; The training method for the neural network also includes: The first training input video block is cropped to obtain multiple training sub-input video blocks with overlapping regions; The step of processing the first training input video block using the analysis network to obtain N levels of initial training feature video blocks arranged from high to low resolution specifically includes: Based on each trainer input video block, N initial feature video blocks of the trainer are obtained in descending order of resolution, where N is a positive integer and N>2; The process of cyclically scaling the initial training feature video blocks of level 1 to obtain intermediate training feature video blocks based on the initial training feature video blocks of levels 2 to N specifically includes: Based on the initial feature video blocks of the training sub-sub from the 2nd to Nth levels, the initial feature video blocks of the training sub-sub from the 1st level are subjected to cyclic scaling to obtain intermediate feature video blocks of the training sub-sub ... The synthesis network is used to synthesize the intermediate training feature video blocks to obtain the first training output video block, specifically including: The intermediate feature video blocks of the training sub are synthesized to obtain corresponding output video blocks of the training sub, wherein the resolution of the output video blocks of the training sub is the same as the resolution of the input video blocks of the training sub; and The training sub-output video blocks corresponding to the plurality of training sub-input video blocks are concatenated to form the first training output video block; Wherein, the relative position of the training sub-output video block pixel in the training output video block is the same as the relative position of the corresponding training sub-input video block pixel in the training input video block, and the step of concatenating the training sub-output video blocks corresponding to the plurality of training sub-input video blocks into a first training output video block includes: Initialize the initial output video matrix and the initial counting matrix, wherein the resolution of the initial output video matrix and the initial counting matrix are the same as the resolution of the output video block; The pixel values ​​of the multiple training sub-output video blocks are added to the corresponding positions in the initial output video matrix using a window function to obtain the output video matrix. Each time a pixel value is added to the initial output video matrix, a floating-point number equal to the value of the window function is added to the element value corresponding to the initial counting matrix to obtain the counting matrix; and The corresponding elements of the output video matrix and the counting matrix are processed to generate the first training output video block.

10. The neural network training method according to claim 9, wherein, The step of processing the first training input video block using the analysis network to obtain N levels of initial training feature video blocks arranged from high to low resolution includes: The first training input video block is concatenated with a random noise video block to obtain a joint training input video block; and The analysis network is used to perform N different levels of analysis on the training joint input video block to obtain the N initial feature video blocks for training, arranged in descending order of resolution.

11. The neural network training method according to claim 10, wherein, The step of calculating the loss value of the neural network based on the first training output video block using a loss function includes: processing the first training output video block using a discriminant network, and calculating the loss value of the neural network based on the output of the discriminant network corresponding to the first training output video block.

12. The neural network training method according to claim 11, wherein, The discriminant network includes: an M-1 level downsampling sub-network, an M level discriminant branch network, a synthesis sub-network, and an activation layer; The M-1 level downsampling subnetworks are used to perform downsampling processing on the input of the discriminant network at different levels to obtain the output of the M-1 level downsampling subnetworks; The input of the discriminant network and the output of the M-1 level downsampling sub-networks are respectively used as the inputs of the M level discriminant branch networks; Each level of the discrimination subnetwork includes a luminance processing subnetwork, a first convolutional subnetwork, a second convolutional subnetwork, and a third convolutional subnetwork connected in sequence; the output of the second convolutional subnetwork in the discrimination subnetwork of level t is concatenated with the output of the first convolutional subnetwork in the discrimination subnetwork of level t+1 and used as the input of the second convolutional subnetwork in the discrimination subnetwork of level t+1, where t = 1, 2, ..., M-2; The output of the second convolutional subnetwork in the discriminative branch network of the M-1 level is concatenated with the output of the first convolutional subnetwork in the discriminative branch network of the M level and then used as the input of the third convolutional subnetwork. The synthesis subnetwork is used to synthesize the output of the third convolutional subnetwork to obtain a discriminative output video block; the activation layer is used to process the discriminative output video block to obtain a value characterizing the quality of the input of the discriminative network.

13. The neural network training method according to claim 12, wherein, The loss function is expressed as: ； Where L(Y, X) represents the loss function, Y represents the first training output video block, including Y n =1 and Y n =0, where X represents the first training input video block corresponding to the first training input video block. This represents the generation loss function. This refers to the first training output video block obtained when the noise amplitude of the random noise video block is not zero. This represents the first training output video block obtained when the noise amplitude of the random video block is 0. S represents the contrastive loss function. f This represents the downsampling process of the bicubic difference of factor f. Let λ1, λ2, λ3, λ4, and λ5 represent the preset weights, respectively. The generation loss function is expressed as: ， ； ， ； The content loss function is expressed as: ； Where S1 is a constant, F ij P represents the value at the j-th position in the first content feature block of the first training output video block extracted by the i-th convolutional kernel in content feature extraction. ij This represents the value at the j-th position in the second content feature block of the first training standard video block extracted by the i-th convolutional kernel in the content feature extraction.

14. The neural network training method according to claim 13, wherein, The contrastive loss function is: ； Where Loss(Y, y) represents the loss function, ⊙ represents the pixel product, and || represents the pixel loss function. or ; Where L1 and L2 represent pixel loss functions, y represents the output video block, and Y represents the first standard video block.

15. The method for training a neural network according to claim 10 or 14, wherein, The training method for the neural network includes: Based on the aforementioned neural network, the discriminant network is trained; and The training process of the discriminant network and the training process of the neural network are executed alternately to obtain a trained neural network; The training of the discrimination network based on the neural network includes: Obtain the second training input video block; The neural network is used to process the second training input video block to obtain the second training output video block; Based on the second training output video block, a discriminant loss value is calculated using a discriminant loss function; and The parameters of the discrimination network are corrected based on the discrimination loss value.

16. A neural network processor, wherein, Neural network processors include analysis circuits, loop scaling circuits, and synthesis circuits; The analysis circuit is configured to obtain N initial feature video blocks of varying resolution from high to low based on the input video blocks, where N is a positive integer and N>2; the input video blocks are three-dimensional video blocks. The cyclic scaling circuit is configured to perform cyclic scaling on the initial feature video block of level 1 based on the initial feature video blocks of levels 2 to N, to obtain intermediate feature video blocks, wherein the resolution of the intermediate feature video blocks is the same as the resolution of the input video blocks; and The synthesis circuit is configured to synthesize the intermediate feature video blocks to obtain an output video block, the resolution of which is the same as that of the input video block. The cyclic scaling circuit includes N-1 nested scaling circuits, each level of which includes a downsampling circuit, a connection circuit, an upsampling circuit, and a residual connection summing circuit; stride convolution and transposed convolution are combined, one consuming boundary pixels and the other generating boundary pixels, so that the output resolution remains unchanged from the input resolution. The downsampling circuit of the i-th level downsamples the input of the scaling circuit of the i-th level to obtain the downsampled output of the i-th level. The connection circuit of the i-th level connects the downsampled output of the i-th level and the initial feature video block of the (i+1)-th level to obtain the joint output of the i-th level. The upsampling circuit of the i-th level obtains the upsampled output of the i-th level based on the joint output of the i-th level. The residual linking and adding circuit of the i-th level residual links and adds the input of the scaling circuit of the i-th level and the upsampled output of the i-th level to obtain the output of the scaling circuit of the i-th level. Where i = 1, 2, ..., N-1; The scaling circuit of the (j+1)th level is nested between the downsampling circuit of the jth level and the connection circuit of the jth level. The output of the downsampling circuit of the jth level is used as the input of the scaling circuit of the (j+1)th level, where j = 1, 2, ..., N-2. Each of the downsampling circuits and each of the upsampling circuits uses different parameters during processing; Also includes: The input video block is cropped to obtain multiple sub-input video blocks with overlapping areas; The specific steps of obtaining N levels of initial feature video blocks arranged from high to low resolution based on the input video blocks include: Based on each sub-input video block, N sub-initial feature video blocks of varying resolutions from high to low are obtained, where N is a positive integer and N>2; The process of cyclically scaling the initial feature video blocks of level 1 based on the initial feature video blocks of levels 2 to N to obtain intermediate feature video blocks specifically includes: Based on the sub-initial feature video blocks of the 2nd to Nth levels, the sub-initial feature video blocks of the 1st level are subjected to cyclic scaling to obtain sub-intermediate feature video blocks, the resolution of which is the same as that of the sub-input video blocks. The process of synthesizing the intermediate feature video blocks to obtain the output video blocks specifically includes: The sub-intermediate feature video blocks are synthesized to obtain corresponding sub-output video blocks, wherein the resolution of the sub-output video blocks is the same as the resolution of the sub-input video blocks; and The sub-output video blocks corresponding to the plurality of sub-input video blocks are concatenated to form the output video block; Wherein, the relative position of the sub-output video block pixels in the output video block is the same as the relative position of the corresponding sub-input video block pixels in the input video block, and the step of concatenating the sub-output video blocks corresponding to the plurality of sub-input video blocks into the output video block includes: Initialize the initial output video matrix and the initial counting matrix, wherein the resolution of the initial output video matrix and the initial counting matrix are the same as the resolution of the output video block; The pixel values ​​of the multiple sub-output video blocks are added to the corresponding positions in the initial output video matrix using window functions to obtain the output video matrix. Each time a pixel value is added to the initial output video matrix, a floating-point number equal to the value of the window function is added to the element value corresponding to the initial counting matrix to obtain the counting matrix; and The corresponding elements of the output video matrix and the counting matrix are processed to generate the output video block.

17. A video block processing apparatus, wherein, include: The acquisition module is used to acquire an input video block, which includes multiple video frames arranged in chronological order. The input video block is a three-dimensional video block; The processing module is configured to: obtain N initial feature video blocks of varying resolutions from high to low based on the input video block, where N is a positive integer and N>2; perform cyclic scaling on the initial feature video block of the first level based on the initial feature video blocks of the second to Nth levels to obtain intermediate feature video blocks; and perform compositing on the intermediate feature video blocks to obtain an output video block, wherein the resolution of the intermediate feature video block is the same as the resolution of the input video block, and the resolution of the output video block is the same as the resolution of the input video block. The cyclic scaling process includes: N-1 levels of nested scaling processes, each level of which includes downsampling, concatenation, upsampling and residual concatenation summation; stride convolution and transposed convolution are combined, one consumes boundary pixels and the other generates boundary pixels, so that the output resolution remains unchanged from the input resolution. The downsampling process of level i is based on the input of the scaling process of level i to obtain the downsampled output of level i. The concatenation process of level i is based on the downsampled output of level i and the initial feature video block of level i+1 to obtain the joint output of level i. The upsampling process of level i is based on the joint output of level i to obtain the upsampled output of level i. The residual linking and addition process of level i adds the input of the scaling process of level i and the upsampled output of level i to obtain the output of the scaling process of level i. Where i = 1, 2, ..., N-1; The scaling process at level (j+1) is nested between the downsampling process at level j and the connection process at level j. The output of the downsampling process at level j is used as the input of the scaling process at level (j+1), where j = 1, 2, ..., N-2. Each of the downsampling processes and each of the upsampling processes uses different parameters; Also includes: The input video block is cropped to obtain multiple sub-input video blocks with overlapping areas; The specific steps of obtaining N levels of initial feature video blocks arranged from high to low resolution based on the input video blocks include: Based on each sub-input video block, N sub-initial feature video blocks of varying resolutions from high to low are obtained, where N is a positive integer and N>2; The process of cyclically scaling the initial feature video blocks of level 1 based on the initial feature video blocks of levels 2 to N to obtain intermediate feature video blocks specifically includes: Based on the sub-initial feature video blocks of the 2nd to Nth levels, the sub-initial feature video blocks of the 1st level are subjected to cyclic scaling to obtain sub-intermediate feature video blocks, the resolution of which is the same as that of the sub-input video blocks. The process of synthesizing the intermediate feature video blocks to obtain the output video blocks specifically includes: The sub-intermediate feature video blocks are synthesized to obtain corresponding sub-output video blocks, wherein the resolution of the sub-output video blocks is the same as the resolution of the sub-input video blocks; and The sub-output video blocks corresponding to the plurality of sub-input video blocks are concatenated to form the output video block; Wherein, the relative position of the sub-output video block pixels in the output video block is the same as the relative position of the corresponding sub-input video block pixels in the input video block, and the step of concatenating the sub-output video blocks corresponding to the plurality of sub-input video blocks into the output video block includes: Initialize the initial output video matrix and the initial counting matrix, wherein the resolution of the initial output video matrix and the initial counting matrix are the same as the resolution of the output video block; The pixel values ​​of the multiple sub-output video blocks are added to the corresponding positions in the initial output video matrix using window functions to obtain the output video matrix. Each time a pixel value is added to the initial output video matrix, a floating-point number equal to the value of the window function is added to the element value corresponding to the initial counting matrix to obtain the counting matrix; and The corresponding elements of the output video matrix and the counting matrix are processed to generate the output video block.

18. A computer device, wherein, The device includes a processor and a memory, the memory storing computer-readable instructions, the processor being configured to execute the computer-readable instructions, which, when executed by the processor, perform a video block processing method according to any one of claims 1-8, or a neural network training method according to any one of claims 9-15.

19. A storage medium storing computer-readable instructions, wherein, When the computer-readable instructions are executed by a computer, the video block processing method according to any one of claims 1-8 is performed, or the neural network training method according to any one of claims 9-15 is performed.

Images