Video image processing method, device, electronic device and storage medium

Abstract

The present application proposes a video image processing method, device, electronic device, and storage medium, relating to the field of video image processing technology. The method comprises: inputting the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed, wherein the multi-scale optical flow motion estimation network is a network that characterizes optical flows and optical flow guidance information at different scales; inputting the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain a target image. The video image to be processed can be encoded based on motion estimation information at different scales, thereby reducing computational complexity, improving image processing efficiency, and shortening image processing time, so that the obtained target image can accurately reflect the motion trajectory of objects in the video image to be processed, thereby meeting the user's needs for video image usage.

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to a method, device, electronic device and storage medium for processing video images. Background Art

[0002] Currently, in the field of video coding, in order to obtain better video quality, it is necessary to segment video data into blocks in different modes and compress video data of different frames to ensure different application requirements.

[0003] However, in the process of encoding and segmenting video data, it is necessary to recursively traverse all possible division methods and select the processing mode with the lowest distortion cost to process the video data. After dividing different areas in the video data, different macroblocks (such as 32*32, etc.) are usually simply divided according to position information, which can easily lead to inaccurate motion estimation of objects in the video and the corresponding motion compensation cannot meet the image usage requirements.

[0004] Although the above processing process improves the video compression rate, it greatly increases the computational complexity and prolongs the processing time of the video file, which cannot meet the user's demand for real-time processing of video files. Summary of the Invention

[0005] The present application provides a method, device, electronic device and storage medium for processing video images.

[0006] An embodiment of the present application provides a method for processing a video image, the method comprising: inputting a video image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed, wherein the multi-scale optical flow motion estimation network is a network that characterizes optical flows and optical flow guidance information at different scales; inputting the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain a target image.

[0007] An embodiment of the present application provides a video image processing device, which includes: a motion estimation module, configured to input the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation, and obtain motion estimation information of the video image to be processed, where the multi-scale optical flow motion estimation network is a network that characterizes optical flows and optical flow guidance information at different scales; and an encoding module, configured to input the video image to be processed and its corresponding motion estimation information into an encoder for encoding, and obtain a target image.

[0008] An embodiment of the present application provides an electronic device, comprising: one or more processors; a memory on which one or more programs are stored. When the one or more programs are executed by one or more processors, the one or more processors implement any one of the video image processing methods in the embodiments of the present application.

[0009] An embodiment of the present application provides a readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements any one of the video image processing methods in the embodiments of the present application.

[0010] According to the video image processing method, device, electronic device and storage medium of the embodiments of the present application, by inputting the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation, the motion estimation information of the video image to be processed is obtained, which can make the motion estimation information of the video image to be processed reflect the motion estimation information of different scales corresponding to the video image to be processed, thereby facilitating the subsequent processing of the video image to be processed; the video image to be processed and its corresponding motion estimation information are input into the encoder for encoding, and the video image to be processed can be encoded based on the motion estimation information of different scales respectively, thereby reducing the computational complexity, improving the image processing efficiency, and reducing the image processing time, so that the obtained target image can accurately reflect the motion trajectory of the object in the video image to be processed, thereby meeting the user's requirements for the use of video images.

[0011] With respect to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart illustrating a method for processing a video image provided by an embodiment of the present application is shown.

[0013] Figure 2 A block diagram of a video image processing system provided by an embodiment of the present application is shown.

[0014] Figure 3 A schematic diagram of the structure of a multi-scale optical flow motion estimation network based on pyramid image determination provided by an embodiment of the present application is shown.

[0015] Figure 4 A schematic diagram of the training process of the multi-scale optical flow motion estimation network provided in an embodiment of the present application is shown.

[0016] Figure 5 A flowchart illustrating a method for processing video images provided in another embodiment of the present application is shown.

[0017] Figure 6A block diagram showing the composition of a video image processing device provided in an embodiment of the present application.

[0018] Figure 7 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the video image processing method and apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions and advantages of this application more clear, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any way.

[0020] In the High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC) protocols, the Coding Tree Unit (CTU) is usually used as the basic processing structure. The CTU can be further divided into Coding Units (CU). During intra-frame prediction or inter-frame prediction, the CTU and CU can also be divided into multiple Prediction Units (PUs), and the working parameters (such as coding mode) are shared between each PU. In the HEVC protocol, the CU can use a quadtree to represent its internal structure.

[0021] The unit partitioning methods supported by the HEVC protocol and the VVC protocol include: no partitioning, four-element partitioning, two binary partitionings, and two ternary partitionings (for example, 1 / 4, 2 / 4, 1 / 4 horizontal or vertical partitioning of the CU). In the VVC protocol, 64 geometric PU partitioning methods are also introduced to allow non-horizontal or non-vertical partitioning in rectangular CUs or square CUs. Each of the 64 geometric partitions is represented by an index value pointing to its parameters (for example, angle and / or distance, etc.). This geometric PU partitioning method cannot be applied to CUs with a width (or height) greater than 64 or a width (or height) less than 8. The VVC protocol also includes: a specific partitioning mode for intra sub-partitions (ISP). Multiple partitioning methods can bring higher partitioning flexibility.

[0022] While maintaining the same video quality, intra-frame coding implemented using the versatile video coding (VVC) protocol can improve image compression efficiency by approximately 50%. However, during the coding block compression process, the VVC protocol typically uses a block partitioning scheme that combines quadtrees, ternary trees, and binary trees. This block partitioning process requires recursively traversing all possible partitioning schemes and selecting the one with the lowest rate-distortion penalty as the target partitioning scheme. This significantly increases computational complexity, prolongs video file processing time, and reduces image compression rates, failing to meet users' demands for real-time video file processing.

[0023] Motion estimation is a key part of video encoding and decoding. However, in traditional encoding methods, motion estimation is processed based on PU. Moreover, after dividing different areas in the video data, different macroblocks are usually simply divided according to position information, which can easily lead to inaccurate motion estimation of objects in the video and the corresponding motion compensation cannot meet the usage requirements of the image.

[0024] Figure 1 FIG. 1 is a flow chart showing a method for processing a video image according to an embodiment of the present application. The method can be applied to a device for processing a video image. Figure 1 As shown, the video image processing method in the embodiment of the present application includes but is not limited to the following steps.

[0025] Step S110 : Inputting the video image to be processed into a multi-scale optical flow motion estimation network to perform motion estimation, thereby obtaining motion estimation information of the video image to be processed.

[0026] The multi-scale optical flow motion estimation network is a network that characterizes optical flow and optical flow guidance information at different scales. Optical flow (or optical flow) is a concept used to describe the motion of objects in video images, that is, the motion of observed objects, surfaces, or edges caused by the motion of the observer.

[0027] Step S120 : inputting the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain a target image.

[0028] In this embodiment, by inputting the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation, motion estimation information of the video image to be processed is obtained, so that the motion estimation information of the video image to be processed can reflect motion estimation information of different scales corresponding to the video image to be processed, thereby facilitating subsequent processing of the video image to be processed; the video image to be processed and its corresponding motion estimation information are input into the encoder for encoding, so that the video image to be processed can be encoded based on motion estimation information of different scales respectively, thereby reducing computational complexity, improving image processing efficiency, and reducing image processing time, so that the obtained target image can accurately reflect the motion trajectory of objects in the video image to be processed, thereby meeting the user's needs for video image usage.

[0029] In some specific implementations, before executing step S110 of inputting the video image to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed, the method further includes: obtaining sample optical flow data and sample video image data; pre-training the optical flow motion estimation network based on the endpoint error function and the sample optical flow data to obtain the network to be processed; and inputting the sample video image into the network to be processed for fine-tuning training to obtain the multi-scale optical flow motion estimation network.

[0030] The sample video images include multiple layers of sample images, each with a different image resolution. Optical flow data uses the temporal changes in pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. Generally speaking, optical flow data is generated by the movement of foreground objects in the scene, the movement of the camera, or both. Sample optical flow data can be obtained by manually annotating optical flow data.

[0031] The sample optical flow data is input into the optical flow motion estimation network for training, and an endpoint error function (End Point Error Loss) is used as a loss function to determine whether to terminate the pre-training of the optical flow motion estimation network. For example, the endpoint error function is used to calculate the two-dimensional Euclidean distance between the predicted optical flow of each pixel in the sample optical flow data and the pre-annotated optical flow, and it is determined whether the two-dimensional Euclidean distance is within a preset distance threshold range, thereby determining whether to terminate the pre-training of the optical flow motion estimation network.

[0032] When it is determined that the two-dimensional space Euclidean distance is within a preset distance threshold range (ie, the calculation result of the endpoint error function is in stable convergence), the network to be processed is obtained; otherwise, the pre-training process is continued.

[0033] After obtaining the network to be processed, it is also necessary to input sample video images into the network to be processed for fine-tuning training so that the fine-tuned network can meet the processing requirements of images of different scales.

[0034] In some specific implementations, step S120 of inputting the video image to be processed and its corresponding motion estimation information into the encoder for encoding to obtain the target image can be implemented as follows: inputting the video image to be processed and its corresponding motion estimation information into the encoder for encoding to obtain the encoded image; and obtaining the target image when it is determined that the encoded image meets the preset image quality evaluation index.

[0035] The preset image quality evaluation index includes at least one of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and encoding speed.

[0036] By evaluating the encoded image through a variety of different evaluation indicators, the encoded image can be reasonably evaluated, thereby obtaining a target image that meets user needs and improving the user experience.

[0037] Figure 2 FIG. 1 is a block diagram of a video image processing system according to an embodiment of the present application. Figure 2 As shown, the video image sequence 201 is input into the motion estimation network 202 for training to obtain motion estimation information corresponding to the video image sequence 201, and then the motion estimation information and the video image sequence 201 are input into the video encoder 203 for encoding, thereby obtaining a compressed video image sequence 204 output by the video encoder 203.

[0038] The motion estimation network 202 is a multi-scale optical flow motion estimation network capable of representing optical flows and optical flow guidance information at different scales. The motion estimation network 202 may include at least one or more of a convolution module, a deconvolution module, a linear rectifier function (ReLU) processing module, a neural network activation function (e.g., a sigmoid function) processing module, a fully connected layer (Full-Connection), and a reconstruction function (Reshape) processing module.

[0039] In some specific implementations, step S110 of inputting the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed can be implemented as follows:

[0040] The video images to be processed are layered according to the image resolution to obtain pyramid images corresponding to the video images to be processed; each layer of the images to be processed is input into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation parameters and residual values ​​corresponding to each layer of the images to be processed; the motion estimation information of the video images to be processed is determined based on the residual values ​​corresponding to each layer of the images to be processed and a preset residual threshold.

[0041] The pyramid image includes multiple layers of images to be processed, and each layer of images to be processed corresponds to a different image resolution.

[0042] It should be noted that different image resolutions correspond to different levels of the pyramid image, so that the images to be processed at different levels are input into the multi-scale optical flow motion estimation network for motion estimation. The obtained motion estimation parameters correspond to different image resolutions, that is, each layer of the image to be processed in the pyramid image corresponds to a motion estimation parameter and residual value.

[0043] Furthermore, the residual value corresponding to a layer of images to be processed is compared with a preset residual threshold to determine whether to terminate the motion estimation for that layer of images to be processed. For example, if the residual value corresponding to a layer of images to be processed is greater than the preset residual threshold, the motion estimation for that layer of images to be processed can be terminated and the motion estimation information corresponding to that layer of images to be processed can be obtained; otherwise, the motion estimation for that layer of images to be processed needs to continue.

[0044] After motion estimation has been completed for each layer of the to-be-processed images in the pyramid image, a comprehensive analysis can be performed based on the motion estimation information corresponding to each layer of the to-be-processed images to determine the motion estimation information for the to-be-processed video image. For example, the motion estimation information corresponding to each layer of the to-be-processed images can be combined to obtain the motion estimation information for the to-be-processed video image. This motion estimation information for the to-be-processed video image can include motion estimation information for multiple different image resolutions.

[0045] In some specific implementations, each layer of the image to be processed is input into the multi-scale optical flow motion estimation network for motion estimation, and the motion estimation parameters and residual values ​​corresponding to each layer of the image to be processed are obtained. This can be achieved in the following way:

[0046] The following processing is performed on each layer of the image to be processed: the image to be processed is input into the multi-scale optical flow motion estimation network for motion estimation to obtain the pixel-based optical flow data corresponding to the image to be processed: based on the number of macroblocks in the image to be processed and the motion vectors of multiple pixels in each macroblock, the mean of the motion vectors of multiple pixels in each macroblock in the image to be processed is determined; based on the motion vector corresponding to the macroblock and the mean of the motion vectors of multiple pixels in each macroblock, the residual value corresponding to the macroblock is determined.

[0047] Each layer of the image to be processed includes multiple macroblocks, each macroblock includes multiple pixels, the motion estimation parameters include motion vector mean values, and the pixel-based optical flow data includes motion vectors of multiple pixels in each macroblock in the image to be processed.

[0048] By inputting the image to be processed into a multi-scale optical flow motion estimation network for motion estimation, pixel-based optical flow data corresponding to the image to be processed is obtained, and pixel-level optical flow data (for example, motion vectors of multiple pixels in each macroblock in the image to be processed) can be obtained to improve the processing accuracy of the image to be processed. In addition, the number of macroblocks in the image to be processed and the motion vectors of multiple pixels in each macroblock can be calculated by adding and averaging, and the mean of the motion vectors of multiple pixels in each macroblock in the image to be processed can be obtained to reflect the directionality of motion in each macroblock. Furthermore, the motion vector corresponding to the macroblock is compared with the mean of the motion vectors of multiple pixels in each macroblock to determine whether the motion vector of the macroblock is the same as the mean of the motion vectors of multiple pixels in the macroblock, and the residual value corresponding to the macroblock is determined to facilitate subsequent processing of the macroblock.

[0049] The video image to be processed in step S110 includes at least: a first continuous sample image frame and a second continuous sample image frame. Figure 3 The schematic diagram of the structure of the multi-scale optical flow motion estimation network based on pyramid image determination provided by the embodiment of the present application is shown. Figure 3 As shown, F1 and F2 respectively represent two consecutive frame image sample images in the video image (for example, images with an image resolution of 512*512), wherein F2 is the current frame image and F1 is the previous frame image of F2.

[0050] Construct n-layer pyramid structures for F1 and F2 respectively (for example, Figure 3 As shown, n is equal to 3), a first pyramid image corresponding to F1 and a second pyramid image corresponding to F2 are obtained, the first pyramid and the second pyramid have the same number of layers, for example, the first pyramid image and the second pyramid image each include n layers of images to be processed.

[0051] Then, the k-th layer of the image to be processed in the first pyramid image and the k-th layer of the image to be processed in the second pyramid image are extracted respectively; the k-th layer of the image to be processed in the first pyramid image is input into a multi-scale optical flow motion estimation network for motion estimation, and motion estimation information corresponding to the k-th layer of the image to be processed in the second pyramid image is obtained; based on the motion estimation information corresponding to the k-th layer of the image to be processed in the second pyramid image and the k-th layer of the image to be processed in the second pyramid image, the pixel-based optical flow data corresponding to the k-th layer of the image to be processed is determined; and at the same time, the residual value corresponding to the k-th layer of the image to be processed is calculated.

[0052] Here, k is an integer greater than or equal to 1 and less than or equal to the number of layers of the first pyramid image (ie, n).

[0053] If the residual value corresponding to the image to be processed at the k-th layer is within the range of a preset residual threshold (for example, the residual value corresponding to the image to be processed at the k-th layer is greater than the preset residual threshold), the motion estimation information of the video image to be processed is determined based on the motion estimation parameters corresponding to the image to be processed at the k-1 layer; wherein the motion estimation parameters corresponding to the image to be processed at the k-1 layer may include: the mean of the motion vectors of multiple pixels in each macroblock in each layer of the image to be processed.

[0054] For example, if the kth layer is an image with an image resolution of 128*128, then the running vector means of multiple pixel points in each macroblock in the kth layer of the image to be processed (for example, an image with an image resolution of 32*32, or an image with an image resolution of 64*64, etc.) can be obtained.

[0055] It should be noted that when processing each layer of the image to be processed, the above steps need to be repeated until the residual values ​​corresponding to the images to be processed at all levels in the pyramid image meet the requirements of the preset residual threshold, and finally the mean value of the motion vector of multiple pixels in macroblocks of different sizes is obtained to reflect the running estimation parameters corresponding to the images to be processed at each layer, which is convenient for subsequent use.

[0056] In some specific implementations, the motion estimation information of the video image to be processed is determined based on the residual values ​​corresponding to the images to be processed in each layer and a preset residual threshold, including: when it is determined that the residual value corresponding to the image to be processed in the nth layer is within the range of the preset residual threshold, the motion estimation information of the video image to be processed is determined based on the motion estimation parameters corresponding to the image to be processed in the n-1 layer.

[0057] The motion estimation parameters corresponding to the image to be processed at the n-1 layer include: the mean value of the motion vectors of multiple pixels in each macroblock in each layer of the image to be processed, n is an integer greater than 1 and less than or equal to the number of layers of the pyramid image, and the image to be processed at the first layer is the image to be processed at the initial layer.

[0058] Figure 4 FIG. 1 is a schematic diagram showing the training process of the multi-scale optical flow motion estimation network provided in the embodiment of the present application. Figure 4 As shown, the training network of the multi-scale optical flow motion estimation network includes but is not limited to the following modules.

[0059] Forward function module (previous feature) 401, Deformable Convolutional Networks (DCN) 402, feature deformation (Warped feature) module 403, optical flow data 404, LR feature module 405, residual offset calculation (Residual offsets) module 406, DCN offset (DCN offsets) module 407, DCN identification (DCN masks) module 408, and the first convolution kernel (Conv C 0 )411 and the mth convolution kernel (Conv C m )41m, where m represents the number of convolution kernels and is an integer greater than or equal to 1.

[0060] It should be noted that the diversity of offsets in DCN enables deformation alignment to outperform optical flow data alignment. However, deformation alignment is difficult to train. During network training, the instability of training results can easily lead to divergence of offsets, which in turn affects the performance of the network model. To fully utilize the diversity of offsets and overcome the instability of network training, based on the strong correlation between deformation alignment and optical flow alignment, optical flow data 404 is used to guide the deformation data to obtain good output data.

[0061] For example, limiting offsets to a preset range of optical flow data can limit offset overflow and greatly increase the stability of network training.

[0062] In the process of the jth time compensation, if the deformation feature output by the LR feature module 405 is g i , the feature output by the forward function module (previous feature) 401 is f i-1 ; Optical flow data 404 is represented by s i→i-1 , (i.e., the optical flow data moving from the i-th position to the i-1-th position), then the f i-1 Perform affine transformation to obtain the transformation result The affine transformation process can be expressed by the following formula:

[0063]

[0064] Furthermore, the DCN offset module 407 outputs the offset o i→i-1 , and modulates the mask m through the DCN identification module 408 i→i-1 , thereby completing the pre-alignment of features. Wherein, i is an integer greater than 1.

[0065] In the above process, by calculating the optical flow residual rather than directly calculating the offset, offset-based learning can assist in feature alignment by using the optical flow data 404; while the convolutional neural network only learns the residual, it can reduce the burden on the conventional deformation alignment module. The DCN identification module 408 also acts as an attention mechanism, making the trained network more flexible.

[0066] Figure 5 FIG. 1 is a flow chart showing a method for processing a video image according to another embodiment of the present application. Figure 5 As shown, the video image processing method in the embodiment of the present application includes but is not limited to the following steps.

[0067] Step S501: Obtain sample optical flow data and sample video image data.

[0068] The sample video image data includes: multiple layers of sample images, each layer of sample images has a different image resolution. For example, the multiple sample video images are divided based on a preset number to obtain training set data and test set data.

[0069] The sample optical flow data may include: data obtained by downloading public datasets from the Internet, and / or data obtained by manually annotating optical flows.

[0070] Step S502 : Pre-training the optical flow motion estimation network based on the endpoint error function and the sample optical flow data to obtain a network to be processed.

[0071] The optical flow motion estimation network may include a convolutional neural network (e.g., an optical flow network (FlowNet)). The above optical flow motion estimation network is merely an example and may be configured specifically based on actual needs. Other optical flow motion estimation networks not described herein are also within the scope of protection of this application and will not be described in detail here.

[0072] In some specific implementations, an endpoint error function can be used to calculate the two-dimensional Euclidean distance between the predicted optical flow of each pixel in the sample optical flow data and the pre-labeled optical flow. The two-dimensional Euclidean distance is used to determine whether the pre-training of the optical flow motion estimation network is completed.

[0073] When it is determined that the two-dimensional space Euclidean distance is within a preset distance threshold range (ie, the calculation result of the endpoint error function is in stable convergence), the network to be processed is obtained; otherwise, the pre-training process is continued.

[0074] Step S503: Input the sample video image into the network to be processed for fine-tuning training to obtain a multi-scale optical flow motion estimation network.

[0075] Among them, fine-tuning training is to input sample video images into the network to be processed for training, for example, using the mean square error (MSE) loss function and / or the L1 loss function to fine-tune the training results. When the MSE loss function and / or the L1 loss function converge stably, a multi-scale optical flow motion estimation network is obtained; otherwise, the fine-tuning training process continues.

[0076] It should be noted that the L1 loss function is used to minimize the error, which is the sum of all absolute differences between the true value and the predicted value.

[0077] Step S504: input the video image to be processed into a multi-scale optical flow motion estimation network to perform motion estimation, and obtain motion estimation information of the video image to be processed.

[0078] Step S505 : Connect the multi-scale optical flow motion estimation network to the encoder, so that the video image to be processed and its corresponding motion estimation information are input into the encoder for encoding to obtain an encoded image.

[0079] Step S506: When it is determined that the encoded image meets the preset image quality evaluation index, a target image is obtained.

[0080] It should be noted that if the encoded image cannot meet the preset image quality evaluation index, the multi-scale optical flow motion estimation network needs to be trained until the encoded image meets the preset image quality evaluation index.

[0081] The preset image quality evaluation indicators include: image quality indicators (such as PSNR, SSIM, etc.), compression performance indicators (such as coding compression ratio, coding speed, etc.) and any one or more of the weighted sum values ​​of the above indicators.

[0082] In this embodiment, by using an optical flow network for pixel-level inter-frame prediction motion estimation, the accuracy of the prediction unit's motion estimation and motion compensation is effectively improved, making the obtained motion estimation information of the processed video image more accurate. Furthermore, the use of a multi-scale optical flow network prediction method can reduce computational complexity, thereby improving image processing efficiency, making the inter-frame prediction process more accurate, and enhancing encoding and decoding efficiency.

[0083] The following describes in detail a video image processing device according to an embodiment of the present application with reference to the accompanying drawings. Figure 6 FIG. 1 is a block diagram showing the composition of the video image processing device provided by the embodiment of the present application. Figure 6 As shown, the video image processing device 600 includes but is not limited to the following modules.

[0084] The motion estimation module 601 is configured to input the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed. The multi-scale optical flow motion estimation network is a network that characterizes optical flows and optical flow guidance information at different scales.

[0085] The encoding module 602 is configured to input the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain a target image.

[0086] In some specific implementations, the motion estimation module 601 is specifically used to: layer the video image to be processed according to the image resolution to obtain a pyramid image corresponding to the video image to be processed, wherein the pyramid image includes multiple layers of images to be processed, and the image resolution corresponding to each layer of images to be processed is different; input each layer of images to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation parameters and residual values ​​corresponding to each layer of images to be processed; and determine motion estimation information of the video image to be processed based on the residual values ​​corresponding to each layer of images to be processed and a preset residual threshold.

[0087] In some specific implementations, each layer of the image to be processed includes multiple macroblocks, each macroblock includes multiple pixels, and the motion estimation parameters include a mean motion vector. Each layer of the image to be processed is input into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation parameters and residual values ​​corresponding to each layer of the image to be processed, including: performing the following processing on each layer of the image to be processed: inputting the image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain pixel-based optical flow data corresponding to the image to be processed, wherein the pixel-based optical flow data includes motion vectors for multiple pixels in each macroblock in the image to be processed; determining the mean motion vector for multiple pixels in each macroblock in the image to be processed based on the number of macroblocks in the image to be processed and the motion vectors for multiple pixels in each macroblock; and determining the residual value corresponding to the macroblock based on the motion vector corresponding to the macroblock and the mean motion vector for multiple pixels in each macroblock.

[0088] In some specific implementations, a video image to be processed includes at least: a first frame sample image and a second frame sample image, wherein the first frame sample image corresponds to a first pyramid image, the second frame sample image corresponds to a second pyramid image, and the first pyramid image and the second pyramid image have the same number of layers; inputting the image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain pixel-based optical flow data corresponding to the image to be processed, including: extracting the k-th layer of the image to be processed in the first pyramid image and the k-th layer of the image to be processed in the second pyramid image; inputting the k-th layer of the image to be processed in the first pyramid image into the multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information corresponding to the k-th layer of the image to be processed in the second pyramid image; and determining the pixel-based optical flow data corresponding to the k-th layer of the image to be processed based on the motion estimation information corresponding to the k-th layer of the image to be processed in the second pyramid image and the k-th layer of the image to be processed in the second pyramid image.

[0089] Here, k is an integer greater than or equal to 1 and less than or equal to the number of layers of the first pyramid image.

[0090] In some specific implementations, determining the motion estimation information of the video image to be processed based on the residual value corresponding to each layer of the image to be processed and a preset residual threshold includes:

[0091] When it is determined that the residual value corresponding to the image to be processed at the nth layer is within the range of the preset residual threshold, the motion estimation information of the video image to be processed is determined based on the motion estimation parameters corresponding to the image to be processed at the n-1th layer;

[0092] The motion estimation parameters corresponding to the image to be processed at the n-1 layer include: the mean value of the motion vectors of multiple pixels in each macroblock in each layer of the image to be processed, n is an integer greater than 1 and less than or equal to the number of layers of the pyramid image, and the image to be processed at the first layer is the image to be processed at the initial layer.

[0093] In some specific implementations, the video image processing device also includes: an acquisition module for acquiring sample optical flow data and sample video image data; a pre-training module for pre-training the optical flow motion estimation network based on the endpoint error function and the sample optical flow data to obtain a network to be processed; and a fine-tuning training module for inputting the sample video image into the network to be processed for fine-tuning training to obtain a multi-scale optical flow motion estimation network.

[0094] The sample video images include multiple layers of sample images, and each layer of sample images has a different image resolution.

[0095] In some specific implementations, the encoding module 602 is specifically used to: input the video image to be processed and its corresponding motion estimation information into the encoder for encoding to obtain an encoded image; and obtain a target image when it is determined that the encoded image meets the preset image quality evaluation index.

[0096] In some specific implementations, the preset image quality evaluation index includes at least one of peak signal-to-noise ratio, image similarity, and encoding speed.

[0097] It should be noted that the video image processing device 600 in this embodiment can implement any video image processing method in the embodiments of the present application.

[0098] According to the device of the embodiment of the present application, the motion estimation module inputs the video image to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed, which can enable the motion estimation information of the video image to be processed to reflect the motion estimation information of different scales corresponding to the video image to be processed, thereby facilitating subsequent processing of the video image to be processed; the encoding module inputs the video image to be processed and its corresponding motion estimation information into the encoder for encoding, which can encode the video image to be processed based on the motion estimation information of different scales respectively, thereby reducing computational complexity, improving image processing efficiency, and reducing image processing time, so that the obtained target image can accurately reflect the motion trajectory of the object in the video image to be processed, thereby meeting the user's requirements for the use of video images.

[0099] It should be understood that the present application is not limited to the specific configurations and processes described in the above embodiments and illustrated in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here. The specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] Figure 7 A structural diagram showing an exemplary hardware architecture of a computing device capable of implementing the video image processing method and apparatus according to an embodiment of the present application.

[0101] like Figure 7 As shown, computing device 700 includes an input device 701, an input interface 702, a central processing unit 703, a memory 704, an output interface 705, and an output device 706. The input interface 702, the central processing unit 703, the memory 704, and the output interface 705 are interconnected via a bus 707. The input device 701 and the output device 706 are connected to the bus 707 via the input interface 702 and the output interface 705, respectively, and are further connected to other components of computing device 700.

[0102] Specifically, the input device 701 receives input information from the outside and transmits the input information to the central processing unit 703 through the input interface 702; the central processing unit 703 processes the input information based on the computer-executable instructions stored in the memory 704 to generate output information, stores the output information temporarily or permanently in the memory 704, and then transmits the output information to the output device 706 through the output interface 705; the output device 706 outputs the output information to the outside of the computing device 700 for user use.

[0103] In one embodiment, Figure 7 The computing device shown may be implemented as an electronic device, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the video image processing method described in the above embodiment.

[0104] In one embodiment, Figure 7 The computing device shown can be implemented as a video image processing system, which may include: a memory configured to store a program; and a processor configured to run the program stored in the memory to execute the video image processing method described in the above embodiment.

[0105] The foregoing is merely an exemplary embodiment of the present application and is not intended to limit the scope of protection of the present application. Generally speaking, various embodiments of the present application can be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although the present application is not limited thereto.

[0106] Embodiments of the present application may be implemented by executing computer program instructions by a data processor of a mobile device, for example, in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0107] The block diagram of any logical flow in the accompanying drawings of the present application can represent program steps, or can represent interconnected logical circuits, modules and functions, or can represent a combination of program steps and logical circuits, modules and functions. The computer program can be stored on a memory. The memory can have any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical memory device and system (digital versatile disc DVD or CD optical disc) etc. Computer-readable media can include non-transient storage media. The data processor can be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a programmable logic device (FPGA) and a processor based on a multi-core processor architecture.

[0108] The above description of exemplary embodiments of the present application has been provided by way of exemplary and non-limiting examples. However, various modifications and adaptations of the above embodiments will be apparent to those skilled in the art, when considered in conjunction with the accompanying drawings and the appended claims, without departing from the scope of the present application. Therefore, the proper scope of the present application will be determined by reference to the appended claims.

Claims

1. A method for processing a video image, characterized in that: The method comprises: Inputting the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed, wherein the multi-scale optical flow motion estimation network is a network that characterizes optical flows and optical flow guidance information at different scales; Inputting the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain a target image; Inputting the video image to be processed into a multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation information of the video image to be processed includes: Layering the video image to be processed according to image resolution to obtain a pyramid image corresponding to the video image to be processed, wherein the pyramid image includes multiple layers of images to be processed, and each layer of the images to be processed corresponds to a different image resolution; Inputting the to-be-processed images of each layer into the multi-scale optical flow motion estimation network for motion estimation, and obtaining motion estimation parameters and residual values ​​corresponding to the to-be-processed images of each layer; Determining motion estimation information of the video image to be processed based on the residual value corresponding to the image to be processed in each layer and a preset residual threshold; Each layer of the image to be processed includes a plurality of macroblocks, each of the macroblocks includes a plurality of pixels, and the motion estimation parameter includes a motion vector mean; Inputting each layer of the image to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation parameters and residual values ​​corresponding to the image to be processed in each layer includes: The following processing is performed on each layer of the image to be processed: Inputting the image to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain pixel-based optical flow data corresponding to the image to be processed, wherein the pixel-based optical flow data includes motion vectors of multiple pixels in each macroblock in the image to be processed: Determining a mean value of the motion vectors of the plurality of pixels in each macroblock in the image to be processed according to the number of macroblocks in the image to be processed and the motion vectors of the plurality of pixels in each macroblock; The residual value corresponding to the macroblock is determined according to the motion vector corresponding to the macroblock and the average value of the motion vectors of multiple pixels in each macroblock.

2. The method according to claim 1, characterized in that The video image to be processed includes at least: a first frame sample image and a second frame sample image, wherein the first frame sample image corresponds to a first pyramid image, the second frame sample image corresponds to a second pyramid image, and the first pyramid image and the second pyramid image have the same number of layers; Inputting the image to be processed into the multi-scale optical flow motion estimation network to perform motion estimation, and obtaining pixel-based optical flow data corresponding to the image to be processed, includes: extracting the k-th layer of to-be-processed image in the first pyramid image and the k-th layer of to-be-processed image in the second pyramid image respectively; Inputting the k-th layer of the image to be processed in the first pyramid image into the multi-scale optical flow motion estimation network to perform motion estimation, and obtaining motion estimation information corresponding to the k-th layer of the image to be processed in the second pyramid image; determining pixel-based optical flow data corresponding to the k-th layer of the image to be processed in the second pyramid image based on motion estimation information corresponding to the k-th layer of the image to be processed in the second pyramid image and the k-th layer of the image to be processed in the second pyramid image; Here, k is an integer greater than or equal to 1 and less than or equal to the number of layers of the first pyramid image.

3. The method according to claim 1, characterized in that The determining of the motion estimation information of the video image to be processed based on the residual value corresponding to the image to be processed at each layer and a preset residual threshold comprises: When it is determined that the residual value corresponding to the image to be processed at the nth layer is within the range of the preset residual threshold, determining the motion estimation information of the video image to be processed based on the motion estimation parameter corresponding to the image to be processed at the n-1th layer; The motion estimation parameters corresponding to the image to be processed at the n-1 layer include: a mean value of motion vectors of multiple pixels in each macroblock in the image to be processed at each layer, n is an integer greater than 1 and less than or equal to the number of layers of the pyramid image, and the image to be processed at the first layer is the image to be processed at the initial layer.

4. The method according to any one of claims 1 to 3, characterized in that Before inputting the video image to be processed into the multi-scale optical flow motion estimation network for motion estimation and obtaining the motion estimation information of the video image to be processed, the method further includes: Obtain sample optical flow data and sample video image data; Pre-training an optical flow motion estimation network based on an endpoint error function and the sample optical flow data to obtain a network to be processed; The sample video image is input into the network to be processed for fine-tuning training to obtain the multi-scale optical flow motion estimation network; wherein the sample video image includes multiple layers of sample images, and the image resolution corresponding to each layer of the sample images is different.

5. The method according to any one of claims 1 to 3, characterized in that The step of inputting the video image to be processed and the corresponding motion estimation information into an encoder for encoding to obtain a target image includes: Inputting the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain an encoded image; When it is determined that the encoded image meets a preset image quality evaluation index, the target image is obtained.

6. The method according to claim 5, characterized in that The preset image quality evaluation index includes at least one of peak signal-to-noise ratio, image similarity and encoding speed.

7. A video image processing device, characterized in that: It includes: a motion estimation module configured to input a video image to be processed into a multi-scale optical flow motion estimation network for motion estimation, thereby obtaining motion estimation information of the video image to be processed, wherein the multi-scale optical flow motion estimation network is a network that characterizes optical flows and optical flow guidance information at different scales; An encoding module is configured to input the video image to be processed and its corresponding motion estimation information into an encoder for encoding to obtain a target image; The motion estimation module is specifically configured to: layer the video image to be processed according to image resolution to obtain a pyramid image corresponding to the video image to be processed, wherein the pyramid image includes multiple layers of images to be processed, and each layer of the images to be processed has a different image resolution; input each layer of the images to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation parameters and residual values ​​corresponding to each layer of the images to be processed; and determine motion estimation information of the video image to be processed based on the residual values ​​corresponding to each layer of the images to be processed and a preset residual threshold; Each layer of the image to be processed includes a plurality of macroblocks, each of the macroblocks includes a plurality of pixels, and the motion estimation parameter includes a motion vector mean; Inputting each layer of the image to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain motion estimation parameters and residual values ​​corresponding to the image to be processed in each layer includes: The following processing is performed on each layer of the image to be processed: Inputting the image to be processed into the multi-scale optical flow motion estimation network for motion estimation to obtain pixel-based optical flow data corresponding to the image to be processed, wherein the pixel-based optical flow data includes motion vectors of multiple pixels in each macroblock in the image to be processed: Determining a mean value of the motion vectors of the plurality of pixels in each macroblock in the image to be processed according to the number of macroblocks in the image to be processed and the motion vectors of the plurality of pixels in each macroblock; The residual value corresponding to the macroblock is determined according to the motion vector corresponding to the macroblock and the average value of the motion vectors of multiple pixels in each macroblock.

8. An electronic device, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the video image processing method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that The readable storage medium stores a computer program, and when the computer program is executed by a processor, the video image processing method according to any one of claims 1 to 6 is implemented.

Images