Video processing method and device and electronic equipment

Abstract

The invention provides a video processing method and apparatus, and an electronic device. The method comprises the steps of obtaining a first video; based on a video processing model, the first video is processed to obtain a second video, the training stage of the video processing model comprises a differentiable encoder, the differentiable encoder is used for simulating the process of quantizing and encoding the video by the encoder, and the differentiable encoder can perform gradient return; and carrying out coding processing on the second video.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of video processing technology, and in particular to a video processing method, device and electronic device. Background Art

[0002] Before sending the video to the terminal device, the server can pre-process the video to improve the video playback effect.

[0003] Currently, the server can pre-process the video based on the video processing model. For example, the server can perform super-resolution processing on the video based on the trained video super-resolution model to improve the video clarity. However, before the server sends the processed video to the terminal device, it needs to encode the video. The encoding process will lose some information in the video, resulting in poor accuracy of video processing. Summary of the invention

[0004] The present disclosure provides a video processing method, device and electronic device, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides a video processing method, the method comprising:

[0006] Get the first video;

[0007] Based on a video processing model, the first video is processed to obtain a second video, wherein the training phase of the video processing model includes a differentiable encoder, the differentiable encoder is used to simulate a process in which an encoder quantizes and encodes a video, and the differentiable encoder can perform gradient feedback;

[0008] The second video is encoded.

[0009] In a second aspect, the present disclosure provides a video processing device, the video processing device comprising an acquisition module, a processing module and an encoding module, wherein:

[0010] The acquisition module is used to acquire the first video;

[0011] The processing module is used to process the first video based on a video processing model to obtain a second video, wherein the training phase of the video processing model includes a differentiable encoder, the differentiable encoder is used to simulate the process of quantizing and encoding the video by the encoder, and the differentiable encoder can perform gradient feedback;

[0012] The encoding module is used to encode the second video.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the video processing method as described in the first aspect and various possible aspects of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the video processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0017] The present disclosure provides a video processing method, device and electronic device, wherein the electronic device can obtain a first video, and based on a video processing model, process the first video to obtain a second video, and the electronic device can encode the second video, wherein the training phase of the video processing model includes a differentiable encoder, and the differentiable encoder is used to simulate the process of the encoder quantizing and encoding the video, and the differentiable encoder can perform gradient feedback. In the above method, since the differentiable encoder can perform gradient feedback, during the training phase of the video processing model, the video processing model can learn the distortion information after the encoder encodes the video, so that the accuracy of the video processing model can be improved, and the accuracy of video processing can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic diagram of the structure of a video processing model training phase provided by an embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of a training method for a video processing model provided in an embodiment of the present disclosure;

[0023] Figure 5A schematic diagram of an image block provided in an embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of determining multiple candidate image blocks provided by an embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of a training process of a video processing model provided in an embodiment of the present disclosure;

[0026] Figure 8 A schematic diagram of an image encoding processing method provided by an embodiment of the present disclosure;

[0027] Fig. 9 A schematic diagram of a process for determining a second residual image provided by an embodiment of the present disclosure;

[0028] Fig.10 A schematic diagram of a process for generating a second image provided by an embodiment of the present disclosure;

[0029] Fig.11 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure; and

[0030] Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0032] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0033] Electronic device: is a device with wireless transceiver function. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted. The electronic device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a vehicle-mounted electronic device, a wireless terminal in self-driving, a wireless electronic device in remote medical care, a wireless electronic device in smart grid, a wireless electronic device in transportation safety, a wireless electronic device in smart city, a wireless electronic device in smart home, a wearable electronic device, etc. The electronic device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access electronic device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote electronic device, a mobile device, a UE electronic device, a wireless communication device, a UE agent or a UE device, etc. The electronic device can also be fixed or mobile.

[0034] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0035] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including: a server, a terminal device and video 1. The server may include a super-resolution model. The server may process video 1 based on the super-resolution model to obtain a super-resolution video 2 (the definition of video 2 is greater than that of video 1), and the server may encode video 2 to obtain the encoding of video 2. The server may send the encoding of video 2 to the terminal device, and after the terminal device receives the encoding of video 2, it may decode the encoding of video 2 and then play video 2. In this way, before sending the video to the terminal device, the server may improve the definition of the video, thereby improving the playback effect of the video.

[0036] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0037] In the related art, the pre-processing of the video can effectively improve the quality of the video. For example, before the server sends the video to the terminal device, the video can be processed by super-resolution, which can improve the clarity of the video. At present, the server can pre-process the video based on the video processing model. For example, the server can perform super-resolution processing on the video based on the trained video super-resolution model, and then obtain a super-resolution video, wherein the clarity of the super-resolution video is greater than the clarity of the video before the super-resolution processing. However, before the server sends the processed video to the terminal device, the video needs to be encoded, and the encoding process will lose some information in the video. The video processing model does not learn the distortion information of the video encoding during the training stage, which leads to poor accuracy of video processing.

[0038] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides a video processing method, an electronic device can obtain a first video, and based on a video processing model, process the first video to obtain a second video, and the electronic device can encode the second video, wherein the video processing model is trained based on the following steps: obtaining a sample video, a target video corresponding to the sample video, and a target bit rate of the target video, processing the sample video based on the video processing model to obtain a first predicted video, processing the first predicted video based on a differentiable encoder to obtain a second predicted video and a predicted bit rate, and training the video processing model based on the second predicted video, the target video, the predicted bit rate, and the target bit rate. In this way, since the differentiable encoder can simulate the process of the encoder quantizing and encoding the video, and the differentiable encoder can perform gradient backpropagation, the differentiable encoder can assist the video processing model in learning the distortion information after video encoding, improve the training accuracy of the video processing model, and thus improve the accuracy of video processing.

[0039] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0040] Figure 2 A flowchart of a video processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0041] S201, obtaining a first video.

[0042] The execution subject of the embodiment of the present disclosure may be an electronic device, or a video processing device arranged in the electronic device. The video processing device may be implemented based on software, or may be implemented based on a combination of software and hardware, which is not limited in the embodiment of the present disclosure.

[0043] Optionally, the electronic device may be any device with on-end computing capability. For example, the electronic device may be a server, a computer, etc., which is not limited in the embodiments of the present disclosure.

[0044] Optionally, the first video may be a video to be processed. For example, the first video may be a video to be processed with super resolution, or a video to be tracked, etc., which is not limited in the embodiments of the present disclosure.

[0045] Optionally, the electronic device may receive the first video sent by other devices. For example, after the terminal device captures a video, it may send the captured video to the server, and the server may determine the video as the first video.

[0046] Optionally, the electronic device may obtain the first video from a database. For example, the database may store multiple videos in advance, and when the server needs to process the videos, the multiple first videos may be obtained from the database.

[0047] It should be noted that the electronic device can obtain the first video based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0048] S202: Process the first video based on a video processing model to obtain a second video.

[0049] The video processing model can be used to process the video. For example, if the video processing model is a super-resolution model, the video processing model can perform super-resolution processing on the video; if the video processing model is an object tracking model, the video processing model can track objects in the video, etc.

[0050] It should be noted that the video processing model in the embodiments of the present disclosure may be a model with any video processing function, and the embodiments of the present disclosure are not limited to this.

[0051] The second video may be a video processed from the first video. For example, the video processing model may be a pre-trained model, and after the electronic device inputs the first video into the video processing model, the second video corresponding to the first video may be obtained. For example, if the video processing model is a super-resolution model, the electronic device may input the first video into the video processing model, and the video processing model may perform super-resolution processing on the first video to obtain the second video, wherein the definition of the second video is greater than the definition of the first video.

[0052] The training stage of the video processing model includes a differentiable encoder. For example, the training stage of the video processing model may include a differentiable encoder. Since the differentiable encoder can perform model training, the video processing model can learn information about coding distortion based on the output results of the differentiable encoder during the training stage, thereby improving the training accuracy of the video processing model.

[0053] Among them, the differentiable encoder can be used to simulate the process of the encoder quantizing and encoding the video, and the differentiable encoder can perform gradient feedback. For example, the differentiable encoder can simulate the process of the encoder encoding the video, so the electronic device can obtain the loss caused by the encoding of the video based on the differentiable encoder, and because the differentiable encoder can perform gradient feedback, when training the video processing model, the differentiable encoder can perform gradient feedback processing to the video processing model, and then train the model parameters in the video processing model to improve the accuracy of the model training.

[0054] Next, combine Figure 3 , explaining the structure of the video processing model training phase.

[0055] Figure 3 This is a schematic diagram of the structure of a video processing model training phase provided by an embodiment of the present disclosure. Figure 3 , including: a video processing model and a differentiable encoder. The output end of the video processing model can be connected to the input end of the differentiable encoder, so that during the video processing model training process, since the differentiable encoder can perform gradient feedback, the video processing model can learn the distortion information of the video encoding, thereby improving the training accuracy of the video processing model, and when the video processing model is used, the accuracy of video processing can be improved.

[0056] S203: Encode the second video.

[0057] Optionally, after the electronic device acquires the second video, it can encode the second video. For example, the electronic device can encode the second video based on any feasible standard encoding method, and after encoding the second video, the electronic device can send the encoding of the second video to the terminal device, and after receiving the encoding of the second video, the terminal device can decode the encoding and play the second video.

[0058] The disclosed embodiment provides a video processing method, wherein an electronic device can obtain a first video, and based on a video processing model, process the first video to obtain a second video, and the electronic device can encode the second video. The training stage of the video processing model includes a differentiable encoder. Since the differentiable encoder can simulate the process of quantizing and encoding the video by the encoder, and the differentiable encoder can perform gradient feedback, during the training stage of the video processing model, the video processing model can learn the distortion information after the encoder encodes the video. In this way, the accuracy of the video processing model can be improved, and the accuracy of video processing can be improved.

[0059] exist Figure 2 Based on the embodiment shown below, combined with Figure 4 The above video processing method also includes a method for training a video processing model. Figure 4 , the training method of the video processing model is explained in detail.

[0060] Figure 4 A schematic diagram of a video processing model training method provided by an embodiment of the present disclosure. Figure 4 The method flow includes:

[0061] S401, obtaining a sample video, a target video corresponding to the sample video, and a target bit rate of the target video.

[0062] Optionally, the sample video may be any video. For example, if the video processing model is a super-resolution model, the sample video may be a low-resolution video; if the video processing model is an object tracking model, the sample video may include an object to be tracked.

[0063] It should be noted that the electronic device may obtain the sample video based on the function of the video processing model, or may obtain the sample video based on any feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0064] The target video may be a video corresponding to the sample video. For example, if the video processing model is a super-resolution model, the sample video may be a low-resolution video, and the target video may be a high-resolution video corresponding to the sample video, wherein the video content in the target video is the same as the video content in the sample video.

[0065] Optionally, the target bitrate may be the bitrate of the target video. For example, during the video processing model training process, based on the target bitrate of the target video, the video processing model can learn the coding distortion information at different bitrates, thereby improving the training effect of the video processing model.

[0066] It should be noted that the electronic device can obtain the sample video, the target video corresponding to the sample video, and the target bit rate of the target video based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0067] S402: Process the sample video based on the video processing model to obtain a first predicted video.

[0068] The first predicted video may be a video obtained after the video processing model processes the sample video. Optionally, the electronic device may input the sample video to the video processing model, and the video processing model may output the first predicted video corresponding to the sample video.

[0069] For example, if the video processing model is a super-resolution model, after the electronic device inputs a sample video into the video processing model, the video processing model can predict the video after super-resolution processing of the sample video based on the sample video, and thereby obtain a first predicted video.

[0070] S403 . Process the first predicted video based on a differentiable encoder to obtain a second predicted video and a predicted bit rate.

[0071] The second predicted video may be a video obtained after the differentiable encoder encodes the first predicted video. For example, since the differentiable encoder can simulate the process of quantizing and encoding the video by the video encoder, the differentiable encoder can obtain the distortion information generated when the first predicted video is encoded, and add the distortion information to the first predicted video to obtain the second predicted video.

[0072] The predicted bit rate may be a bit rate at which a differentiable encoder encodes the first predicted video. For example, the differentiable encoder includes an entropy encoding module, and the predicted code may be obtained by processing the quantized information based on the entropy encoding module.

[0073] Among them, the electronic device can obtain the second predicted video and the predicted bit rate based on the following feasible implementation method: perform image division processing on each image in the first predicted video to obtain multiple image blocks corresponding to each image, determine the encoding method corresponding to each image in the first predicted video, and based on a differentiable encoder, the encoding method corresponding to each image and the multiple image blocks corresponding to each image, perform encoding processing on each image to obtain the second predicted video and the predicted bit rate.

[0074] The encoding method may include intra-frame prediction encoding and inter-frame prediction encoding. For example, intra-frame prediction encoding may be an electronic device encoding the image to be encoded in the current frame image based on pixel information in the current frame image; inter-frame prediction encoding may be an electronic device encoding the image to be encoded in the current frame image based on pixel information in adjacent frame images.

[0075] It should be noted that the electronic device can determine the encoding method corresponding to each image in the first predicted video based on any feasible implementation method (for example, pre-setting the first image as intra-frame prediction encoding, the 50th image as intra-frame prediction encoding, and other images as inter-frame prediction encoding, etc.), and the embodiments of the present disclosure are not limited to this.

[0076] Optionally, each image in the first predicted video may include multiple image blocks. For example, for any image in the first video, the electronic device may divide the first video into multiple image blocks based on the image information, wherein the sizes of the multiple image blocks may be the same or different, which is not limited in the embodiments of the present disclosure.

[0077] Optionally, the electronic device may process each image in the first prediction video based on a pre-trained image segmentation model to obtain a plurality of image blocks corresponding to each image. For example, the electronic device may input the images in the first prediction video into the pre-trained image segmentation model, and the image segmentation model may perform image segmentation processing on each image.

[0078] Next, combine Figure 5 , describe the image block corresponding to the image.

[0079] Figure 5 A schematic diagram of an image block provided by an embodiment of the present disclosure. Figure 5 , including: images and image segmentation models. Electronic devices ( Figure 5 (not shown) can input an image into the image division model, and the image division model can divide the image into image blocks. Among them, the image division model can divide the image into 10 image blocks, image blocks A, image blocks B and image blocks C have the same size, image blocks D, image blocks E and image blocks F have the same size, and image blocks G, image blocks H, image blocks I and image blocks J have the same size. In this way, the electronic device can combine the image information to accurately divide the image into multiple image blocks, thereby improving the efficiency of the differentiable encoder in encoding the image.

[0080] It should be noted that, before the electronic device performs image segmentation processing on the images in the first predicted video, each image may be converted into a YUV space (color space), thereby improving the image processing effect.

[0081] The electronic device encodes any first image in the first predicted video based on a differentiable encoder, an encoding method corresponding to each image, and a plurality of image blocks corresponding to each image to obtain a second predicted video and a predicted bit rate. There are two cases as follows:

[0082] Case 1: The encoding method is intra-frame prediction encoding.

[0083] If the encoding method is intra-frame prediction encoding, the electronic device can determine multiple candidate image blocks corresponding to each image block in the first image, and encode the first image based on each image block and the candidate image blocks corresponding to each image block.

[0084] Optionally, for any image block in the first image, if the encoding method is intra-frame prediction encoding, the electronic device can encode the image block based on the pixel information adjacent to the image block. In this way, the electronic device can determine the pixel value in the current frame, predict the value of each pixel in the current frame, and thus reduce the redundant information of adjacent pixels in the frame, that is, predict the value of the current pixel based on the pixel values ​​around the current pixel, thereby improving the encoding effect.

[0085] The following describes a process in which an electronic device determines a plurality of candidate image blocks corresponding to an image block through a specific example.

[0086] Optionally, the size of the image block is N (an integer greater than 0), the area to be encoded is N*N, and the electronic device can use the N*N area adjacent to the top, upper left, left, lower left and upper right of the image block as candidate image blocks.

[0087] Optionally, the electronic device may determine multiple candidate image blocks based on a single row or multiple rows of pixels on the upper side, left side, upper left side, lower left side, and right side of the image block. For example, the electronic device may calculate the mean of a single row or multiple rows of pixels in each direction, and calculate the difference based on the relative relationship between the positions of the image blocks, thereby obtaining multiple candidate image blocks. In this way, since the electronic device can determine the candidate image blocks based on the pixel information of the current image, the electronic device can determine multiple candidate image blocks corresponding to each image block in parallel, thereby improving the efficiency of determining the candidate image blocks.

[0088] It should be noted that the electronic device may also determine adjacent image blocks and image blocks calculated based on mean and interpolation as candidate image blocks, which is not limited in the embodiments of the present disclosure.

[0089] Next, combine Figure 6 , in this case, the process of determining multiple candidate image blocks corresponding to the image block is explained.

[0090] Figure 6 A schematic diagram of determining multiple candidate image blocks provided by an embodiment of the present disclosure. Figure 6 , including a first image. The first image includes 16 image blocks, and image block A is an image block to be processed. Electronic device ( Figure 6 (not shown) can obtain image block B, image block C, image block D, image block E and image block F, and perform pixel averaging processing on image block B to obtain image block G, perform pixel averaging processing on image block C to obtain image block H, perform pixel averaging processing on image block D to obtain image block I, perform pixel averaging processing on image block E to obtain image block J, and perform pixel averaging processing on image block F to obtain image block K. The electronic device can determine image block B, image block C, image block D, image block E, image block F, image block G, image block H, image block I, image block J and image block K as multiple candidate image blocks of image block A.

[0091] Case 2: The encoding method is inter-frame prediction encoding.

[0092] If the encoding method is inter-frame prediction encoding, multiple candidate image blocks corresponding to each image block of the first image are determined in at least one image adjacent to the first image, and the first image is encoded based on each image block and the multiple candidate image blocks corresponding to each image block.

[0093] Optionally, for any image block in the first image, if the encoding method is inter-frame legacy coding, the electronic device can encode the image block in the first image based on pixel information in multiple images adjacent to the first image, thereby reducing the difference between the current frame and the reference frame (adjacent frame).

[0094] Optionally, the electronic device can obtain multiple candidate image blocks based on a binary (0, 1) search mask matrix. The search strategy of the binary search mask matrix is ​​a diamond search strategy. If the image block is N*N, the electronic device can map the position in the search space (K*K, K is greater than N) to a binary matrix, and the diamond position to be searched is set to 1, and the remaining positions are set to 0. In this way, the reference frame can be combined to obtain an ordered search candidate area corresponding to the image block, and in order to achieve multi-position parallel processing, when constructing a search candidate area for the image block N*N, for different reference frames, on the basis of the same area, the size is expanded to be equal to the size of the candidate search area, so that the candidate image blocks can be searched in parallel to improve processing efficiency.

[0095] It should be noted that the electronic device may also determine multiple candidate image blocks for inter-frame prediction coding based on any feasible implementation method (such as a half-pixel search method, etc.), and the embodiments of the present disclosure are not limited to this.

[0096] Optionally, the electronic device may encode the first image based on each image block and multiple candidate image blocks corresponding to each image block, and further encode each image in the first predicted video to obtain a second predicted video and a predicted bit rate.

[0097] S404: Training a video processing model based on the second predicted video, the target video, the predicted bit rate, and the target bit rate.

[0098] Optionally, the electronic device may determine a first loss based on the second predicted video and the target video, determine a second loss based on the predicted bit rate and the target bit rate, and further train a video processing model based on the first loss and the second loss.

[0099] For example, if the first video frame in the second predicted video is image A and the first video frame in the target video is image B, the electronic device may determine the first loss based on image A and image B.

[0100] It should be noted that the loss function in the embodiment of the present disclosure may be any loss function, and the embodiment of the present disclosure is not limited to this.

[0101] Next, combine Figure 7 , the training process of the video processing model is explained.

[0102] Figure 7 A schematic diagram of the training process of a video processing model provided in an embodiment of the present disclosure. Figure 7 , including: sample video, target bit rate, target video, video processing model and differentiable encoder. Electronic device ( Figure 7 (not shown) a sample video may be input into a video processing model, the video processing model may output a first predicted video, and the first predicted video may be input into a differentiable encoder.

[0103] See also Figure 7 , the differentiable encoder can encode and decode the first predicted video to obtain the second predicted video and the predicted bit rate. The electronic device can determine the first loss based on the second predicted video and the target video, determine the second loss based on the predicted bit rate and the target bit rate, and update the video processing model based on the first loss and the second loss. In this way, since the differentiable encoder can perform gradient feedback, the video processing model can learn the information of the coding distortion, thereby improving the training accuracy of the video processing model.

[0104] The disclosed embodiment provides a method for training a video processing model, obtaining a sample video, a target video corresponding to the sample video, and a target bit rate of the target video, processing the sample video based on the video processing model to obtain a first predicted video, processing the first predicted video based on a differentiable encoder to obtain a second predicted video and a predicted bit rate, and training the video processing model based on the second predicted video, the target video, the predicted bit rate, and the target bit rate. In this way, since the differentiable encoder can perform gradient feedback, the differentiable encoder can participate in the training of the video processing model, and the video processing model can learn the distortion information after the encoder encodes the video, thereby improving the training accuracy of the video processing model, and the encoder can perform encoding processing on multiple image blocks in the current image in parallel based on the image information before encoding, thereby improving the training efficiency of the video processing model.

[0105] exist Figure 4 Based on the embodiment shown below, combined with Figure 8 In the training method of the above-mentioned video processing model, a method for encoding the first image based on each image block and multiple candidate image blocks corresponding to each image block is described.

[0106] Figure 8 A schematic diagram of an image encoding processing method provided by an embodiment of the present disclosure. Figure 8 The method flow includes:

[0107] S801 . For any image block, determine a target image block corresponding to the image block based on a plurality of candidate image blocks corresponding to the image block.

[0108] Optionally, the electronic device may determine the target image block based on the following two feasible implementations:

[0109] A possible implementation method:

[0110] Convolution processing is performed on multiple candidate image blocks to obtain a target image block corresponding to the image block. For example, the electronic device may pre-train an image synthesis model, the training goal of which is to reduce redundant pixel information between multiple image blocks, so that multiple candidate image blocks can be processed based on the image synthesis model to obtain a target image block.

[0111] In this way, based on the pre-trained model, the electronic device can obtain more pixel information and can then accurately encode the image.

[0112] Another possible implementation:

[0113] The electronic device may determine the residual between the image block and each candidate image block, and determine the candidate image block with the smallest residual between the image block and the image block as the target image block. For example, the image block includes candidate image block 1, candidate image block 2, and candidate image block 3. The electronic device determines the residual between the image block and candidate image block 1 as residual 1, the residual between the image block and candidate image block 2 as residual 2, and the residual between the image block and candidate image block 3 as residual 3. If residual 1 is the smallest, the electronic device may determine candidate image block 1 as the target image block, if residual 2 is the smallest, the electronic device may determine candidate image block 2 as the target image block, and if residual 3 is the smallest, the electronic device may determine candidate image block 3 as the target image block.

[0114] It should be noted that the electronic device can determine the residual between the image block and the candidate image block based on any feasible implementation method (such as subtracting the image blocks, etc.), and the embodiments of the present disclosure are not limited to this.

[0115] In this way, the residual between the target image block and the image block is small, and the encoding accuracy can be improved during the predictive encoding process.

[0116] S802: Based on multiple image blocks in the first image and a target image block corresponding to each image block, perform encoding processing on the first image block.

[0117] Among them, the electronic device can encode the first image block based on the following feasible implementation method: for any image block, determine the first residual image between the image block and the target image block corresponding to the image block, process multiple first residual images to obtain multiple second residual images after encoding the multiple first residual images, and obtain the second image after the first image is encoded based on the multiple first residual images, the multiple second residual images and the first image.

[0118] Optionally, the first residual image may indicate a difference between the image block and the target image block. For example, the electronic device may subtract the image block from the target image block to obtain a first residual image corresponding to the image block, wherein each pixel value in the first residual image may represent a pixel difference between the image block and the target image block.

[0119] The second residual image may be an image obtained by encoding and decoding the first residual image. For example, the electronic device may quantize and encode the first residual image, and then decode the quantization result to obtain the second residual image, wherein the difference between the first residual image and the second residual image may be the encoding loss.

[0120] The electronic device may process the plurality of first residual images based on a differentiable encoder to obtain a plurality of second residual images after encoding the plurality of first residual images. For example, the electronic device may encode and decode the plurality of first residual images based on a differentiable encoder to obtain a plurality of second residual images.

[0121] Among them, the electronic device processes multiple first residual images to obtain multiple second residual images after the multiple first residual images are encoded. Specifically, it can be: for any first residual image, the first residual image is subjected to Fourier transform processing to obtain a frequency domain image corresponding to the first residual image, and the frequency domain image is quantized and rounded to obtain a quantized image. The electronic device can perform inverse quantization processing on the quantized image to obtain an inverse quantized image, and perform inverse Fourier transform processing on the inverse quantized image to obtain a second residual image corresponding to the first residual image.

[0122] The frequency domain image may indicate a frequency domain signal in the first residual image. For example, the frequency domain image may indicate the position of a high frequency signal and a low frequency signal in the first residual image, and the electronic device may convert the first residual image from the spatial domain to the frequency domain based on Fourier transform.

[0123] Optionally, the electronic device may determine the frequency domain image corresponding to the first residual image based on any feasible implementation manner, which is not limited in the embodiments of the present disclosure.

[0124] Optionally, the quantized image may be an image obtained by quantizing the frequency domain image. For example, the quantization process may include a quantization operation and a rounding operation, wherein the differentiable encoder may include a quantization module, and the quantization module may perform quantization and rounding on the frequency domain image.

[0125] Among them, the functions corresponding to the quantization and rounding processing are differentiable functions. For example, the quantization module in the differentiable encoder may include a differentiable rounding function, so that the quantization operation performed on the image based on the differentiable encoder is differentiable, and therefore, the differentiable encoder can perform gradient feedback.

[0126] Optionally, the function corresponding to the quantization and rounding process may be the following function:

[0127]

[0128] Among them, Q r It can be the result after quantization and rounding; n is a preset parameter (can be set arbitrarily), and Q is the result after quantization (quantization is not rounded).

[0129] Optionally, the dequantized image may be an image obtained by dequantizing the quantized image. For example, the dequantized image may be a process of restoring the quantized image. Since quantization and rounding may lose some information, the image quality of the dequantized image is lower than that of the quantized image.

[0130] Optionally, the electronic device may perform inverse quantization processing on the quantized image based on any feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0131] Optionally, the electronic device may perform an inverse Fourier transform on the inverse quantized image, thereby converting the inverse quantized image from the frequency domain to the spatial domain to obtain a second residual image. The electronic device may convert the inverse quantized image from the frequency domain to the spatial domain based on any feasible implementation method, which is not limited in the embodiments of the present disclosure.

[0132] Optionally, after obtaining the quantized image corresponding to each first residual image, the electronic device may further perform entropy coding on multiple quantized images associated with the first predicted video to obtain a predicted bit rate.

[0133] Among them, the electronic device can perform entropy coding processing on multiple quantized images based on the entropy coding model. For example, if the first predicted video includes 100 video frame images, the electronic device can determine the quantized image corresponding to each image block in each video frame image, and then process the multiple quantized images based on the entropy coding model to obtain a predicted bit rate, which can be the bit rate of the second predicted video.

[0134] Next, combine Fig. 9 , the process of determining the second residual image is described.

[0135] Fig. 9 A schematic diagram of a process for determining a second residual image provided by an embodiment of the present disclosure. Fig. 9 , including: a first image. The first image may include an image block to be encoded (black area). An electronic device ( Fig. 9 (not shown) Based on the image block to be encoded, 6 candidate image blocks can be acquired, and a target image block corresponding to the image block to be encoded is determined from the 6 candidate image blocks.

[0136] See also Fig. 9 The electronic device can obtain a first residual image A based on the residual between the image block to be encoded and the target image block. The electronic device can process the first residual image A based on the Fourier transform module to obtain a frequency domain image B, and process the frequency domain image B based on the quantization model to obtain a quantized image C.

[0137] See also Fig. 9The electronic device can input the quantized image C into the entropy coding module to obtain the code rate corresponding to the image block to be encoded, input the quantized image C into the dequantization module to obtain the dequantized image D, and input the dequantized image D into the inverse Fourier transform module to obtain the second residual image E. In this way, the electronic device can determine the encoding loss of the encoding process based on the first residual image and the second residual image.

[0138] Among them, after the electronic device obtains multiple second residual images, it can obtain the second image after the first image is encoded based on the multiple first residual images, the multiple second residual images and the first image. The electronic device can determine the second image based on the following feasible implementation method: for any first residual image, determine the difference between the first residual image and the second residual image corresponding to the first residual image to obtain the sub-residual image corresponding to the first residual image; determine the residual image based on the multiple sub-residual images, and add the first image to the residual image to obtain the second image after the first image is encoded.

[0139] Optionally, the sub-residual image may be an image corresponding to the difference between the first residual image and the second residual image. For example, the electronic device may subtract the first residual image from the second residual image to obtain a sub-residual image, which may represent the encoding loss of the first residual image after encoding processing.

[0140] Optionally, the electronic device may combine multiple sub-residual images to obtain a residual image. For example, for a first image, the first image may include multiple image blocks, each of which may correspond to a sub-residual image. The electronic device may combine multiple sub-residual images based on the position of the image block corresponding to the sub-residual image in the first image to obtain a residual image corresponding to the first image.

[0141] Optionally, the electronic device may add the first image to the residual image to obtain the second image. For example, the residual image corresponding to the first image is the encoding loss after the first image is encoded by a differentiable encoder, so the electronic device may add the residual image to the first image to obtain an image after encoding and decoding the first image.

[0142] In this way, the electronic device can determine the encoded image corresponding to each image in the first predicted video, and then obtain the second predicted video. In addition, the electronic device can obtain the bit rate of the second predicted video based on the entropy coding module. In this way, the video processing model can learn the loss of image coding during the training process, thereby improving the training accuracy of the video processing model.

[0143] Next, combine Fig.10 , the process of generating the second image is described.

[0144] Fig.10 A schematic diagram of a process for generating a second image provided by an embodiment of the present disclosure. Fig.10 , including: a first image. The first image includes 16 image blocks to be encoded. An electronic device ( Fig.10 (not shown) can determine a candidate image block set based on 16 image blocks to be encoded, wherein the candidate image block set includes 16 groups of candidate image blocks, each group of candidate image blocks includes 6 candidate image blocks, and each group of candidate image blocks corresponds to one image block to be encoded.

[0145] See also Fig.10 , the electronic device can determine the target image block in each group of candidate image blocks to obtain a target image block set. The target image block set includes 16 target image blocks, each of which corresponds to the image block to be encoded. The electronic device can obtain a first residual image set based on the difference between the image block to be encoded and the target image block corresponding to the image block to be encoded, wherein the first residual image set includes 16 first residual images.

[0146] See also Fig.10 The electronic device can send the codec module (the processing steps of the codec module can refer to Fig. 9 The processing steps in the embodiment shown are not repeated in detail in the embodiment of the present disclosure) input a first residual image set, and the encoding and decoding module can determine the second residual image corresponding to each first residual image to obtain a second residual image set.

[0147] See also Fig.10 , the electronic device can subtract the 16 first residual images from the 16 second residual images (the subtraction can be based on the position) to obtain a residual image, which can represent the coding loss after the first image is encoded. Therefore, the electronic device can add the residual image to the first image to obtain the second image after the first image is encoded. In this way, based on the differentiable encoder, the electronic device can encode all image blocks in the first image in parallel, and the electronic device can obtain the residual image corresponding to each image in a timely manner, thereby improving the training efficiency of the video processing model.

[0148] Optionally, the electronic device may further include a method for dividing an image block, wherein, if the size of the image block is M*K, the image block division size may include 4, 8, 16, 32 and 64, and the electronic device may divide the image into N*N image blocks based on the division size of each image block (the boundary area less than N*N may be supplemented to N*N by filling the boundary value).

[0149] The electronic device may determine the first residual images corresponding to the multiple image blocks in each division method, and further determine the division method of the image blocks based on the comparison of the first residual images in each division method. For example, the electronic device may divide an image (of size M*K) into 4 blocks and 8 blocks, wherein, for the 4-block division method, the electronic device may determine the first residual image corresponding to each image block to obtain 4 first residual images, and for the 8-block division method, the electronic device may determine the first residual image corresponding to each image block to obtain 8 first residual images, and the electronic device may transform the 4 first residual images to obtain a target residual image of (M / 4)*(K / 4), and the electronic device may transform the 8 first residual images to obtain a target residual image of (M / 8)*(K / 8). Since the sizes of the two target residual images are different, the electronic device may add the values ​​in the target residual image of (M / 4)*(K / 4), and then align the sizes of the target residual image of (M / 4)*(K / 4) and the target residual image of (M / 8)*(K / 8). For any region, the electronic device may determine the division method corresponding to the region with the smallest residual as the division method of the image. For example, for the first area in the 4*4 divided image, the first prediction residual corresponding to the image block of the first area is residual 1, and for the second area in the 8*8 divided image (corresponding to the position of the first area), the first prediction residual corresponding to the image block of the second area is residual 2. If residual 1 is smaller than residual 2, the image division method of the area is 4*4.

[0150] The disclosed embodiment provides an image coding processing method, for any image block in an image, based on a plurality of candidate image blocks corresponding to the image block, determining a target image block corresponding to the image block, for any image block, determining a first residual image between the image block and the target image block corresponding to the image block, processing the plurality of first residual images, obtaining a plurality of second residual images after encoding the plurality of first residual images, and obtaining a second image after encoding the first image based on the plurality of first residual images, the plurality of second residual images and the first image. In this way, since the quantization and rounding functions are differentiable functions, the differentiable encoder can perform gradient backpropagation, the video processing model can learn the coding loss after video encoding, and improve the training accuracy of the video processing model, and the differentiable encoder can encode the plurality of image blocks in the current image in parallel, and improve the efficiency of obtaining the residual image corresponding to each image, thereby improving the training efficiency of the video processing model.

[0151] Fig.11 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure. Fig.11The video processing device 110 includes an acquisition module 111, a processing module 112 and an encoding module 113, wherein:

[0152] The acquisition module is used to acquire the first video;

[0153] The processing module is used to process the first video based on a video processing model to obtain a second video, wherein the training phase of the video processing model includes a differentiable encoder, the differentiable encoder is used to simulate the process of quantizing and encoding the video by the encoder, and the differentiable encoder can perform gradient feedback;

[0154] The encoding module is used to encode the second video.

[0155] According to one or more embodiments of the present disclosure, the video processing device 110 further includes a training module 114, wherein the training module 114 is used to:

[0156] Obtaining a sample video, a target video corresponding to the sample video, and a target bit rate of the target video;

[0157] Based on the video processing model, the sample video is processed to obtain a first predicted video;

[0158] Processing the first predicted video based on the differentiable encoder to obtain a second predicted video and a predicted bit rate;

[0159] The video processing model is trained based on the second predicted video, the target video, the predicted bit rate and the target bit rate.

[0160] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0161] Performing image segmentation processing on each image in the first predicted video to obtain a plurality of image blocks corresponding to each image;

[0162] Determine a coding mode corresponding to each image in the first predicted video, wherein the coding mode includes intra-frame prediction coding and inter-frame prediction coding;

[0163] Based on the differentiable encoder, the encoding mode corresponding to each image and the multiple image blocks corresponding to each image, each image is encoded to obtain the second predicted video and predicted bit rate.

[0164] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0165] For any first image in the first predicted video;

[0166] If the encoding mode is intra-frame prediction encoding, a plurality of candidate image blocks corresponding to each image block are determined in the first image, and encoding processing is performed on the first image based on each image block and the plurality of candidate image blocks corresponding to each image block;

[0167] If the encoding method is inter-frame prediction encoding, then in at least one image adjacent to the first image, multiple candidate image blocks corresponding to each image block of the first image are determined, and the first image is encoded based on each image block and the multiple candidate image blocks corresponding to each image block.

[0168] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0169] For any image block, determining a target image block corresponding to the image block based on a plurality of candidate image blocks corresponding to the image block;

[0170] Based on a plurality of image blocks in the first image and a target image block corresponding to each image block, encoding processing is performed on the first image block.

[0171] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0172] Performing convolution processing on the multiple candidate image blocks to obtain target image blocks corresponding to the image blocks;

[0173] or,

[0174] The residual between the image block and each candidate image block is determined, and the candidate image block having the smallest residual with the image block is determined as the target image block.

[0175] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0176] For any image block, determining a first residual image between the image block and a target image block corresponding to the image block;

[0177] Processing the plurality of first residual images to obtain a plurality of second residual images obtained by encoding the plurality of first residual images;

[0178] Based on the multiple first residual images, the multiple second residual images and the first image, a second image after the first image is encoded is obtained.

[0179] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0180] For any first residual image, determine a difference between the first residual image and a second residual image corresponding to the first residual image to obtain a sub-residual image corresponding to the first residual image;

[0181] Based on the multiple sub-residual images, a residual image is determined, and the first image is added to the residual image to obtain a second image after the first image is encoded.

[0182] According to one or more embodiments of the present disclosure, the training module 114 is specifically used for:

[0183] For any first residual image;

[0184] Performing Fourier transform processing on the first residual image to obtain a frequency domain image corresponding to the first residual image;

[0185] Quantizing and rounding the frequency domain image to obtain a quantized image, wherein the function corresponding to the quantization and rounding is a differentiable function;

[0186] Performing inverse quantization processing on the quantized image to obtain an inverse quantized image;

[0187] The dequantized image is subjected to inverse Fourier transform processing to obtain the second residual image corresponding to the first residual image.

[0188] According to one or more embodiments of the present disclosure, the training module 114 is further configured to:

[0189] Perform entropy coding on multiple quantized images associated with the first predicted video to obtain the predicted bit rate.

[0190] The video processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0191] Fig.12 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.12 , which shows a schematic diagram of the structure of an electronic device 1200 suitable for implementing the embodiment of the present disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.12 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0192] like Fig.12 As shown, the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 to a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0193] Typically, the following devices may be connected to the I / O interface 1205: input devices 1206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.12 The electronic device 1200 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0194] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1209, or installed from a storage device 1208, or installed from a ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0195] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0196] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0197] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0198] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, various methods that may be involved in the above embodiments are implemented.

[0199] An embodiment of the present disclosure provides a computer program product, including a computer program, which implements various methods that may be involved in the above embodiments when executed by a processor.

[0200] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0201] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0202] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0203] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0204] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0205] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0206] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0207] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0208] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from a user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0209] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0210] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0211] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0212] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0213] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video processing method, characterized in that: include: Get the first video; Based on a video processing model, the first video is processed to obtain a second video, wherein the training phase of the video processing model includes a differentiable encoder, the differentiable encoder is used to simulate a process in which an encoder quantizes and encodes a video, and the differentiable encoder can perform gradient feedback; The second video is encoded.

2. The method according to claim 1, characterized in that The video processing model is trained based on the following steps: Obtaining a sample video, a target video corresponding to the sample video, and a target bit rate of the target video; Based on the video processing model, the sample video is processed to obtain a first predicted video; Processing the first predicted video based on the differentiable encoder to obtain a second predicted video and a predicted bit rate; The video processing model is trained based on the second predicted video, the target video, the predicted bit rate and the target bit rate.

3. The method according to claim 2, characterized in that The encoder based on the differentiable process processes the first predicted video to obtain a second predicted video and a predicted bit rate, including: Performing image segmentation processing on each image in the first predicted video to obtain a plurality of image blocks corresponding to each image; Determine a coding mode corresponding to each image in the first predicted video, wherein the coding mode includes intra-frame prediction coding and inter-frame prediction coding; Based on the differentiable encoder, the encoding mode corresponding to each image and the multiple image blocks corresponding to each image, each image is encoded to obtain the second predicted video and predicted bit rate.

4. The method according to claim 3, characterized in that The encoding process of each image based on the differentiable encoder, the encoding mode corresponding to each image, and the multiple image blocks corresponding to each image to obtain the second predicted video and the predicted bit rate includes: For any first image in the first predicted video; If the encoding mode is intra-frame prediction encoding, a plurality of candidate image blocks corresponding to each image block are determined in the first image, and encoding processing is performed on the first image based on each image block and the plurality of candidate image blocks corresponding to each image block; If the encoding method is inter-frame prediction encoding, then in at least one image adjacent to the first image, multiple candidate image blocks corresponding to each image block of the first image are determined, and the first image is encoded based on each image block and the multiple candidate image blocks corresponding to each image block.

5. The method according to claim 4, characterized in that The encoding process of the first image based on each image block and a plurality of candidate image blocks corresponding to each image block includes: For any image block, determining a target image block corresponding to the image block based on a plurality of candidate image blocks corresponding to the image block; Based on a plurality of image blocks in the first image and a target image block corresponding to each image block, encoding processing is performed on the first image block.

6. The method according to claim 5, characterized in that The determining, based on a plurality of candidate image blocks corresponding to the image block, a target image block corresponding to the image block comprises: Performing convolution processing on the multiple candidate image blocks to obtain target image blocks corresponding to the image blocks; or, The residual between the image block and each candidate image block is determined, and the candidate image block having the smallest residual with the image block is determined as the target image block.

7. The method according to claim 5, characterized in that The encoding process of the first image block based on the multiple image blocks in the first image and the target image block corresponding to each image block includes: For any image block, determining a first residual image between the image block and a target image block corresponding to the image block; Processing the plurality of first residual images to obtain a plurality of second residual images obtained by encoding the plurality of first residual images; Based on the multiple first residual images, the multiple second residual images and the first image, a second image after the first image is encoded is obtained.

8. The method according to claim 7, characterized in that The step of obtaining, based on the plurality of first residual images, the plurality of second residual images and the first image, a second image after encoding the first image comprises: For any first residual image, determine a difference between the first residual image and a second residual image corresponding to the first residual image to obtain a sub-residual image corresponding to the first residual image; Based on the multiple sub-residual images, a residual image is determined, and the first image is added to the residual image to obtain a second image after the first image is encoded.

9. The method according to claim 7 or 8, characterized in that: The processing of the plurality of first residual images to obtain a plurality of second residual images after encoding the plurality of first residual images comprises: For any first residual image; Performing Fourier transform processing on the first residual image to obtain a frequency domain image corresponding to the first residual image; Quantizing and rounding the frequency domain image to obtain a quantized image, wherein the function corresponding to the quantization and rounding is a differentiable function; Performing inverse quantization processing on the quantized image to obtain an inverse quantized image; The dequantized image is subjected to inverse Fourier transform processing to obtain the second residual image corresponding to the first residual image.

10. The method according to claim 9, characterized in that After obtaining the quantized image, the method further includes: Perform entropy coding on multiple quantized images associated with the first predicted video to obtain the predicted bit rate.

11. A video processing device, characterized in that: It includes an acquisition module, a processing module and an encoding module, among which: The acquisition module is used to acquire the first video; The processing module is used to process the first video based on a video processing model to obtain a second video, wherein the training phase of the video processing model includes a differentiable encoder, the differentiable encoder is used to simulate the process of quantizing and encoding the video by the encoder, and the differentiable encoder can perform gradient feedback; The encoding module is used to encode the second video.

12. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the video processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the video processing method according to any one of claims 1 to 10 is implemented.

Images