Video image processing method, system, device and storage medium

Abstract

The present application provides a video image processing method, system, device, and storage medium, the method comprising: obtaining a first 3D lookup table; using the pre-processed image of the current frame and the pre-processed image of the previous frame as two consecutive pre-processed frames, performing discretization processing on each to obtain a discretization processing result, and combining the discretization processing results of the two consecutive pre-processed frames to obtain a set of discrete sampling values; discretizing the first 3D lookup table based on the discrete sampling values ​​to obtain a second 3D lookup table corresponding to the current frame; and processing the pre-processed image of the current frame through the second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame. By adopting the present application, the inter-frame consistency of the processed image is improved, and the problems of picture jumps and inconsistent image adjustment levels in video scenes are solved.

Description

Technical Field

[0001] The present application relates to the field of video image processing technology, and in particular to a video image processing method, system, device and storage medium. Background Art

[0002] In video conferencing, poor image quality can affect the appearance of people on the screen. Furthermore, certain digital video image processing operations in video software, such as sharpening and gamma shifting, can also affect facial images. Therefore, image adjustments are necessary. In video conferencing scenarios, image adjustments include, but are not limited to, facial enhancement (including whitening, filters, and skin smoothing), which can enhance the enjoyment of video conferencing.

[0003] There are two general approaches to portrait beautification: traditional methods and deep learning-based methods. Traditional solutions use digital video image processing techniques to globally multiply the image by a weighting factor (the weighting factor can be adjusted based on the specific application objective). Alternatively, a global mapping can be achieved by constructing preset 3D lookup tables to obtain the desired portrait beautification effect. Deep learning-based methods use autoencoders to map pre- and post-beautification images. These methods are simple to implement and easy to use. However, their input source is a single image, and they fail to consider inter-frame consistency in video beautification. This can lead to jittery images and inconsistent facial beautification levels when used in video scenarios.

[0004] In addition, in the field of video image adjustment in other fields, there is also the problem of insufficient consideration of the inter-frame consistency of video image adjustment. For example, when performing adjustment measures such as brightness enhancement, saturation adjustment, and color correction on images in surveillance videos, processing a single image can easily lead to screen jumps and inconsistent image adjustment levels during use in video scenes.

[0005] Application Contents

[0006] In response to the problems in the prior art, the purpose of this application is to provide a video image processing method, system, device and storage medium, which improves the inter-frame consistency of the processed image and solves the problems of picture jitter and inconsistent image adjustment degree in video scenes.

[0007] The present invention provides a method for processing video images, including the following steps:

[0008] Obtain a first 3D lookup table;

[0009] The pre-processing image of the current frame and the pre-processing image of the previous frame are used as two consecutive pre-processing images, and discretization processing is performed on each of them to obtain discretization processing results, and a set of discrete sampling values ​​is obtained by combining the discretization processing results of the two consecutive pre-processing images;

[0010] Discretize the first 3D lookup table based on the discrete sampling values ​​to obtain a second 3D lookup table corresponding to the current frame;

[0011] The pre-processed image of the current frame is processed through a second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame.

[0012] By adopting the method of the present invention, after obtaining the first 3D lookup table, the first 3D lookup table is discretized based on two consecutive frames of pre-processed images to obtain a discretized second 3D lookup table. The second 3D lookup table, which combines the discretization results of the two frames of pre-processed images, is used to process the pre-processed image of the current frame, so that the discrete values ​​of the second 3D lookup table used for processing the current frame are also associated with the previous frame image. When the second 3D lookup table is used to process the pre-processed image of the current frame, the image information of the two consecutive frames of images is simultaneously considered, thereby improving the consistency of the video image processing process, thereby improving the inter-frame consistency of the processed image and solving the problems of picture jumps and inconsistent image adjustment levels in the video scene. This method can be applied to facial beauty adjustment in video conferencing, and can also be applied to the adjustment processing of other categories of targets in the video (such as vehicles, pedestrians, obstacles, etc.). It can also be used for parameter adjustment of the overall image in the video, where the image adjustment may include one or more aspects of image parameter adjustment such as brightness adjustment, saturation adjustment, and color adjustment.

[0013] In some embodiments, obtaining the first 3D lookup table comprises the following steps:

[0014] obtaining at least one image adjustment degradation kernel;

[0015] Acquire a plurality of first adjusted images, and obtain a first unadjusted image corresponding to each of the first adjusted images based on the image adjustment degradation kernel, and use the first adjusted images and the first unadjusted images as training image data;

[0016] Build the initial 3D lookup table;

[0017] The initial 3D lookup table is optimized based on the training image data to obtain the first 3D lookup table.

[0018] In some embodiments, obtaining at least one image adjustment degradation kernel comprises the following steps:

[0019] constructing at least one image adjustment degradation kernel;

[0020] Acquiring a plurality of second adjusted images and a plurality of second unadjusted images;

[0021] Extracting first edge features from the plurality of second adjusted images, and inputting the extracted first edge features and the plurality of image adjustment degradation kernels into a generative adversarial network;

[0022] classifying the output image of the generative adversarial network and the second unadjusted image through a classifier network, and optimizing the at least one image adjustment degradation kernel through a classification loss function of the classifier network;

[0023] Correspondingly, the method of obtaining the corresponding first unadjusted image based on the image adjustment degradation kernel includes: inputting the collected first adjusted image and the optimized image adjustment degradation kernel into the generative adversarial network to obtain the corresponding first unadjusted image output by the generative adversarial network.

[0024] In some embodiments, obtaining the first 3D lookup table comprises the following steps:

[0025] Acquire training image data including a first unadjusted image and a first adjusted image;

[0026] After multiplying the first unadjusted image by a preset global weight factor according to preset scene requirements, updating the first unadjusted image to obtain updated training image data;

[0027] Build the initial 3D lookup table;

[0028] The initial 3D lookup table is optimized based on the updated training image data to obtain the first 3D lookup table.

[0029] In some embodiments, obtaining the first 3D lookup table comprises the following steps:

[0030] Construct multiple initialized sub-lookup tables of different categories;

[0031] Obtaining a value of each of the initialization sub-lookup tables and a weight value of each of the initialization sub-lookup tables based on training image data including a first unadjusted image and a first adjusted image;

[0032] The initialization sub-lookup tables are weightedly fused based on the weight values ​​of the initialization sub-lookup tables to obtain the first 3D lookup table.

[0033] In some embodiments, a pre-processed image of a current frame and a pre-processed image of a previous frame are used as two consecutive pre-processed images, and discretization processing is performed on each of them to obtain discretization processing results. Combining the discretization processing results of the two consecutive pre-processed images to obtain a set of discrete sampling values ​​includes the following steps:

[0034] Sampling the two consecutive frames of pre-processed images respectively to obtain a discretization processing result corresponding to the current frame and a discretization processing result corresponding to the previous frame respectively;

[0035] A set of discrete sampling values ​​is obtained by fusing or combining the discretization processing result corresponding to the current frame and the discretization processing result corresponding to the previous frame.

[0036] In some embodiments, after the pre-processed image of the current frame is processed through the second 3D lookup table to obtain the first processed image of the current frame, the following steps are further included:

[0037] Inputting the first processed image of the current frame into a pre-trained inter-frame consistency constraint network to obtain a second processed image of the current frame output by the inter-frame consistency constraint network;

[0038] The inter-frame consistency constraint network is trained using the following steps:

[0039] Acquire multiple frames of training images, and process each frame of the training image using a corresponding second lookup table to obtain a first processed image of each frame of the training image;

[0040] Selecting a plurality of training groups from the first processed images of the plurality of frames of training images, each of the training groups including two frames of training images;

[0041] In each iteration, two frames of the training images in one of the training groups are respectively input into the inter-frame consistency constraint network, and two frames of constraint network output images are respectively obtained;

[0042] The error value of the output images of the two-frame constraint network is used as a fitting target, and the network parameters of the inter-frame consistency constraint network are iteratively optimized to obtain an optimized inter-frame consistency constraint network.

[0043] In some embodiments, after obtaining the first processed image of the current frame, the following steps are further included:

[0044] Acquire a first target mask area image from the first processed image of the current frame;

[0045] After performing downsampling and bilateral filtering on the first target mask area, performing a mean operation on each m×m area, where m is a positive integer greater than 1, to obtain a second target mask area;

[0046] The second target mask area is up-sampled and fused with the unprocessed image of the current frame to obtain an output image of the current frame.

[0047] An embodiment of the present invention further provides a video image processing system, which is applied to the video image processing method. The system includes:

[0048] A lookup table acquisition module, configured to acquire a first 3D lookup table;

[0049] a discrete processing module configured to discretize the pre-processed image of the current frame and the pre-processed image of the previous frame as two consecutive pre-processed frames, obtain discretization results, combine the discretization results of the two consecutive pre-processed frames to obtain a set of discrete sampling values, and discretize the first 3D lookup table based on the discrete sampling values ​​to obtain a second 3D lookup table corresponding to the current frame;

[0050] The image processing module is configured to process the pre-processed image of the current frame through a second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame.

[0051] By adopting the system of the present invention, after obtaining a first 3D lookup table through a lookup table acquisition module, the first 3D lookup table is discretized by a discrete processing module based on two consecutive pre-processed frames of images to obtain a discretized second 3D lookup table. The pre-processed image of the current frame is processed using the second 3D lookup table that combines the discretization results of the two pre-processed frames, so that the discrete values ​​of the second 3D lookup table used for processing the current frame are also associated with the previous frame. When the second 3D lookup table is used to process the pre-processed image of the current frame, the image information of the two consecutive frames is also considered, thereby improving the consistency of the video image processing process, thereby improving the inter-frame consistency of the processed image and solving the problems of picture jump and inconsistent image adjustment in the video scene. The system can be applied to face beauty adjustment in video conferencing, and can also be applied to the adjustment processing of other categories of objects in the video (such as vehicles, pedestrians, obstacles, etc.), or for the overall image parameter adjustment processing of the image in the video. Here, the image adjustment may include one or more aspects of image parameter adjustment such as brightness adjustment, saturation adjustment, and color adjustment.

[0052] The present application also provides a video image processing device, including:

[0053] processor;

[0054] a memory storing executable instructions for the processor;

[0055] The processor is configured to execute the steps of the video image processing method by executing the executable instructions.

[0056] By adopting the video image processing device provided by the present application, the processor executes the video image processing method when executing the executable instructions, thereby achieving the beneficial effects of the above-mentioned video image processing method.

[0057] An embodiment of the present application also provides a computer-readable storage medium for storing a program, which implements the steps of the video image processing method when executed by a processor.

[0058] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the video image processing method when being executed, thereby achieving the beneficial effects of the above-mentioned video image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings.

[0060] Figure 1 is a flowchart of a video image processing method according to an embodiment of the present application;

[0061] Figure 2 This is a flowchart of obtaining a first 3D lookup table according to an embodiment of the present application;

[0062] Figure 3 is a schematic diagram of an image adjustment degradation kernel according to an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of a process for optimizing a current frame based on a 3D lookup table according to an embodiment of the present application;

[0064] Figure 5 is a schematic diagram of a video image processing system according to an embodiment of the present application;

[0065] Figure 6 is a structural diagram of a video image processing device according to an embodiment of the present application;

[0066] Figure 7 It is a structural diagram of a computer storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Identical reference numerals in the drawings represent identical or similar structures, and thus repeated descriptions thereof will be omitted.

[0068] like Figure 1 As shown, in one embodiment, the present application provides a video image processing method, comprising the following steps:

[0069] S100: Obtain a first 3D lookup table;

[0070] A 3D lookup table is a list of corresponding values ​​that can be used to query any input value and its corresponding output value. It is a commonly used technology in color conversion technology. The core idea is to divide the source color space into regular cubes. The data of the eight vertices of each cube is known, and all the known points in the source space form a 3D lookup table. Generally, the three RGB channels are divided into N vertices, forming N×N×N nodes. When in use, the other nodes are directly interpolated.

[0071] S200: Discretizing the pre-processed image of the current frame and the pre-processed image of the previous frame as two consecutive pre-processed images, respectively, to obtain discretization results, and combining the discretization results of the two consecutive pre-processed images to obtain a set of discrete sampling values;

[0072] Here, combining the discretization processing results of the two consecutive frames of pre-processed images may be combining or fusing the discretization processing result corresponding to the pre-processed image of the current frame and the discretization processing result corresponding to the pre-processed image of the previous frame. Combining refers to directly merging the two sets of discretization processing results, and fusing refers to averaging the values ​​of points at corresponding pixel positions.

[0073] S300: Discretize the first 3D lookup table based on the discrete sampling values ​​to obtain a second 3D lookup table corresponding to the current frame;

[0074] Discretization processing here refers to discretizing the continuous first 3D lookup table using the discrete sampling values ​​obtained in step S200, reducing the dimensions of the first 3D lookup table, thereby reducing the size of the first 3D lookup table, which is beneficial for improving the speed of image mapping lookup. In this case, the first 3D lookup table is discretized using the discretization processing results of two consecutive frames. The obtained second 3D lookup table is beneficial for improving the consistency between adjacent frames when processing the current frame.

[0075] S400: Processing the pre-processed image of the current frame through a second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame;

[0076] Here, processing through the second 3D lookup table corresponding to the current frame refers to performing mapping and interpolation processing through the second 3D lookup table corresponding to the current frame.

[0077] After obtaining a first 3D lookup table in step S100, the present invention discretizes the first 3D lookup table based on two consecutive pre-processed image frames in steps S200 and S300 to obtain a discretized second 3D lookup table. In step S400, the current frame is processed using the second 3D lookup table, which combines the discretization results of the two pre-processed image frames, so that the discrete values ​​of the second 3D lookup table used for processing the current frame are also associated with the previous frame. When the second 3D lookup table is used to process the pre-processed image of the current frame, the image information of the two consecutive frames is simultaneously considered, which is beneficial for improving the consistency of the video image processing process, thereby improving the inter-frame consistency of the processed image and solving the problems of picture jumps and inconsistent image adjustment levels in video scenes. This method can be applied to facial beauty adjustment in video conferencing, and can also be applied to the adjustment processing of other categories of objects in the video (such as vehicles, pedestrians, obstacles, etc.), and can also be used for the adjustment processing of the overall image in the video. Here, image adjustment refers to the adjustment of one or more image parameters, for example, including one or more aspects of parameter adjustment such as brightness adjustment, saturation adjustment, and color adjustment.

[0078] The following uses the method applied to face beautification adjustment as an example to illustrate the implementation process of this method. The target to be recognized is a face.

[0079] Existing methods use fixed 3D lookup tables, resulting in poor robustness and inability to generalize to real-world scenarios. This can lead to artifacts in facial areas, preventing natural portrait enhancement. Deep learning-based methods, however, lack sufficient training data, similarly leading to poor robustness and portrait enhancement. To address this, in this embodiment, multiple image adjustment degradation kernels are pre-built to obtain more training image data, enabling optimized training of the 3D lookup table and ultimately resulting in a more robust 3D lookup table.

[0080] Specifically, the step S100 of obtaining a first 3D lookup table includes the following steps:

[0081] S110: Acquire at least one image adjustment degradation kernel;

[0082] The processing effect of the image adjustment degradation kernel is the opposite effect of the target effect of the image parameter adjustment processing. For example, if the target effect is to increase brightness, the purpose of the degradation is to reduce brightness. If the target effect is to reduce brightness, the purpose of the degradation is to increase brightness. When multiple image adjustment degradation kernels are included, the multiple image adjustment degradation kernels can be multiple degradation kernels of different sizes but with the same function, or multiple degradation kernels with different functions, or multiple degradation kernels with different functional sizes. In the embodiment of the present application, the multiple image adjustment degradation kernels including multiple degradation kernels with different functions are used as an example for illustration. For example, when image adjustment includes brightness adjustment, color correction, and saturation adjustment, the image adjustment degradation kernels may include a brightness degradation kernel, a saturation degradation kernel, and a chromaticity degradation kernel.

[0083] S120: Collecting a plurality of first adjusted images in a target scene, and obtaining corresponding first unadjusted images A based on the at least one image adjustment degradation kernel, and using the first adjusted images and the first unadjusted images A as training image data;

[0084] Therefore, by adopting the image adjustment degradation kernel, it is only necessary to collect multiple first adjusted images to obtain the first unadjusted image A that is paired with them. For example, multiple facial beautification images can be collected in advance, and the corresponding pre-beautification images can be obtained based on the image adjustment degradation kernel to obtain multiple training image data. By pre-constructing multiple image adjustment degradation kernels, more training image groups can be obtained for optimizing the training of the 3D lookup table, solving the problem of insufficient training data in the prior art, thereby obtaining a 3D lookup table with better robustness.

[0085] S130: constructing an initial 3D lookup table;

[0086] Here, the initial 3D lookup table is a pre-built initialized 3D lookup table, in which the values ​​can adopt preset default initialization values. The specific implementation is described below;

[0087] S140: Optimizing the initial 3D lookup table based on the training image data to obtain the first 3D lookup table. For example, the first unadjusted image may be input into the initial 3D lookup table, and an image processed by the initial 3D lookup table is obtained. The error between the image processed by the initial 3D lookup table and the first adjusted image is calculated, and the initial 3D lookup table is reversely optimized with the goal of reducing the error.

[0088] In another embodiment, steps S110 and S120 may not be used to obtain the first pre-adjustment image and the first post-adjustment image for training. Instead, training image data for training may be directly captured in the target scene or obtained from the Internet, and then steps S130 and S140 may be continued to train the 3D lookup table.

[0089] In one embodiment, the step S110 of obtaining at least one image adjustment degradation kernel includes the following steps:

[0090] constructing at least one image adjustment degradation kernel;

[0091] For example, Figure 3 As shown, the at least one image adjustment degradation kernel may include one or more of a brightness degradation kernel, a saturation degradation kernel, and a chromaticity degradation kernel. The kernel size may be 3×3 or 5×5, and its parameters may be initialized to 0 or simple brightness, saturation, and chromaticity template parameters. Each image adjustment degradation kernel is equivalent to an image with a channel number of 1, and the height H and width W of the image are the same as those of the input second adjusted image. Then, the height H and width W are repeatedly assigned according to the kernel size.

[0092] Acquire a plurality of second adjusted images and a plurality of second unadjusted images in the target scene;

[0093] Here, the second adjusted image and the second unadjusted image may not correspond one to one, so the requirements for the collected data are not particularly high;

[0094] Extracting first edge features from the plurality of second adjusted images using a plurality of low-pass filters, and inputting the extracted first edge features and the plurality of image adjustment degradation kernels into a generative adversarial network, wherein an output image of the generative adversarial network is a predicted image before image adjustment;

[0095] The generative adversarial network and the image adjustment degradation kernel are trained in an unsupervised learning manner. Specifically, the output image of the generative adversarial network and the second image before adjustment are classified by a classifier network, and the image adjustment degradation kernel is optimized by the classification loss function of the classifier network. Specifically, when optimizing the image adjustment degradation kernel by the classification loss function of the classifier network, the parameters of the image adjustment degradation kernel are optimized by the classification loss function until the classifier cannot distinguish between the output image of the generative adversarial network and the second image before adjustment (the image without beautification). The image adjustment degradation kernel thus optimized is a degradation kernel that can meet the usage needs.

[0096] In step S120, obtaining the corresponding first unadjusted image A based on the at least one image adjustment degradation kernel includes inputting the collected first adjusted image and the at least one optimized image adjustment degradation kernel into the generative adversarial network to obtain the corresponding first unadjusted image A output by the generative adversarial network.

[0097] In this embodiment, the first unadjusted image A and the corresponding first adjusted image can also be directly captured in the target scene and added to the training image data, and used together with the training image data obtained by the image adjustment degradation kernel to optimize the subsequent 3D lookup table. The image can be captured in the target scene by camera shooting or crawling from the Internet. In another embodiment, the above-mentioned method of constructing the image adjustment degradation kernel can be omitted, and the first unadjusted image A and the corresponding first adjusted image can be directly captured in the target scene as training image data, which also falls within the scope of protection of this application.

[0098] In this embodiment, after obtaining the corresponding first unadjusted image A, the following steps are further included:

[0099] The first unadjusted image A is multiplied by a preset global weighting factor based on preset scene requirements to obtain a first unadjusted image B. The first unadjusted image B and the first adjusted image are combined to form updated training image data. When optimizing the initial 3D lookup table, this updated training image data is used to optimize the initial 3D lookup table. The first unadjusted image A processed using the global weighting factor can be a first unadjusted image obtained by using an image adjustment degradation kernel, or can be a first unadjusted image directly captured from the target scene or crawled from the Internet.

[0100] Specifically, the value of the global weight factor can be selected and set based on the actual scenario. For example, if a portrait needs to be whitened, the global weight factor value should be greater than 1, and a larger global weight factor value indicates a higher degree of whitening. If a portrait needs to be tanned, the global weight factor value should be less than 1, and a smaller global weight factor value indicates a higher degree of tanning. Here, the global weight factor is used to roughly increase or decrease the brightness of the first pre-adjusted image A, which helps reduce the difficulty of subsequent 3D lookup table optimization.

[0101] As described above, in another embodiment, if the image adjustment degradation kernel is not required and the first pre-adjusted image and the first post-adjusted image are directly photographed or obtained from the internet, then step S100 does not include steps S110 and S120, i.e., step S100 only includes steps S130 and S140. The following specifically defines the implementation of steps S130 and S140, and the following definitions apply to both embodiments that include and exclude steps S110 and S120.

[0102] In one embodiment, step S130 of constructing an initial 3D lookup table includes constructing a plurality of different types of initialization sub-lookup tables as the initial 3D lookup table. For example, the initialization sub-lookup tables may include a brightness sub-lookup table, a chroma sub-lookup table, and a saturation sub-lookup table. The parameters may adopt default initial values ​​or refer to Photoshop settings. Each sub-lookup table is sized 256×256×256, but the present application is not limited thereto. Each initialization sub-lookup table may adjust a template matrix for an initialization image.

[0103] The step S140 of optimizing the initial 3D lookup table based on the training image data to obtain the first 3D lookup table includes the following steps:

[0104] Acquire a value of each of the initialization sub-lookup tables and a weight value of each of the initialization sub-lookup tables based on the training image data;

[0105] Specifically, a low-pass filter is used to extract the second edge feature from the first unadjusted image B, and the second edge feature is obtained by downsampling and scaling the feature size and changing the feature through nonlinear operation, etc., to obtain the weight value corresponding to each sub-lookup table. At the same time, the first unadjusted image B is processed using the sub-lookup table to obtain a mapped adjusted image, and the mapped adjusted image is compared with the first adjusted image to update the values ​​of each sub-lookup table and the weight value corresponding to each sub-lookup table through reverse gradient propagation, and finally the tuned sub-lookup table and the weight value corresponding to each sub-lookup table are optimized. In another embodiment, the second edge feature can also be input into a CNN (convolutional neural network) to learn and obtain the weight value of each sub-lookup table through the convolutional neural network. The use of the first unadjusted image B here can reduce the difficulty of optimization training compared to the first unadjusted image A, but the present application is not limited to this. In another embodiment, the first unadjusted image A can be directly used for optimization training of the 3D lookup table, that is, the training image data is directly used to optimize the 3D lookup table.

[0106] The weight values ​​of each of the initialization sub-lookup tables are weightedly fused to obtain the first 3D lookup table; that is, the value of each position in the first 3D lookup table is equal to the sum of the values ​​of the corresponding positions of each initialization sub-lookup table multiplied by the weight value.

[0107] like Figure 4As shown, the initialized sub-lookup table weights are fused with the initialized brightness sub-lookup table, chroma sub-lookup table, and saturation sub-lookup table to obtain a fused first 3D lookup table. Fusion herein involves weighted summing or averaging the values ​​of corresponding positions of the three sub-lookup tables based on the weights of the three sub-lookup tables, and using the resulting value as the value of the corresponding position of the fused first 3D lookup table. For example, the value of the i-th position in the first 3D lookup table is equal to the value of the i-th position of the first sub-lookup table * the weight of the first sub-lookup table + the value of the i-th position of the second sub-lookup table * the weight of the second sub-lookup table + the value of the i-th position of the third sub-lookup table * the weight of the third sub-lookup table, where i>1. The size of the fused 3D lookup table is, for example, 256×256×256.

[0108] In this embodiment, step S200 of discretizing the pre-processed image of the current frame and the pre-processed image of the previous frame as two consecutive pre-processed images, obtaining discretization results after each discretization process, and combining the discretization results of the two consecutive pre-processed images to obtain a set of discrete sampling values ​​includes the following steps:

[0109] Performing transposition, filtering, nonlinear processing, and Nyquist sampling operations on two consecutive frames of pre-processed images in the video to be processed, respectively, to obtain Nyquist sampling values, that is, obtaining discretization processing results corresponding to the current frame and the discretization processing results corresponding to the previous frame, respectively;

[0110] The Nyquist sampling values ​​of the two consecutive frames of unprocessed images are fused or combined to obtain the discrete sampling value.

[0111] Only one method for obtaining discrete sampling values ​​is given here. In other embodiments, other discrete sampling methods may also be used. For example, the values ​​of the i-th position, i+n-th position, i+2n-th position, and i+3n-th position in the image may be pre-set and selected as discrete sampling results, that is, each frame of the image is uniformly sampled; or key point detection is performed on the image, and the sampling values ​​are determined based on the image key points, that is, the image is non-uniformly sampled based on the image key points to obtain a discrete processing result; in other embodiments, one or more of the above-mentioned image processing operations may be removed, or other processing operations may be added, such as omitting filtering processing or omitting nonlinear processing, etc., all of which fall within the scope of protection of this application.

[0112] like Figure 4As shown, in step S300, the first 3D lookup table is discretized using the discrete sampling values ​​obtained in step S200 to obtain a second 3D lookup table. This reduces the computational effort and allows for a smaller 3D lookup table, for example, 16×16×16 in size. Furthermore, because the discretized values ​​are derived by fusing the Nyquist sampling values ​​of two consecutive frames before processing, this method improves the inter-frame consistency of image processing, ensuring consistency in the beautification effect across adjacent frames of the video, i.e., a consistent degree of beautification.

[0113] like Figure 4 As shown, in the step S400, the pre-processed image of the current frame is processed through the second 3D lookup table to obtain the first processed image of the current frame, that is, the pre-processed image of the current frame is mapped and searched through the second 3D lookup table after discretization processing to obtain the corresponding first processed image of the current frame.

[0114] like Figure 4 As shown, in this embodiment, after the step S400 of processing the pre-processed image of the current frame through the second 3D lookup table to obtain the first processed image of the current frame, the following steps are also included:

[0115] The first processed image of the current frame is input into a pre-trained inter-frame consistency constraint network to obtain a second processed image of the current frame output by the inter-frame consistency constraint network.

[0116] The inter-frame consistency constraint network is trained using the following steps:

[0117] Acquire multiple frames of training images, and process each frame of the training image using a corresponding second lookup table to obtain a first processed image of each frame of the training image;

[0118] Here, each frame of training image can be a plurality of frames of training image extracted from the video to be processed, which can be continuous or discontinuous, or can be a plurality of frames of training image extracted from another video shot in the same scene as the video to be processed;

[0119] The second lookup table corresponding to each frame of the training image can be obtained by using the method of steps S200 and S300 above, that is, for each frame of the training image, the image and the previous frame of the image are discretized and then combined to obtain discrete sampling values, and then the discrete sampling values ​​are discretized in the first 3D lookup table to obtain the corresponding second 3D lookup table. Alternatively, the second lookup table corresponding to each frame of the training image can also be obtained by discretizing each frame of the training image to obtain discrete sampling values, and directly discretizing the first 3D lookup table to obtain the corresponding second 3D lookup table.

[0120] selecting a plurality of training groups from the first processed images of the plurality of training images, each of the training groups including two frames of the training images; the two training images in each training group may be two consecutive frames of training images or two discontinuous frames of training images, preferably two frames of training images having similar images, that is, two frames of training images with a short interval between them forming a training group;

[0121] In each iteration, two frames of the training images in one of the training groups are respectively input into the autoencoder to obtain two frames of constraint network output images respectively;

[0122] The inter-frame consistency constraint network may be an autoencoder, and the autoencoder may adopt an existing network structure for image pixel-level tasks, such as UResnet, but the present application is not limited thereto. The inter-frame consistency constraint network may also be a convolutional neural network or other neural network types;

[0123] The error value of the output images of the two-frame constraint network is used as a fitting target, and the network parameters of the inter-frame consistency constraint network are iteratively optimized to obtain an optimized inter-frame consistency constraint network.

[0124] Therefore, the present application can train an inter-frame consistency constraint network that effectively performs inter-frame consistency constraint processing on images by adopting a training group. The first processed image of the current frame is further processed by the constraint network to obtain the second processed image of the current frame, thereby further improving the consistency of the image adjustment effect between video frames and ensuring the consistency of the timing between video frames, that is, the output of similar scenes is consistent.

[0125] After the above steps S100 to S400, the image adjustment operation is implemented on the entire image of the current frame. However, in a video conferencing scenario, it may be necessary to only beautify the face area, and the background image needs to be consistent with the original image. Therefore, the first processed image of the current frame can be obtained in step S400 and then fused with the pre-processed image of the current frame.

[0126] Specifically, after obtaining the first processed image of the current frame in step S400, the following steps are further included:

[0127] Obtaining a first target mask area image for the first processed image of the current frame; for face beautification processing, the target mask area image is the facial mask area;

[0128] Performing a downsampling operation on the first target mask area to reduce the resolution, and then performing a bilateral filtering process to obtain a target mask area with a fuzzy boundary;

[0129] In the target mask area of ​​the fuzzy boundary, the mean operation is performed on each m×m area, where m is a positive integer greater than 1, to obtain the second target mask area. m can be 2, 3, 4, etc. The mean operation is to calculate the mean of all pixel values ​​in this m×m area, and then set all pixel values ​​in this m×m area to the mean, making the boundary transition more fine-grained and the grayscale more continuous, which is conducive to a more natural boundary transition in subsequent fusion.

[0130] After upsampling the second target mask area, a new target mask area with the same size as the first target mask area is obtained, and the new target mask area is fused pixel by pixel with the pre-processed image of the current frame to obtain the output image of the current frame. Here, pixel-by-pixel fusion means multiplying the new target mask area by the corresponding pixels of the pre-processed image of the current frame, replacing the pixel values ​​of the corresponding pixels in the pre-processed image of the current frame, and then using the fused image as the output image of the current frame.

[0131] Therefore, this embodiment performs downsampling operations, bilateral filtering operations, and averaging operations on the facial mask area of ​​the processed image after beautification, so as to make the transition of the facial contour boundary more natural and the beautification effect more realistic, and effectively solves the problem of obvious unnatural boundaries when traditional video portrait beautification algorithms perform local beautification.

[0132] In another embodiment, after executing steps S100 to S400 to obtain the first processed image of the current frame, the above-mentioned inter-frame consistency constraint network can be used to process the image to obtain a second processed image of the current frame, and then the second processed image of the current frame is fused with the pre-processed image of the current frame to obtain the output image of the current frame. That is, after obtaining the second processed image of the current frame, the following steps are performed:

[0133] Obtaining a first target mask area image for the second processed image of the current frame; for face beautification processing, the target mask area image is the facial mask area;

[0134] Performing a downsampling operation on the first target mask area to reduce the resolution, and then performing a bilateral filtering process to obtain a target mask area with a fuzzy boundary;

[0135] In the target mask area of ​​the fuzzy boundary, the mean operation is performed on each m×m area, where m is a positive integer greater than 1, to obtain the second target mask area. m can be 2, 3, 4, etc. The mean operation is to calculate the mean of all pixel values ​​in this m×m area, and then set all pixel values ​​in this m×m area to the mean, making the boundary transition more fine-grained and the grayscale more continuous, which is conducive to a more natural boundary transition in subsequent fusion.

[0136] After upsampling the second target mask area, a new target mask area with the same size as the first target mask area is obtained, and the new target mask area is fused pixel by pixel with the pre-processed image of the current frame to obtain the output image of the current frame. Here, pixel-by-pixel fusion means multiplying the new target mask area by the corresponding pixels of the pre-processed image of the current frame, replacing the pixel values ​​of the corresponding pixels in the pre-processed image of the current frame, and then using the fused image as the output image of the current frame.

[0137] Therefore, this implementation, on the basis of further improving the inter-frame consistency through inter-frame consistency constraints, performs downsampling operations, bilateral filtering operations, and averaging operations on the facial mask area of ​​the processed image after beautification. This can make the transition of the facial contour boundary more natural and the beautification effect more realistic, effectively solving the problem of obviously unnatural boundaries when traditional video portrait beautification algorithms perform local beautification.

[0138] Here is an example of fusing the pre-processed and post-processed images of the current frame. In implementations where adjustments are required to the entire image, fusion is not necessary, and the post-processed image of the current frame is directly used as the output image of the current frame. Alternatively, fusion can be performed by directly replacing the target mask area of ​​the post-processed image of the current frame with the corresponding area of ​​the post-processed image of the current frame. All of the above variations fall within the scope of protection of this application.

[0139] In summary, the video image processing method of this embodiment is fully automatic and controllable, avoiding the need to formulate corresponding processes based on specific rules, simplifying the steps, and solving the problems of poor robustness and unnatural portrait beautification caused by data problems and the setting of fixed 3D lookup tables. In addition, the problem of inter-frame jumps and inconsistent portrait beautification levels in actual use of traditional video portrait beautification algorithms is solved. In view of the need to beautify only the face area in the video conferencing scenario, while the background image needs to be consistent with the original image, the post-beautification processed image can be added on the basis of the above steps to be fused with the pre-processed image, and the problem of unnatural boundaries during local beautification can be effectively solved.

[0140] like Figure 5 As shown, the embodiment of the present application further provides a video image processing system, which is applied to the video image processing method, and the system includes:

[0141] A lookup table acquisition module M100 is configured to acquire a first 3D lookup table;

[0142] The discrete processing module M200 is configured to discretize the pre-processed image of the current frame and the pre-processed image of the previous frame as two consecutive pre-processed frames, obtain discretization results, combine the discretization results of the two consecutive pre-processed frames to obtain a set of discrete sampling values, and discretize the first 3D lookup table based on the discrete sampling values ​​to obtain a second 3D lookup table corresponding to the current frame.

[0143] The image processing module M300 is configured to process the pre-processed image of the current frame using a second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame.

[0144] After the present invention obtains the first 3D lookup table through the lookup table acquisition module M100, the first 3D lookup table is discretized based on two consecutive frames of pre-processed images by the discrete processing module M200 to obtain a discretized second 3D lookup table. The image processing module M300 uses the second 3D lookup table combined with the discretization processing of the two frames of pre-processed images to process the current frame, so that the discrete values ​​of the second 3D lookup table used for the current frame processing are also associated with the previous frame image. When the second 3D lookup table is used to process the pre-processed image of the current frame, the image information of the two consecutive frames of images is also considered at the same time, which is conducive to improving the consistency of the video image processing process, thereby improving the inter-frame consistency of the processed image and solving the problems of picture jump and inconsistent image adjustment degree in the video scene. This method can be applied to face beauty adjustment in video conferencing, and can also be applied to the adjustment processing of other categories of targets in the video (such as vehicles, pedestrians, obstacles, etc.), and can also be used for the adjustment processing of the overall image in the video. Here, image adjustment refers to the adjustment of one or more parameters of the image, for example, it can include one or more aspects of parameter adjustment such as brightness adjustment, saturation adjustment, and color adjustment.

[0145] An embodiment of the present application also provides a video image processing device, comprising a processor; a memory storing executable instructions of the processor; wherein the processor is configured to perform the steps of the video image processing method by executing the executable instructions.

[0146] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0147] Refer to the following Figure 6 hereinafter, an electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0148] like Figure 6 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, at least one processing unit 610, at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), a display unit 640, and the like.

[0149] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present application described in the above-mentioned electronic prescription circulation processing method section of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0150] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0151] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0152] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0153] The electronic device 600 can also communicate with one or more external devices 700 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. The network adapter 660 can communicate with other modules of the electronic device 600 via the bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0154] By adopting the video image processing device provided by the present application, the processor executes the video image processing method when executing the executable instructions, thereby achieving the beneficial effects of the above-mentioned video image processing method.

[0155] The present application also provides a computer-readable storage medium for storing a program that, when executed by a processor, implements the steps of the video image processing method. In some possible implementations, various aspects of the present application may also be implemented in the form of a program product, which includes program code. When the program product is executed on a terminal device, the program code is used to cause the terminal device to execute the steps of the various exemplary embodiments of the present application described in the video image processing method section above.

[0156] refer to Figure 7 As shown, a program product 800 for implementing the above method according to an embodiment of the present application is described. The program product 800 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0157] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0158] The computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein the readable program code is carried. The data signal propagated may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or component. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0159] The program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or cluster. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0160] By adopting the computer-readable storage medium provided in the present application, the program stored therein implements the steps of the video image processing method when being executed, thereby achieving the beneficial effects of the above-mentioned video image processing method.

[0161] The above content is a further detailed description of the present application in conjunction with specific preferred embodiments, and the specific implementation of the present application cannot be considered to be limited to these descriptions. For ordinary technicians in the technical field to which the present application belongs, several simple deductions or substitutions can be made without departing from the concept of the present application, and all of them should be considered to fall within the scope of protection of the present application.

Claims

1. A video image processing method, characterized in that: The steps include: Obtain a first 3D lookup table, where the dimension of the first 3D lookup table is N×N×N, where N is the number of vertices divided by each RGB channel; The pre-processing image of the current frame and the pre-processing image of the previous frame are used as two consecutive pre-processing images, and discretization processing is performed on each of them to obtain discretization processing results. The discretization processing results corresponding to the pre-processing image of the current frame and the discretization processing results corresponding to the pre-processing image of the previous frame are combined or fused to obtain a set of discrete sampling values; performing discretization processing on the first 3D lookup table based on the discrete sampling values ​​to reduce the dimension of the first 3D lookup table to obtain a second 3D lookup table corresponding to the current frame, wherein the size of the second 3D lookup table is smaller than that of the first 3D lookup table; The pre-processed image of the current frame is processed through a second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame.

2. The video image processing method according to claim 1, wherein: The step of obtaining the first 3D lookup table comprises the following steps: obtaining at least one image adjustment degradation kernel; Acquire a plurality of first adjusted images, and obtain a first unadjusted image corresponding to each of the first adjusted images based on the image adjustment degradation kernel, and use the first adjusted images and the first unadjusted images as training image data; Build the initial 3D lookup table; The initial 3D lookup table is optimized based on the training image data to obtain the first 3D lookup table.

3. The video image processing method according to claim 2, characterized in that: The step of obtaining at least one image adjustment degradation kernel comprises the following steps: constructing at least one image adjustment degradation kernel; Acquiring a plurality of second adjusted images and a plurality of second unadjusted images; Extracting first edge features from the plurality of second adjusted images, and inputting the extracted first edge features and the plurality of image adjustment degradation kernels into a generative adversarial network; classifying the output image of the generative adversarial network and the second unadjusted image through a classifier network, and optimizing the at least one image adjustment degradation kernel through a classification loss function of the classifier network; Correspondingly, obtaining the corresponding first unadjusted image based on the image adjustment degradation kernel includes: inputting the collected first adjusted image and the optimized image adjustment degradation kernel into the generative adversarial network to obtain the corresponding first unadjusted image output by the generative adversarial network.

4. The video image processing method according to claim 1, wherein: The step of obtaining the first 3D lookup table comprises the following steps: Acquire training image data including a first unadjusted image and a first adjusted image; After multiplying the first unadjusted image by a preset global weight factor according to preset scene requirements, updating the first unadjusted image to obtain updated training image data; Build the initial 3D lookup table; The initial 3D lookup table is optimized based on the updated training image data to obtain the first 3D lookup table.

5. The video image processing method according to claim 1, wherein: The step of obtaining the first 3D lookup table comprises the following steps: Construct multiple initialized sub-lookup tables of different categories; Obtaining a value of each of the initialization sub-lookup tables and a weight value of each of the initialization sub-lookup tables based on training image data including a first unadjusted image and a first adjusted image; The initialization sub-lookup tables are weightedly fused based on the weight values ​​of the initialization sub-lookup tables to obtain the first 3D lookup table.

6. The video image processing method according to claim 1, characterized in that: After the pre-processed image of the current frame is processed by the second 3D lookup table to obtain the first processed image of the current frame, the following steps are further included: Inputting the first processed image of the current frame into a pre-trained inter-frame consistency constraint network to obtain a second processed image of the current frame output by the inter-frame consistency constraint network; The inter-frame consistency constraint network is trained using the following steps: Acquire multiple frames of training images, and process each frame of the training images using a corresponding second 3D lookup table to obtain a first processed image of each frame of the training images; Selecting a plurality of training groups from the first processed images of the plurality of frames of training images, each of the training groups including two frames of training images; In each iteration, two frames of the training images in one of the training groups are respectively input into the inter-frame consistency constraint network, and two frames of constraint network output images are respectively obtained; The error value of the output images of the two-frame constraint network is used as a fitting target, and the network parameters of the inter-frame consistency constraint network are iteratively optimized to obtain an optimized inter-frame consistency constraint network.

7. The video image processing method according to claim 1, characterized in that: After obtaining the first processed image of the current frame, the following steps are also included: Acquire a first target mask area image from the first processed image of the current frame; After performing downsampling and bilateral filtering on the first target mask area, performing a mean operation on each m×m area, where m is a positive integer greater than 1, to obtain a second target mask area; The second target mask area is up-sampled and fused with the unprocessed image of the current frame to obtain an output image of the current frame.

8. A video image processing system, characterized in that: The video image processing method according to any one of claims 1 to 7, wherein the system comprises: A lookup table acquisition module, configured to acquire a first 3D lookup table; a discrete processing module configured to discretize the pre-processed image of the current frame and the pre-processed image of the previous frame as two consecutive pre-processed frames, obtain discretization results, combine the discretization results of the two consecutive pre-processed frames to obtain a set of discrete sampling values, and discretize the first 3D lookup table based on the discrete sampling values ​​to obtain a second 3D lookup table corresponding to the current frame; The image processing module is configured to process the pre-processed image of the current frame through a second 3D lookup table corresponding to the current frame to obtain a first processed image of the current frame.

9. A video image processing device, characterized in that: include: processor; a memory storing executable instructions for the processor; The processor is configured to execute the steps of the video image processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the video image processing method according to any one of claims 1 to 7 are implemented.

Images