Processing method and processing device
By converting RGB format face images to YCrCb space and using a lightweight depthwise separable convolution model to process the Y channel data, the problem of blurry images caused by insufficient network bandwidth in video conferencing is solved, achieving efficient image restoration and sharpness improvement.
Patent Information
- Application Number
- CN202411061053.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-19
AI Technical Summary
In video conferencing, insufficient network bandwidth can lead to blurry images and reduced image quality, affecting participants' ability to clearly perceive facial expressions and gestures and their overall communication experience.
By converting RGB format face images to YCrCb space, a lightweight depthwise separable convolutional model is used to process the Y channel data, and the Cr and Cb channel data are superimposed on the Y channel restoration result to achieve efficient image restoration.
It effectively reduces the computational load of the model, improves image clarity, and enables users to perceive the facial details of participants more clearly in video conferences, thus enhancing the communication experience.
Smart Images

Figure CN118982490A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and more specifically, to a processing method and a processing device. Background Art
[0002] With the development of the economy and society, the application scenarios of video conferencing have gradually increased, and it has become an indispensable communication tool in all walks of life. Video conferencing has not only improved work efficiency, but also promoted information exchange and cooperation across regions and borders.
[0003] In video conferencing applications, the image is often blurred due to insufficient network bandwidth. This is usually manifested as blurred details, motion freezes, or reduced image quality, affecting participants' clear perception and communication experience of each other's facial expressions, gestures, and conference content. Summary of the invention
[0004] The present application provides a processing method and a processing device, which can more efficiently restore low-definition images to high-definition images, so that users can more clearly perceive the facial details of participants in a video conference, thereby improving the communication experience of the video conference.
[0005] In a first aspect, a processing method is provided, the method comprising: acquiring a first facial image, the first facial image being an image in RGB format; performing format conversion on the first facial image to obtain Y channel data, Cr channel data and Cb channel data in YCrCb space; inputting the Y channel data into a first model to obtain a second facial image; performing size alignment on the Cr channel data and the Cb channel data with the second facial image to obtain a first processing result, and superimposing the first processing result on the second facial image to obtain a third facial image, the third facial image being an image in the YCrCb format; performing format conversion on the third facial image to obtain a fourth facial image, the fourth facial image being an image in the RGB format, and the clarity of the fourth facial image is higher than the clarity of the first facial image.
[0006] In an embodiment of the present application, the first model is used to process the Y channel data, and then the Cr channel data and the Cb channel data are superimposed on the processing result of the Y channel data to perform image restoration. Compared with directly processing images in RGB format, the calculation amount of the model can be greatly reduced, thereby more efficiently restoring low-definition images to high-definition images, allowing users to more clearly perceive the facial details of participants in video conferences, thereby improving the communication experience of video conferences.
[0007] In combination with the first aspect, in some implementations of the first aspect, the first model includes a depthwise separable convolutional layer, and the depthwise separable convolutional layer consists of two layers of structure: depthwise convolution and pointwise convolution.
[0008] In an embodiment of the present application, the network structure of the first model is improved by using depthwise separable convolution instead of ordinary convolution. Since the depthwise separable convolution consists of two layers of structure, namely depthwise convolution and pointwise convolution, it has less computational complexity than ordinary convolution and does not reduce the reasoning performance of the neural network, thus enabling the first model to obtain high-quality facial images more efficiently.
[0009] In combination with the first aspect, in certain implementations of the first aspect, the first model is obtained by pruning and fine-tuning the second model, the second model is obtained based on a first training sample, and the first training sample includes: sample Y channel data and a sample second face image.
[0010] In the embodiment of the present application, the first model used is obtained by pruning and fine-tuning the second model, rather than directly applying the trained second model. In this way, unnecessary connections and parameters in the second model can be removed, thereby reducing the complexity and computational complexity of the model, making the model more lightweight and efficient, and helping to improve the performance and generalization ability of the model in practical applications.
[0011] In a second aspect, a model training method is provided, including: obtaining a second model, the second model is obtained based on a first training sample, the first training sample includes: sample Y channel data and a sample second face image; pruning the second model to obtain the first model; obtaining Y channel data, the Y channel data is obtained by format conversion of the first face image, the first face image is an image in RGB format; inputting the Y channel data into the first model to obtain a second processing result; inputting the Y channel data into the second model to obtain a third processing result; training the first model according to the second processing result, the third processing result and the mean absolute error loss function.
[0012] In the embodiment of the present application, by training the first model through the second processing result, the third processing result and the average error loss function, a first model with higher accuracy can be obtained, so as to facilitate face restoration through the first model.
[0013] In combination with the second aspect, in some implementations of the second aspect, the method further includes: using a convolution kernel to transform the number of intermediate layer feature channels of the first model to be the same as the number of intermediate layer feature channels of the second model.
[0014] In an embodiment of the present application, by using a convolution kernel to transform the number of intermediate layer feature channels of the first model to be the same as the number of intermediate layer feature channels of the second model, the training difficulty of the first model can be reduced, helping the first model to quickly converge to optimal performance.
[0015] According to a third aspect, a processing device is provided, comprising: an acquisition unit and a processing unit; the acquisition unit is used to acquire a first facial image, the first facial image being an image in RGB format; the processing unit is used to: perform format conversion on the first facial image to obtain Y channel data, Cr channel data and Cb channel data in YCrCb space; input the Y channel data into a first model to obtain a second facial image; size-align the Cr channel data and the Cb channel data with the second facial image to obtain a first processing result, and superimpose the first processing result on the second facial image to obtain a third facial image, the third facial image being an image in YCrCb format; perform format conversion on the third facial image to obtain a fourth facial image, the fourth facial image being an image in RGB format, and the clarity of the fourth facial image is higher than that of the first facial image.
[0016] In combination with the third aspect, in some implementations of the third aspect, the first model includes a depthwise separable convolutional layer, and the depthwise separable convolutional layer consists of two layers of structure: depthwise convolution and pointwise convolution.
[0017] In combination with the third aspect, in certain implementations of the third aspect, the first model is obtained by pruning and fine-tuning the second model, the second model is obtained based on a first training sample, and the first training sample includes: sample Y channel data and a sample second face image.
[0018] In a fourth aspect, a model training device is provided, comprising: an acquisition unit and a processing unit; the acquisition unit is used to acquire a second model, wherein the second model is obtained based on a first training sample, wherein the first training sample comprises: sample Y channel data and a sample second face image; the processing unit is used to prune the second model to obtain the first model; the acquisition unit is used to acquire Y channel data, wherein the Y channel data is obtained by format conversion of the first face image, wherein the first face image is an image in RGB format; the processing unit is further used to: input the Y channel data into the first model to obtain a second processing result; input the Y channel data into the second model to obtain a third processing result; and train the first model according to the second processing result, the third processing result and the mean absolute error loss function.
[0019] In combination with the fourth aspect, in certain implementations of the fourth aspect, the processing unit is further used to use a convolution kernel to transform the number of intermediate layer feature channels of the first model to be the same as the number of intermediate layer feature channels of the second model.
[0020] In a fifth aspect, a processing device is provided, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory, so that the device implements the method in any one of the implementation modes in the above-mentioned first aspect.
[0021] In a sixth aspect, a model training device is provided, comprising: at least one processor and a memory, wherein the at least one processor is coupled to the memory and is used to read and execute instructions in the memory, so that the device implements the method in any one of the implementation modes in the above-mentioned second aspect.
[0022] In a seventh aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program code, and when the computer program code runs on a computer, the computer executes a method in any one of the implementations of the first aspect or the second aspect.
[0023] In an eighth aspect, a chip is provided, the chip comprising a circuit, the circuit being used to execute the method in any one of the implementations of the first aspect or the second aspect.
[0024] In a ninth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the method in any one of the implementations of the first aspect or the second aspect is executed. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a schematic flow chart of a processing method provided in an embodiment of the present application;
[0026] Figure 2 is a schematic flow chart of another processing method provided in an embodiment of the present application;
[0027] Figure 3 This is a schematic diagram of comparing the clarity of an image before and after processing provided by an embodiment of the present application;
[0028] Figure 4 is a schematic flow chart of a model training method provided in an embodiment of the present application;
[0029] Figure 5 is a schematic flow chart of a lightweight model training provided in an embodiment of the present application;
[0030] Figure 6is a schematic diagram of a processing device or a model training device provided in an embodiment of the present application;
[0031] Figure 7 It is a schematic diagram of a processing device or a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0032] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.
[0033] The prefixes such as "first" and "second" used in the embodiments of the present application are only used to distinguish different description objects, and have no limiting effect on the position, order, priority, quantity or content of the described objects. The use of prefixes such as ordinal numbers used to distinguish description objects in the embodiments of the present application does not constitute a limitation on the described objects. For the statement of the described objects, please refer to the description in the context of the claims or embodiments, and the use of such prefixes should not constitute an unnecessary limitation.
[0034] The technical solution in this application will be described below in conjunction with the accompanying drawings.
[0035] With the development of the economy and society, the application scenarios of video conferencing have gradually increased, and it has become an indispensable communication tool in all walks of life. Video conferencing has not only improved work efficiency, but also promoted information exchange and cooperation across regions and borders.
[0036] In video conferencing applications, the image is often blurred due to insufficient network bandwidth. This is usually manifested as blurred details, motion freezes, or reduced image quality, affecting participants' clear perception and communication experience of each other's facial expressions, gestures, and conference content.
[0037] The embodiments of the present application provide a processing method and a processing device that can more efficiently restore low-definition images to high-definition images, so that users can more clearly perceive the facial details of participants in a video conference, thereby improving the communication experience of the video conference.
[0038] Figure 1 It is a schematic flowchart of a processing method provided in an embodiment of the present application. Method 100 may include steps S101 to S105.
[0039] S101, obtaining a first face image.
[0040] The first face image is an image in RGB format.
[0041] S102, converting the format of the first face image to obtain Y channel data, Cr channel data and Cb channel data in a YCrCb space.
[0042] Among them, the YCrCb space is a way to separate RGB color components. It can contain three independent color channels, namely the Y channel (brightness component), the Cr channel (red chrominance component) and the Cb channel (blue chrominance component); the Y channel can be used to represent the brightness information of the image, which can be related to the perceived brightness of the human eye; the Cr channel can represent the difference between red chrominance and brightness, and the Cb channel can represent the difference between blue chrominance and brightness.
[0043] S103, inputting the Y channel data into the first model to obtain a second face image.
[0044] Among them, in the present application, the first model can also be called a lightweight model, and the second face image can be the restoration result of the Y channel data, that is, the face image of the Y channel.
[0045] In a possible implementation, the first model includes a depthwise separable convolution layer, which is composed of two layers of structures: depthwise convolution and pointwise convolution. In this way, since the depthwise separable convolution has a smaller amount of computation than ordinary convolution and does not reduce the reasoning performance of the neural network, the first model can more efficiently obtain high-quality face images.
[0046] In a possible implementation, the first model is obtained by pruning and fine-tuning the second model, and the second model is obtained based on the first training sample, and the first training sample includes: sample Y channel data and sample second face image. In this way, unnecessary connections and parameters in the second model can be removed, thereby reducing the complexity and calculation amount of the model, making the model more lightweight and efficient, and helping to improve the performance and generalization ability of the model in practical applications.
[0047] S104, aligning the size of the Cr channel data and the Cb channel data with the second face image to obtain a first processing result, and superimposing the first processing result on the second face image to obtain a third face image.
[0048] The third face image is an image in YCrCb format.
[0049] S105: Convert the format of the third facial image to obtain a fourth facial image.
[0050] The fourth facial image is an image in RGB format, and the clarity of the fourth facial image is higher than that of the first facial image.
[0051] In an embodiment of the present application, the first model is used to process the Y channel data, and then the Cr channel data and the Cb channel data are superimposed on the processing result of the Y channel data to perform image restoration. Compared with directly processing images in RGB format, the calculation amount of the model can be greatly reduced, thereby more efficiently restoring low-definition images to high-definition images, allowing users to more clearly perceive the facial details of participants in video conferences, thereby improving the communication experience of video conferences.
[0052] Figure 2 It is a schematic flow chart of another processing method provided in an embodiment of the present application. Method 200 may be a detailed introduction to steps S101 to S105 in method 100.
[0053] like Figure 2 As shown, a low-definition face image in RGB format (corresponding to the first face image in method 100) can be first obtained, and the low-definition face image can be converted into the YCrCb space to obtain Y channel data, and Cr and Cb channel data respectively. Then, the Y channel data can be input into the lightweight model to obtain the image restoration result of the Y channel (corresponding to the second face image in method 100); for the Cr and Cb channel data, the Cr and Cb size alignment results can be obtained by resizing (corresponding to the first processing result in method 100), that is, the size of the low-definition image corresponding to the Cr and Cb channel data can be adjusted to be consistent with the size of the target high-definition image. Finally, the image restoration result of the Y channel can be combined with the Cr and Cb size alignment results to obtain a YCrCb space image (corresponding to the third face image in method 100), and the YCrCb space image can be converted into the RGB space to obtain a restored high-definition face image (corresponding to the fourth face image in method 100).
[0054] Optionally, the pixels in the face image in RGB format may be converted to corresponding pixels in the YCrCb space by linear transformation.
[0055] For example, Figure 3 As shown, the low-definition face image can be restored to a high-definition face image through the above processing method.
[0056] In the embodiment of the present application, since the Cr and Cb channels are color channels and contain less detail information (to which the human eye is not sensitive), a simple resizing operation can be used to align the images corresponding to the Cr and Cb channel data with the high-definition image size. The Y channel includes rich detail feature information, and a lightweight model is used to process the Y channel data. The computational complexity of the lightweight model is greatly reduced, thereby improving the reasoning efficiency of the model and reducing the difficulty of training the model.
[0057] Figure 4 It is a schematic flowchart of a model training method provided in an embodiment of the present application. Method 400 may include steps S401 to S406.
[0058] S401, obtaining a second model.
[0059] The second model is obtained based on the first training sample, and the first training sample includes: sample Y channel data and a sample second face image.
[0060] S402: Prune the second model to obtain the first model.
[0061] Among them, through pruning, unnecessary parameters in the second model can be reduced. These parameters may contribute little to the final face restoration effect or are redundant. Pruning can help the model execute faster during inference and reduce the computational cost at runtime.
[0062] S403, obtaining Y channel data.
[0063] Exemplarily, a first facial image in RGB format may be first acquired, and the format of the first facial image may be converted to obtain data of a Y channel.
[0064] S404, input the Y channel data into the first model to obtain a second processing result.
[0065] The second processing result can be understood as the face image of the Y channel output by the first model.
[0066] S405, input the Y channel data into the second model to obtain a third processing result.
[0067] Among them, the third processing result can be understood as the face image of the Y channel output by the second model.
[0068] S406, training the first model according to the second processing result, the third processing result and the mean absolute error loss function.
[0069] Exemplarily, the specific process of training the first model may be to constrain the second processing result and the third processing result through a mean absolute error loss function, so that the second processing result output by the first model gradually approaches the third processing result.
[0070] In one possible implementation, training the first model also includes: using a convolution kernel to transform the number of intermediate layer feature channels of the first model to be the same as the number of intermediate layer feature channels of the second model, thereby reducing the training difficulty of the first model and helping the first model to quickly converge to optimal performance.
[0071] Optionally, the convolution kernel may be a 1×1 convolution kernel.
[0072] In the embodiment of the present application, by training the first model through the second processing result, the third processing result and the average error loss function, a first model with higher accuracy can be obtained, so as to facilitate face restoration through the first model.
[0073] Combine the following Figure 5 The training process of the first model is introduced in detail.
[0074] Figure 5 It is a schematic flowchart of lightweight model training and deployment provided in an embodiment of the present application. Method 500 may include steps S501 to S504.
[0075] S501, train a lightweight styleGAN model.
[0076] Specifically, styleGAN is a generative model that can generate realistic facial images. It generates images with realistic visual effects by learning the distribution characteristics of large-scale face datasets. The model input of styleGAN is an N-dimensional vector, which can be mapped into 8 different vectors through a fully connected layer. Each vector is input into the corresponding generation module (a total of 8 generation modules). Each generation module can be responsible for generating a part of the features or structure of the image. Finally, by combining the processing results of each generation module, a high-quality image can be obtained (for example, the face pixel size is set to 256×256).
[0077] Among them, for each generation module, styleGAN can be implemented using convolution, and the specific processing process can be expressed as follows:
[0078] style ′ =modulation(style)
[0079] weight ′ =style ′ ×weight
[0080] demod = demodulation(weight ′ )
[0081] output=conv(input,demod×weight ′ )
[0082] Among them, style is the feature vector of the corresponding generation module obtained after the N-dimensional vector is processed by the fully connected layer; modulation represents the fully connected operation; weight represents the convolutional layer weight; demodulation represents the execution of the square root operation.
[0083] For the lightweight styleGAN model trained in step S501, a depthwise separable convolution can be used instead of a normal convolution. The depthwise separable convolution consists of two layers of structures: depthwise convolution and pointwise convolution. Compared with the normal convolution, the computational complexity is smaller and the performance of the neural network is not reduced. Since the ordinary single-layer convolution needs to be split into two layers, the above calculation formula for the calculation of the convolution weight also needs to be modified. The specific processing process can be expressed as follows:
[0084] style ′ =modulation(style)
[0085] input ′ =style ′ ×input
[0086] output ′ =pwconv(dwconv(input ′ ,dw_weight),pw_weight)
[0087] demod=demodulation(pw_weight×dw_weight)
[0088] output=output ′ ×demod
[0089] S502, add the unet noise removal module and jointly train it with the lightweight styleGAN model to obtain the face restoration model GFPGAN for the Y channel.
[0090] Specifically, GFPGAN (which may correspond to the second model of method 300 and method 400) may be composed of two parts, namely, the styleGAN model obtained in step S501 and the unet noise removal module. The input of the unet noise removal module is a low-definition face image. The unet noise removal module may include 8 down-sampled residual convolution blocks and 8 up-sampled residual convolution blocks. The unet noise removal module may flatten the feature map obtained after the low-definition face image passes through 8 down-sampled residual convolution blocks to obtain an N-dimensional vector F_latent as the input of the styleGAN model. In the up-sampling part of unet, the feature map output of each up-sampled residual convolution block is recorded as F_spatial, which is used to correct the same-scale feature layer of styleGAN. The specific formula is as follows:
[0091] α=Conv1(F_spatial)
[0092] β=Conv2(F_spatial)
[0093] F_output=Concat(F_GAN_split0,α·F_GAN_split1+β)
[0094] Among them, α and β are the results obtained after F_spatial passes through two convolutional layers respectively; F_GAN_split0 and F_GAN_split1 are the splitting of the feature output of the intermediate layer of styleGAN in the feature channel dimension (for example, splitting the feature map of size b×256×h×w into two sub-maps of size b×128×h×w, where b is the batch size, h is the image height, and w is the image width).
[0095] S503, pruning and fine-tuning the trained GFPGAN model to obtain a lightweight model.
[0096] Among them, the lightweight model obtained after pruning and fine-tuning can be the first model in method 300 and method 400.
[0097] Among them, the trained GFPGAN can be recorded as model T, and the pruning operation can be implemented through the torch_pruning pruning framework. Specifically, the Taylor pruner can be selected. After the sparse training, the model T is pruned 5 times, and each pruning reduces the number of convolutions and channels by 5%, until a lightweight model with 25% of the parameters is finally obtained. During the iterative pruning process, the pruned model can be fine-tuned.
[0098] Exemplarily, for the fine-tuning process after a single pruning, the embodiment of the present application designs a special fine-tuning loss function, which can be composed of the following two parts.
[0099] (1) Taking the fine-tuning after the first pruning of 5% parameters as an example, the model after the first pruning is recorded as model S. The low-definition image is input into model T and model S respectively to obtain the output results of the two models. The mean absolute error loss function L1 Loss is used to constrain the output results of the two models to be similar.
[0100] (2) Since model S reduces the number of convolution channels, the number of channels of the intermediate layer features Flat and Spatial also decreases. By using a 1×1 convolution kernel, the number of channels of the intermediate layer features of model S is transformed to the same as the number of channels of the intermediate layer of model T, and L1 Loss is used to constrain their corresponding features to be similar.
[0101] It should be noted that in the process of fine-tuning model S, the parameters of model T need to be fixed and no longer change; the 1×1 convolution kernel parameters described in (2) are also fine-tuned together with model S. After obtaining the fine-tuned model S, this layer of convolution can be directly deleted without affecting the reasoning process of the final lightweight model. In the embodiment of the present application, the output results of the intermediate layer of model T are used to guide the adjustment of the parameters of model S. The reason is that some parameters of model S and model T are the same and the performance is similar, which can reduce the difficulty of model training and help model S quickly converge to the optimal performance.
[0102] S504, converting the lightweight model into a (mobile neural network, mnn) or (tensorflowlite, tflite) format file for edge terminal deployment.
[0103] Among them, edge deployment can be understood as: using lightweight models for inference and processing in mobile devices, embedded systems and other devices.
[0104] In an embodiment of the present application, a lightweight model with good performance can be trained through the above method, so that the lightweight model can be used to perform image restoration on low-definition facial images.
[0105] It should be understood that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.
[0106] The embodiments of the present application also provide a device for implementing any of the above methods, wherein the device includes a unit for implementing each step performed in any of the above methods.
[0107] Figure 6 It is a schematic diagram of a processing device or a model training device provided in an embodiment of the present application.
[0108] The device 600 may include an acquisition unit 610, a storage unit 620, and a processing unit 630. The acquisition unit 610 is used to acquire instructions and / or data, and the acquisition unit 610 may also be referred to as a communication interface or a communication unit. The storage unit 620 is used to implement a corresponding storage function and store corresponding instructions and / or data. The processing unit 630 is used to perform data processing so that the device 600 implements the aforementioned processing method or model training method.
[0109] In one embodiment, the device 600 includes: an acquisition unit 610 and a processing unit 630; the acquisition unit 610 is used to acquire a first facial image, and the first facial image is an image in RGB format; the processing unit 630 is used to: convert the format of the first facial image to obtain Y channel data, Cr channel data and Cb channel data in YCrCb space; input the Y channel data into the first model to obtain a second facial image; align the Cr channel data and the Cb channel data with the second facial image to obtain a first processing result, and superimpose the first processing result on the second facial image to obtain a third facial image, and the third facial image is an image in YCrCb format; convert the format of the third facial image to obtain a fourth facial image, and the fourth facial image is an image in RGB format, and the clarity of the fourth facial image is higher than that of the first facial image.
[0110] In a possible implementation, the first model includes a depthwise separable convolutional layer, which is composed of two layers of structures: depthwise convolution and pointwise convolution.
[0111] In a possible implementation, the first model is obtained by pruning and fine-tuning the second model, and the second model is obtained based on a first training sample, and the first training sample includes: sample Y channel data and a sample second face image.
[0112] In one embodiment, the device 600 includes: an acquisition unit 610 and a processing unit 630; the acquisition unit 610 is used to acquire a second model, the second model is obtained based on a first training sample, and the first training sample includes: sample Y channel data and a sample second face image; the processing unit 630 is used to prune the second model to obtain the first model; the acquisition unit 610 is used to acquire Y channel data, the Y channel data is obtained by format conversion of the first face image, and the first face image is an image in RGB format; the processing unit 630 is also used to: input the Y channel data into the first model to obtain a second processing result; input the Y channel data into the second model to obtain a third processing result; and train the first model according to the second processing result, the third processing result and the mean absolute error loss function.
[0113] In one embodiment, the processing unit 630 is further configured to use a convolution kernel to transform the number of intermediate layer feature channels of the first model to be the same as the number of intermediate layer feature channels of the second model.
[0114] Figure 7 It is a schematic diagram of another processing device or model training device provided in an embodiment of the present application.
[0115] The device 700 includes: a memory 710, a processor 720, and a communication interface 730. The memory 710, the processor 720, and the communication interface 730 are connected through an internal connection path, the memory 710 is used to store instructions, and the processor 720 is used to execute the instructions stored in the memory 710 to control the communication interface 730 to obtain information, or to enable the device 700 to implement the aforementioned processing method or model training method. Optionally, the memory 710 can be coupled to the processor 720 through an interface, or can be integrated with the processor 720.
[0116] It should be noted that the communication interface 730 uses a transceiver device such as, but not limited to, a transceiver. The communication interface 730 may also include an input / output interface.
[0117] The processor 720 stores one or more computer programs, which include instructions. When the instructions are executed by the processor 720, the device 700 executes the processing method or model training method in the above-mentioned embodiments.
[0118] In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 720 or the instruction in the form of software. The method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 710, and the processor 720 reads the information in the memory 710 and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.
[0119] Optionally, Figure 7 The communication interface 730 in the embodiment can be implemented Figure 6 The acquisition unit 610 in Figure 7 The memory 710 in the embodiment can realize Figure 6 The storage unit 620, Figure 7 The processor 720 in the embodiment can implement Figure 6 The processing unit 630 in.
[0120] The embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code, and when the computer program code is executed on a computer, the computer executes the above Figures 1 to 5 Any of the methods in .
[0121] The present application also provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above Figures 1 to 5 Any of the methods in .
[0122] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0124] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0127] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A processing method, characterized in that: The method comprises: Acquire a first face image, where the first face image is an image in RGB format; Convert the first face image into a new format to obtain Y channel data, Cr channel data and Cb channel data in a YCrCb space; Inputting the Y channel data into the first model to obtain a second face image; Align the size of the Cr channel data and the Cb channel data with the second face image to obtain a first processing result, and superimpose the first processing result on the second face image to obtain a third face image, wherein the third face image is an image in the YCrCb format; The third facial image is format-converted to obtain a fourth facial image, wherein the fourth facial image is an image in the RGB format and has a higher definition than the first facial image.
2. The method according to claim 1, characterized in that The first model includes a depthwise separable convolutional layer, which consists of two layers of structures: depthwise convolution and pointwise convolution.
3. The method according to claim 1 or 2, characterized in that The first model is obtained by pruning and fine-tuning the second model, and the second model is obtained based on a first training sample, wherein the first training sample includes: sample Y channel data and a sample second face image.
4. A model training method, characterized in that: The method comprises: Acquire a second model, where the second model is obtained based on a first training sample, where the first training sample includes: sample Y channel data and a sample second face image; Pruning the second model to obtain a first model; Acquire Y channel data, where the Y channel data is obtained by format conversion of the first face image, where the first face image is an image in RGB format; Inputting the Y channel data into the first model to obtain a second processing result; Inputting the Y channel data into the second model to obtain a third processing result; The first model is trained according to the second processing result, the third processing result and the mean absolute error loss function.
5. The method according to claim 4, characterized in that The method further comprises: The number of feature channels of the intermediate layer of the first model is transformed to be the same as the number of feature channels of the intermediate layer of the second model using a convolution kernel.
6. A processing device, characterized in that: The device comprises: an acquisition unit and a processing unit; The acquisition unit is used to acquire a first face image, where the first face image is an image in RGB format; The processing unit is used for: Convert the first face image into a new format to obtain Y channel data, Cr channel data and Cb channel data in a YCrCb space; Inputting the Y channel data into the first model to obtain a second face image; Align the size of the Cr channel data and the Cb channel data with the second face image to obtain a first processing result, and superimpose the first processing result on the second face image to obtain a third face image, wherein the third face image is an image in the YCrCb format; The third facial image is format-converted to obtain a fourth facial image, wherein the fourth facial image is an image in the RGB format and has a higher definition than the first facial image.
7. The device according to claim 6, characterized in that The first model includes a depthwise separable convolutional layer, which consists of two layers of structures: depthwise convolution and pointwise convolution.
8. The device according to claim 6 or 7, characterized in that The first model is obtained by pruning and fine-tuning the second model, and the second model is obtained based on a first training sample, wherein the first training sample includes: sample Y channel data and a sample second face image.
9. A model training device, characterized in that: The device comprises: an acquisition unit and a processing unit; The acquisition unit is used to acquire a second model, where the second model is obtained based on a first training sample, where the first training sample includes: sample Y channel data and a sample second face image; The processing unit is used to perform pruning on the second model to obtain the first model; The acquisition unit is further used to acquire Y channel data, where the Y channel data is obtained by format conversion of the first face image, where the first face image is an image in RGB format; The processing unit is further used for: Inputting the Y channel data into the first model to obtain a second processing result; Inputting the Y channel data into the second model to obtain a third processing result; The first model is trained according to the second processing result, the third processing result and the mean absolute error loss function.
10. The device according to claim 9, characterized in that The processing unit is also used to use a convolution kernel to transform the number of intermediate layer feature channels of the first model to be the same as the number of intermediate layer feature channels of the second model.
11. A processing device, characterized in that: The method comprises a processor and a memory, wherein the processor is coupled to the memory, the memory is used to store computer programs or instructions, and the processor is used to execute the computer program or instructions in the memory, so that the method according to any one of claims 1 to 3 is executed.
12. A model training device, characterized in that: The method comprises a processor and a memory, wherein the processor is coupled to the memory, the memory is used to store computer programs or instructions, and the processor is used to execute the computer program or instructions in the memory, so that the method of claim 4 or 5 is executed.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code, and when the computer program code is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.
