Image restoration method, device and electronic equipment

Abstract

The present invention provides an image restoration method, device and electronic device, which relates to the field of image restoration technology, including: obtaining an image to be restored and obtaining a brightness image corresponding to the image to be restored; wherein the image to be restored includes a face area; the brightness image is used to characterize the brightness channel information of the image to be restored; generating a face semantic feature corresponding to the image to be restored based on the brightness image through a feature generation model; extracting high-frequency detail features corresponding to the image to be restored through a feature extraction model; and obtaining a restoration result of the image to be restored based on the face semantic features and the high-frequency detail features. The present invention can better restore the image to be restored.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an image restoration method, device and electronic equipment. Background Art

[0002] In the process of video and image transmission, in order to save transmission bandwidth and reduce the bit rate, it is usually necessary to perform lossy coding compression on the video and image, which results in the video and image received by the receiving end often having large noise, thus affecting the user's visual experience. At the same time, the influence of human operation and shooting equipment will also cause problems such as blur and color distortion in the video and image. In order to improve the above problems, it is proposed in the prior art that images can be repaired and enhanced, such as repairing images based on generative models. However, although this repair method can generate corresponding ultra-clear images, for images containing human faces, the repair results have the problem that the expression of the portrait and other details cannot be kept consistent. At the same time, it is also easy to cause problems such as excessive beauty, unreal skin and facial features, etc., resulting in a heavy oil painting feeling in the repair results. In summary, the existing image repair methods cannot repair images containing human faces well. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image restoration method, device and electronic device, which can better restore the image to be restored.

[0004] In a first aspect, an embodiment of the present invention provides an image restoration method, the method comprising: obtaining an image to be restored and obtaining a brightness image corresponding to the image to be restored; wherein the image to be restored includes a face area; the brightness image is used to characterize brightness channel information of the image to be restored; generating facial semantic features corresponding to the image to be restored based on the brightness image through a feature generation model; extracting high-frequency detail features corresponding to the image to be restored through a feature extraction model; and obtaining a restoration result of the image to be restored based on the facial semantic features and the high-frequency detail features.

[0005] In one embodiment, the step of obtaining a brightness image corresponding to the image to be repaired includes: converting the image to be repaired from a current format to a YUV format; extracting brightness channel information from the image to be repaired in the YUV format, and obtaining a brightness image corresponding to the image to be repaired based on the brightness channel information.

[0006] In one embodiment, the step of generating facial semantic features corresponding to the image to be repaired based on the brightness image through a feature generation model includes: blurring the brightness image to obtain a blurred image; and segmenting the image to be repaired to obtain a facial image corresponding to the facial area in the image to be repaired; performing bilateral filtering on the blurred image and the facial image to obtain a smoothed filtered image; generating facial semantic features based on the smoothed filtered image through a feature generation model; wherein the feature generation model includes a first encoding module and a first decoding module.

[0007] In one embodiment, the step of segmenting the image to be repaired to obtain a facial image corresponding to the facial area in the image to be repaired includes: inputting the image to be repaired into a face segmentation model, segmenting the image to be repaired through the face segmentation model, and obtaining a facial image corresponding to the facial area in the image to be repaired.

[0008] In one embodiment, the feature extraction model includes a second encoding module, a second decoding module and a Softmax layer.

[0009] In one embodiment, the step of obtaining the restoration result of the image to be restored based on the facial semantic features and the high-frequency detail features includes: bitwise multiplying the facial semantic features and the high-frequency detail features to obtain a bitwise multiplication result; bitwise adding the bitwise multiplication result and the blurred image to obtain a bitwise addition result; and obtaining the restoration result of the image to be restored based on the image to be restored and the bitwise addition result.

[0010] In one embodiment, the step of obtaining a restoration result of the image to be restored based on the image to be restored and the bitwise addition result includes: obtaining a chromaticity image and a saturation image corresponding to the image to be restored; wherein the chromaticity image is used to characterize the chromaticity channel information of the image to be restored, and the saturation image is used to characterize the saturation channel information of the image to be restored; and fusing the chromaticity image, the saturation image and the bitwise addition result to obtain the restoration result of the image to be restored.

[0011] In the second aspect, an embodiment of the present invention also provides an expression recognition device, which includes: an image acquisition module, used to acquire an image to be repaired and a brightness image corresponding to the image to be repaired; wherein the image to be repaired includes a face area; the brightness image is used to characterize the brightness channel information of the image to be repaired; a semantic feature generation module, used to generate facial semantic features corresponding to the image to be repaired based on the brightness image through a feature generation model; a high-frequency feature extraction module, used to extract high-frequency detail features corresponding to the image to be repaired through a feature extraction model; and a repair module, used to obtain a repair result of the image to be repaired based on the facial semantic features and the high-frequency detail features.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory and a processor; the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of any one of the methods described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute any method described in the first aspect.

[0014] An image restoration method, device and electronic device provided by the embodiment of the present invention first obtain an image to be restored containing a face area and obtain a brightness image corresponding to the image to be restored, wherein the brightness image is used to characterize the brightness channel information of the image to be restored, then generate the face semantic features corresponding to the image to be restored based on the brightness image through a feature generation model, and extract the high-frequency detail features corresponding to the image to be restored through a feature extraction model, and then obtain the restoration result of the image to be restored based on the face semantic features and the high-frequency detail features. The above method can generate face semantic features based on the brightness image through the feature generation model on the one hand, and extract the high-frequency detail features of the image to be restored through the feature extraction model on the other hand, and obtain the restoration result of the image to be restored by combining the face semantic features and the high-frequency detail features. The embodiment of the present invention makes full use of the semantic features of the brightness image and the high-frequency detail features of the image to be restored, and can better restore the image to be restored, not only making the image to be restored consistent with the details in the restoration result, but also alleviating the problem of restoration result distortion in the prior art, and also obtaining a vivid and natural restoration result without oil painting feeling, thereby better maintaining the realism of the image to be restored.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a flow chart of an image restoration method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of a flow chart of another image restoration method provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the structure of an image restoration device provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] At present, in the process of image transmission, due to image compression, human operation and shooting equipment, the image may be blurred, distorted, and have serious noise. The usual solution is to assume that there is a problem that needs to be overcome based on expert experience, such as assuming the existence of Gaussian noise, motion blur, compression noise, etc., and then simulate and generate training data for the specific problem, so as to train the model based on the training data, and finally achieve image restoration through the trained model. However, the above method will have the following problems: (1) It is heavily dependent on expert experience, and it is difficult to generate simulated data with data distribution and real damaged images; (2) A single model cannot solve all problems existing in the image. The real damaged image contains different types of noise. The restoration model that only targets a single problem can only solve part of the image within a very small range; (3) Although the restoration method based on the generative model can generate the corresponding ultra-clear image, its restoration result is easily distorted; (4) The restoration method based on the generative model is prone to cause problems such as excessive beauty, unreal skin and facial features, etc., resulting in a restoration result with a heavy oil painting feel and poor sense of reality. Based on this, the embodiments of the present invention provide an image restoration method, device and electronic device, which can better restore the image to be restored. The embodiments of the present invention are described in detail below.

[0024] To facilitate understanding of this embodiment, firstly, an image restoration method disclosed in an embodiment of the present invention is described in detail. Figure 1 The flowchart of an image restoration method shown in FIG. 1 mainly includes the following steps S102 to S108:

[0025] Step S102, obtaining an image to be repaired and a brightness image corresponding to the image to be repaired. The image to be repaired includes a face area, and the brightness image is used to characterize the brightness channel information of the image to be repaired. In some embodiments, an image containing a face area can be collected by an electronic device with a shooting function (such as a camera or a smart phone, etc.), and the collected image can be used as the image to be repaired. An image to be repaired can also be selected locally as the image to be repaired, and an image to be repaired can also be downloaded from the network as the image to be repaired. In addition, in order to obtain a brightness image corresponding to the image to be repaired, the image to be repaired can be converted from the current format to a YUV (Luminance, Chrominance, Chroma) format, wherein the Y channel represents the brightness (Luminance) channel, the U channel represents the chrominance (Chrominance), and the V channel represents the saturation (Chroma), and then the Y channel information is extracted from the image to be repaired in the YUV format, so as to obtain a brightness image for characterizing the Y channel information of the image to be repaired.

[0026] Step S104, generating facial semantic features corresponding to the image to be restored based on the brightness image through the feature generation model. In one embodiment, the brightness image may be preprocessed, wherein the preprocessing may include blurring or bilateral filtering, and then the preprocessing result is input into the feature generation model to generate facial semantic features corresponding to the image to be restored through the feature generation model.

[0027] Step S106, extracting high-frequency detail features corresponding to the image to be repaired through the feature extraction model. The high-frequency detail features are used to characterize the detail features of the high-frequency region (such as the edge of a face) in the image to be repaired. In one embodiment, the image to be repaired is input into the feature extraction model, thereby obtaining the high-frequency detail features output by the feature extraction model for the image to be repaired.

[0028] Step S108, based on the facial semantic features and the high-frequency detail features, the restoration result of the image to be restored is obtained. In one embodiment, the facial semantic features and the high-frequency detail features can be fused, and optionally, the facial semantic features and the high-frequency detail features are bitwise multiplied, and then based on the bitwise multiplication result and the brightness image, chromaticity image, and saturation image corresponding to the image to be restored, the restoration result of the image to be restored is obtained, wherein the chromaticity image is used to represent the chromaticity channel information of the image to be restored, and the saturation image is used to represent the saturation channel information of the image to be restored.

[0029] The above-mentioned image restoration method provided by the embodiment of the present invention can, on the one hand, generate facial semantic features based on the brightness image through a feature generation model, and on the other hand, extract high-frequency detail features of the image to be restored through a feature extraction model, and obtain the restoration result of the image to be restored by combining the facial semantic features and the high-frequency detail features. The embodiment of the present invention makes full use of the semantic features of the brightness image and the high-frequency detail features of the image to be restored, and can better restore the image to be restored. It can not only make the details of the image to be restored and its restoration result consistent, alleviate the distortion problem of the restoration result in the prior art, but also obtain a vivid and natural restoration result without oil painting feeling, thereby better maintaining the realism of the image to be restored.

[0030] For the aforementioned step S102, the embodiment of the present invention provides an implementation method for obtaining a brightness image corresponding to the image to be repaired, and the image to be repaired can be converted from the current format to the YUV format, and then the brightness channel information is extracted from the image to be repaired in the YUV format, and the brightness image corresponding to the image to be repaired is obtained based on the brightness channel information. Among them, the current format can be RGB (Red-Green-Blue) format or HSV (Hue, Saturation, Value) format, etc., and the embodiment of the present invention does not limit the current format of the image to be repaired. Assuming that the current format of the image to be repaired is RGB format, the image to be repaired can be converted from RGB format to YUV format by the following formula: Y = 0.299R + 0.587G + 0.114B; U = -0.1687R-0.3313G + 0.5B + ​​128; V = 0.5R-0.4187G-0.0813B + 128. After obtaining the image to be repaired in YUV format, the brightness image can be obtained by extracting the Y channel information.

[0031] In one implementation, the feature generation model may include a first encoding module (Encoder) and a first decoding module (Decoder). Based on the feature generation model, an embodiment of the present invention provides an implementation for generating facial semantic features corresponding to an image to be restored based on a brightness image through a feature generation model. For details, see the following steps 1 to 3:

[0032] Step 1, blurring the brightness image to obtain a blurred image; and segmenting the image to be repaired to obtain a facial image corresponding to the facial area in the image to be repaired. In one embodiment, the blurring process may be performed by Gaussian blurring, and the corresponding blurred image may be generated by performing Gaussian blurring on the brightness image. Optionally, when the image to be repaired is segmented, the image to be repaired may be input into a face segmentation model, and the image to be repaired may be segmented by the face segmentation model to obtain a facial image corresponding to the facial area in the image to be repaired, wherein the face segmentation model may use a neural network model such as a convolutional neural network that can be used to implement segmentation processing, and a mask of the facial area in the image to be repaired may be obtained by the face segmentation model, for example, the mask value of the facial area in the image to be repaired is 1, and the mask value of the non-face area is 0, thereby obtaining a facial image.

[0033] Step 2: bilaterally filter the blurred image and the face image to obtain a smoothed filtered image. In one embodiment, the face image and the blurred image can be used to perform bilateral filtering specifically on the face region to further blur and smooth the face region to obtain a smoothed filtered image.

[0034] Step 3, generate facial semantic features based on the smoothing filter image through the feature generation model. The feature generation model can use a fully convolutional neural network, which can include an encoder and a decoder. In an optional implementation, a simple convolution operation can be performed on the smoothing filter image, and then it is sent to the feature generation model. The embodiment of the present invention uses a combination of encoder-decoder to process the smoothing filter image, which can not only keep the input and output of the feature generation model consistent in size, but also perform super-resolution output, thereby maintaining the semantic consistency of the input and output of the feature generation model, and ensuring the consistency of the input and output in facial expressions and skin details.

[0035] To facilitate understanding of the aforementioned step S106, an embodiment of the present invention provides a feature extraction model. Optionally, the feature extraction model may adopt a fully convolutional neural network, and the fully convolutional neural network may include a second encoding module (Encoder), a second decoding module (Decoder) and a Softmax layer. When executing the above step S106, a simple convolution operation may be performed on the image to be repaired, and then sent to the feature extraction model, wherein the second encoding module has the same function as the aforementioned first encoding module, the second decoding module has the same function as the aforementioned first decoding module, and the function of the Softmax layer is to output the value range of each channel in [0,1] floating point values, thereby obtaining high-frequency detail features corresponding to the image to be repaired.

[0036] In order to make full use of the facial semantic features and high-frequency detail features of the image to be repaired, the embodiment of the present invention can fuse the facial semantic features with the high-frequency detail features. In one implementation manner, the steps of obtaining the repair result of the image to be repaired based on the facial semantic features and the high-frequency detail features can be performed according to the method shown in the following steps a to c:

[0037] Step a, bitwise multiplying the facial semantic features and the high-frequency detail features to obtain a bitwise multiplication result. For example, the facial semantic features are recorded as style_result, the high-frequency detail features are recorded as weight_result, and the style_result and weight_result are multiplied to obtain a bitwise multiplication result, wherein the bitwise multiplication result is recorded as output_Y_addition.

[0038] Step b, bitwise addition of the bitwise multiplication result and the blurred image to obtain a bitwise addition result. For example, the blurred image is recorded as input_Y_base, and the bitwise addition result can be obtained by adding output_Y_addition to input_Y_base, wherein the bitwise addition result is recorded as output_Y.

[0039] Step c, based on the image to be repaired and the bitwise addition result, obtain the repair result of the image to be repaired. In one embodiment, the chromaticity image and saturation image corresponding to the image to be repaired can be obtained, and then the chromaticity image, the saturation image and the bitwise addition result are fused to obtain the repair result of the image to be repaired. Among them, the chromaticity image is used to characterize the chromaticity channel information of the image to be repaired, and the saturation image is used to characterize the saturation channel information of the image to be repaired. In an optional embodiment, after converting the image to be repaired from the current format to the YUV format, the chromaticity channel information and the saturation channel information in the image to be repaired in the YUV format can be extracted respectively, so as to obtain the corresponding chromaticity image and saturation image, and then the chromaticity image, the saturation image and the bitwise addition result are fused. Optionally, the fusion operation can select the concat fusion operation to obtain the repair result in the YUV format. In practical applications, the repair result can also be converted from YUV format to the above-mentioned current format. Taking the current format as RGB format as an example, the repair result can be converted from YUV format to RGB format according to the following formula: R=Y+1.402(V-128); G=Y-0.34414(U-128)-0.71414(V-128); B=Y+1.772(U-128), thereby obtaining the repair result in RGB format.

[0040] To facilitate understanding of the image restoration method provided in the above embodiment, the present invention provides an application example of the image restoration method, see Figure 2 A flowchart of another image restoration method is shown in FIG. Figure 2 As shown:

[0041] 1) Inputting the image to be restored Input_RGB. In one implementation, the image to be restored may be in RGB format.

[0042] 2) Convert the input Input_RGB to a picture in the YUV color space, namely Input_YUV, and the channels are Input_Y / Input_U / Input_V. Among them, Input_Y is the aforementioned brightness image, Input_U is the aforementioned chroma image, and Input_V is the aforementioned saturation image.

[0043] 3) Input_Y is Gaussian blurred to generate a blurred image Input_Y_base.

[0044] 4) Input_RGB is passed through the face segmentation model to obtain the mask of the face part, which is the face image face_mask.

[0045] 5) Use face_mask and Input_Y_base to perform bilateral filtering on the face part, further blur and smooth the face part, and obtain the smoothed filtered image Input_Y_filter.

[0046] 6) Input_Y_filter undergoes a simple convolution operation Cov and is sent to a fully convolutional neural network of the feature generation model (also called Style Network), which includes Encoder and Decoder. The output of stylenetwork is recorded as the face semantic feature style_result;

[0047] 7) Input_RGB undergoes a simple convolution operation Cov and is sent to a fully convolutional neural network of the feature extraction model (also called WeightNetwork). The neural network includes Encoder, Decoder and a Softmax operation. The purpose of the Softmax operation is to output a floating-point value in the range of [0,1] for each channel. The result is recorded as the high-frequency detail feature weight_result.

[0048] 8) Multiply style_result and weigt_result to obtain the bitwise multiplication result output_Y_addition.

[0049] 9) output_Y_addition is added to input_Y_base to obtain the bitwise addition result output_Y.

[0050] 10) Perform concat fusion operation on output_Y, Input_U and Input_V to obtain the repair result Output_YUV in YUV format;

[0051] 11) The YUV color space of Output_YUV is converted to the RGB color space of Output_RGB to obtain the repair result Output_RGB in RGB format.

[0052] 12) Output Output_RGB.

[0053] To sum up, the image restoration method provided by the embodiment of the present invention not only does not need to rely on expert experience to simulate training data, but also has strong generalization, and can better remove block effect noise and ringing effect noise, etc.; in addition, the image restoration method provided by the embodiment of the present invention can effectively restore severely damaged facial images, specifically, can generate high-definition facial features and hair, and the output restoration result is consistent with the input image to be restored in expression, skin, etc., and generates natural and vivid high-definition faces, thereby effectively improving the visual experience.

[0054] Based on the image restoration method provided in the above embodiment, the embodiment of the present invention provides an image restoration device. Figure 3 The schematic diagram of the structure of an image restoration device shown in FIG. 1 includes at least the following parts:

[0055] The image acquisition module 302 is used to acquire the image to be restored and the brightness image corresponding to the image to be restored, wherein the image to be restored includes a face area; the brightness image is used to represent the brightness channel information of the image to be restored.

[0056] The semantic feature generation module 304 is used to generate the facial semantic features corresponding to the image to be restored based on the brightness image by using a feature generation model.

[0057] The high-frequency feature extraction module 306 is used to extract high-frequency detail features corresponding to the image to be restored through a feature extraction model.

[0058] The restoration module 308 is used to obtain a restoration result of the image to be restored based on the semantic features of the face and the high-frequency detail features.

[0059] The above-mentioned image restoration device provided by the embodiment of the present invention can, on the one hand, generate facial semantic features based on the brightness image through a feature generation model, and on the other hand, extract high-frequency detail features of the image to be restored through a feature extraction model, and obtain the restoration result of the image to be restored by combining the facial semantic features and the high-frequency detail features. The embodiment of the present invention makes full use of the semantic features of the brightness image and the high-frequency detail features of the image to be restored, and can better restore the image to be restored. It can not only make the image to be restored consistent with the details in the restoration result and alleviate the distortion problem of the restoration result in the prior art, but also obtain a vivid and natural restoration result without oil painting feeling, thereby better maintaining the realism of the image to be restored.

[0060] In one embodiment, the image acquisition module 302 is further used to: convert the image to be repaired from the current format to the YUV format; extract brightness channel information from the image to be repaired in the YUV format, and obtain a brightness image corresponding to the image to be repaired based on the brightness channel information.

[0061] In one embodiment, the semantic feature generation module 304 is also used to: blur the brightness image to obtain a blurred image; and segment the image to be repaired to obtain a facial image corresponding to the facial area in the image to be repaired; perform bilateral filtering on the blurred image and the facial image to obtain a smoothed filtered image; generate facial semantic features based on the smoothed filtered image through a feature generation model; wherein the feature generation model includes a first encoding module and a first decoding module.

[0062] In one embodiment, the semantic feature generation module 304 is further used to: input the image to be repaired into a face segmentation model, segment the image to be repaired using the face segmentation model, and obtain a face image corresponding to the face area in the image to be repaired.

[0063] In one embodiment, the feature extraction model includes a second encoding module, a second decoding module and a Softmax layer.

[0064] In one embodiment, the repair module 308 is also used to: bit-wise multiply the facial semantic features and the high-frequency detail features to obtain a bit-wise multiplication result; bit-wise add the bit-wise multiplication result and the blurred image to obtain a bit-wise addition result; and obtain a repair result of the image to be repaired based on the image to be repaired and the bit-wise addition result.

[0065] In one embodiment, the restoration module 308 is also used to: obtain a chromaticity image and a saturation image corresponding to the image to be restored; wherein the chromaticity image is used to represent the chromaticity channel information of the image to be restored, and the saturation image is used to represent the saturation channel information of the image to be restored; and fuse the chromaticity image, the saturation image and the bitwise addition result to obtain a restoration result of the image to be restored.

[0066] The implementation principle and technical effects of the device provided in this embodiment are the same as those of the aforementioned embodiments. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0067] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0068] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0069] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0070] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0071] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0072] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0073] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0074] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0075] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image restoration method, characterized in that: include: Acquire an image to be repaired and acquire a brightness image corresponding to the image to be repaired; wherein the image to be repaired includes a face area; and the brightness image is used to represent brightness channel information of the image to be repaired; Generate a facial semantic feature corresponding to the image to be restored based on the brightness image by using a feature generation model; Extracting high-frequency detail features corresponding to the image to be repaired by a feature extraction model; Based on the facial semantic features and the high-frequency detail features, a restoration result of the image to be restored is obtained.

2. The method according to claim 1, characterized in that The step of obtaining a brightness image corresponding to the image to be repaired comprises: Convert the image to be repaired from the current format to the YUV format; Luminance channel information is extracted from the image to be repaired in YUV format, and a luminance image corresponding to the image to be repaired is obtained based on the luminance channel information.

3. The method according to claim 1, characterized in that The step of generating the facial semantic features corresponding to the image to be restored based on the brightness image by using the feature generation model comprises: Performing blur processing on the brightness image to obtain a blurred image; and performing segmentation processing on the image to be repaired to obtain a face image corresponding to the face area in the image to be repaired; Performing bilateral filtering on the blurred image and the face image to obtain a smoothed filtered image; Generate facial semantic features based on the smoothing filter image through a feature generation model; wherein the feature generation model includes a first encoding module and a first decoding module.

4. The method according to claim 3, characterized in that The step of segmenting the image to be repaired to obtain a face image corresponding to the face area in the image to be repaired includes: The image to be repaired is input into a face segmentation model, and the image to be repaired is segmented by the face segmentation model to obtain a face image corresponding to the face area in the image to be repaired.

5. The method according to claim 1, characterized in that The feature extraction model includes a second encoding module, a second decoding module and a Softmax layer.

6. The method according to claim 1, characterized in that The step of obtaining the restoration result of the image to be restored based on the facial semantic features and the high-frequency detail features comprises: Multiplying the facial semantic feature and the high-frequency detail feature bit by bit to obtain a bitwise multiplication result; Performing bitwise addition on the bitwise multiplication result and the blurred image to obtain a bitwise addition result; A restoration result of the image to be restored is obtained based on the image to be restored and the bitwise addition result.

7. The method according to claim 6, characterized in that The step of obtaining a restoration result of the image to be restored based on the image to be restored and the bitwise addition result comprises: Acquire a chromaticity image and a saturation image corresponding to the image to be repaired; wherein the chromaticity image is used to represent the chromaticity channel information of the image to be repaired, and the saturation image is used to represent the saturation channel information of the image to be repaired; The chromaticity image, the saturation image and the bitwise addition result are fused to obtain a restoration result of the image to be restored.

8. An image restoration device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be restored and a brightness image corresponding to the image to be restored; wherein the image to be restored includes a face area; and the brightness image is used to represent brightness channel information of the image to be restored; A semantic feature generation module, used for generating a facial semantic feature corresponding to the image to be restored based on the brightness image through a feature generation model; A high-frequency feature extraction module, used to extract high-frequency detail features corresponding to the image to be repaired through a feature extraction model; The restoration module is used to obtain a restoration result of the image to be restored based on the facial semantic features and the high-frequency detail features.

9. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores a computer program that can be run on the processor, and the processor implements the steps of the method described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to execute the method according to any one of claims 1 to 7.

Images