Repair model training method, video repair method, device, and medium

By training a repair model based on global and local loss functions, the problem of high video repair costs in the existing technology is solved, video clarity improvement and face repair are achieved, and model deployment and inference costs are reduced.

WO2024244842A1PCT designated stage expired Publication Date: 2024-12-05ALIBABA CLOUD COMPUTING CO LTD

Patent Information

Application Number
PCT/CN2024/089864
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-04-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

When the existing video repair solution removes defects and performs face repair at the same time, the model deployment and inference cost are relatively high, and the functions are relatively single, which cannot meet the high-definition video needs.

Method used

By acquiring sample face images and non-face images, generating target sample images and their image masks, performing degradation processing, and training repair models based on global and local loss functions to achieve video clarity improvement and face repair.

Benefits of technology

It realizes improved video clarity and adaptive repairs for portraits at lower model deployment and inference costs, reducing the need for individual portrait repair models and compression removal models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024089864_05122024_PF_FP_ABST
    Figure CN2024089864_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided are a repair model training method, a video repair method, a device, and a medium. The repair model training method comprises: obtaining a sample face image and a sample non-face image, and attaching the sample face image to the sample non-face image to obtain a target sample image and an image mask corresponding to the target sample image (S201), the image mask being used for representing an area in the target sample image corresponding to the sample face image; degrading the target sample image to obtain a degraded sample image (S202); and using the degraded sample image to train a repair model on the basis of a preset loss function, wherein the loss function comprises a global loss function, and a local loss function which corresponds to the image mask (S203).
Need to check novelty before this filing date? Find Prior Art

Description

Restoration model training method, video restoration method, equipment and medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 26, 2023, with application number 2023106121136 and invention name “Training method of restoration model, video restoration method, device and medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of computer technology, and in particular to a training method for a restoration model, a video restoration method, a device, and a medium. Background Art

[0003] With the development of the multimedia industry, more and more users are watching videos, and their requirements for video clarity are getting higher and higher. To this end, general audio and video playback devices can enhance the video quality through restoration models when playing videos.

[0004] However, the functions of the video restoration solutions in related technologies are relatively simple. For example, some restoration models can remove compression coding noise and defects in the video, while other restoration models can extract facial images from video frames and repair the faces separately, and then merge the repaired faces into the video frames.

[0005] However, if the restoration solutions in related technologies need to remove blemishes and perform face restoration at the same time, multiple models need to work together, and the model deployment and inference costs of the restoration process are high.

[0006] Summary of the Invention

[0007] In view of this, an embodiment of the present application provides a training method for a restoration model and a video restoration solution to at least partially solve the above-mentioned problems.

[0008] According to a first aspect of an embodiment of the present application, a training method for a restoration model is provided, comprising: obtaining a sample facial image and a sample non-facial image, and fitting the sample facial image to the sample non-facial image to obtain a target sample image and an image mask corresponding to the target sample image, wherein the image mask is used to characterize the area in the target sample image corresponding to the sample facial image; degrading the target sample image to obtain a degraded sample image; and using the degraded sample image to train a restoration model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

[0009] According to a second aspect of an embodiment of the present application, a video restoration method is provided, comprising: obtaining a video frame to be restored; and performing restoration processing on the video frame using a restoration model, wherein the restoration model is trained using the method described above.

[0010] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method described in any one of the above items.

[0011] According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, any of the methods described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0013] FIG1 is a schematic diagram of an exemplary system for a training method for a restoration model or a video restoration method applicable to an embodiment of the present application;

[0014] FIG2A is a flowchart of a method for training a repair model according to an embodiment of the present application;

[0015] FIG2B is a schematic diagram of a target sample image and its corresponding image mask in the embodiment shown in FIG2A ;

[0016] FIG3 is a schematic diagram of the structure of a second-order degradation link according to an embodiment of the present application;

[0017] FIG4 is a schematic structural diagram of a repair model provided in an embodiment of the present application;

[0018] FIG5 is a flow chart of a video restoration method according to an embodiment of the present application;

[0019] FIG6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0021] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0022] Figure 1 illustrates an exemplary system for a method for training a restoration model or a video restoration method applicable to embodiments of the present application. As shown in Figure 1 , the system 100 may include a cloud service 102 , a communication network 104 , and / or one or more user devices 106 , with multiple user devices being illustrated in Figure 1 .

[0023] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to a distributed storage system, a server cluster, a cloud computing server cluster, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to store videos. As an example, in some embodiments, the cloud server 102 can be used to train a repair model. As another example, in some embodiments, the cloud server 102 can be used to deploy a repair model.

[0024] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user device 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud service end 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user device 106 and the cloud service end 102, such as a network link, a dial-up link, a wireless link, a hard-wired link, any other suitable communication link, or any suitable combination of such links.

[0025] User device 106 may include any one or more user devices suitable for presenting images or videos. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include a mobile device, a tablet computer, a laptop computer, a desktop computer, a wearable computer, a game console, a media player, a vehicle entertainment system, and / or any other suitable type of user device.

[0026] Based on the above system, an embodiment of the present application provides a training method for a repair model, which is described below through multiple embodiments.

[0027] 2A , which shows a flowchart of the steps of a training method for a repair model provided in an embodiment of the present application, including:

[0028] S201, obtaining a sample face image and a sample non-face image, and fitting the sample face image to the sample non-face image to obtain a target sample image and an image mask corresponding to the target sample image;

[0029] In this embodiment, the sample facial image can be an image of a face obtained by any means, for example, facial data including various postures can be obtained from an open source database, or it can be a facial image collected by a face recognition model, all of which are within the scope of protection of this application.

[0030] The sample non-face image may be an image obtained in any manner and not including a face. In one embodiment, the sample non-face image may be an image corresponding to a scene in which a person may be present.

[0031] In this embodiment, the sample face image can be randomly superimposed on the sample non-face image to obtain the target sample image, and an image mask can be generated according to the position where the sample face image is superimposed on the sample non-face image. In this embodiment, the image mask is used to represent the area in the target sample image corresponding to the sample face image.

[0032] Referring to Figure 2B, the left side shows the target sample image obtained by superimposing the sample face image on the non-sample face image, and the right side shows the image mask corresponding to the target sample image. By way of example, the image mask can be a black and white binary image, in which the area corresponding to the sample non-face image is black, and the area corresponding to the sample face image is white.

[0033] S202, degrading the target sample image to obtain a degraded sample image;

[0034] In this embodiment, any degradation processing scheme may be used to degrade the target sample image to obtain a degraded sample image.

[0035] Degradation processing: Use blurring and compression to process high-quality images to reduce image quality (such as reducing resolution, reducing image size, etc.) to obtain low-quality degraded sample images.

[0036] In one embodiment, the target sample image may be subjected to online degradation processing and / or offline degradation processing to obtain the degraded sample image.

[0037] The online degradation processing includes: performing degradation processing on the target sample image through an online process to obtain a degraded sample image.

[0038] The offline degradation processing includes: submitting the target sample image to an offline video compression encoding program, and obtaining an encoded image output by the offline video compression encoding program as the degraded sample image.

[0039] When both online and offline degradation are used, blur and compression artifacts in real videos can be simulated as closely as possible, allowing the trained restoration model to better remove compression artifacts and improve video clarity. Online degradation can use faster compression schemes, such as JPEG, which reduces the resources and time consumed by the online degradation process. Offline degradation can use slower video compression codecs, such as H264 and H265, to meet the requirements of obtaining a wider range of degraded sample image types without significantly increasing the time and resources consumed by the training process.

[0040] In one embodiment, the online process includes at least two stages of degradation links. In two adjacent stages of the degradation links, the output of the degradation link of the previous stage serves as the input of the degradation link of the next stage, and the output of the degradation link of the last stage is the degraded sample image.

[0041] Through at least two-order degradation links, the blur degree of the degraded sample image obtained after degradation processing can be closer to the actual image of the restoration model, thereby improving the accuracy of the trained restoration model.

[0042] In one embodiment, the degradation link includes a blur processing module for blurring the input target sample image; a sampling module for upsampling and / or downsampling the blurred target sample image; and a compression module for encoding and compressing the sampled target sample image to obtain the degraded target sample image.

[0043] 3 shows a schematic diagram of the structure of a second-order degradation link. As shown in FIG3 , each degradation link in the second-order degradation link includes: a blur processing module, a sampling module, and a compression module. Specifically, the compression module is a JPEG compression module.

[0044] The target sample image is input to a first-order degradation link. In the first-order degradation link, a blur processing module performs blur processing on the input target sample image; a sampling module is used to upsample and / or downsample the blurred target sample image; and a compression module is used to encode and compress the sampled target sample image to obtain the degraded target sample image.

[0045] The target sample image degraded by the first-order degradation link is input to the second-order degradation link, and the above-mentioned degradation process is performed again, and the final output is a degraded sample image.

[0046] It should be noted that this embodiment only takes a second-order degraded link as an example for illustration, and other solutions, such as third-order and above degraded links, are also within the protection scope of this application.

[0047] In one embodiment, in the degradation link, the blur processing performed by the blur processing module on the input target sample image includes at least one of the following: Gaussian blur and motion blur.

[0048] Gaussian blur is the convolution of an image with a Gaussian-distributed probability density function. Gaussian blur can include isotropic Gaussian blur, anisotropic Gaussian blur, and generalized Gaussian blur, all of which are within the scope of this application.

[0049] Motion blur, also known as dynamic blur, is the process of processing an image so that fast-moving objects in it have obvious blurred dragging traces.

[0050] Through Gaussian blur and motion blur, the blurred target sample image can be made closer to the actual video frame, so that the trained restoration model can achieve better results.

[0051] S203 . Using the degraded sample image, train a restoration model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

[0052] In this embodiment, a global loss function is used to globally determine a loss value based on the degraded sample image, and a local loss function corresponding to the image mask is used to determine a loss value based on the region corresponding to the face in the degraded sample image (i.e., the region corresponding to the sample face image). The loss values ​​can then be summed and the restoration model can be adjusted based on the summed loss value to train the restoration model.

[0053] In this embodiment, the global loss function and the local loss function can be any loss function applicable to the image, such as the cross entropy loss function, etc. The formulas of the global loss function and the local loss function can be the same or different, and are all within the scope of protection of this application.

[0054] In this embodiment, any training method can be used to train the repair model, such as supervised training, semi-supervised training or unsupervised training, all of which are within the scope of protection of this application.

[0055] It should be noted that in the embodiment of the present application, in addition to the above-mentioned degraded sample images corresponding to the image mask, images that do not correspond to the image mask can also be added as samples for training the restoration model. In this case, the restoration model can be trained based on the global loss function.

[0056] In one embodiment, the size of the minimum feature map of the restoration model is 1 / 32 of the input image block size, and the input image block size of the restoration model is greater than or equal to 1 / 3 of the target sample image.

[0057] In actual restoration scenarios, the resolution of the input video may be large (such as 1920x1080), so the scale of the portraits contained therein will also be relatively large. Therefore, to ensure consistency between training and testing, during training, the input image patch size of the restoration model is greater than or equal to 1 / 3 of the target sample image. For example, the set input image patch size can be 576. At the same time, in order to give the network a larger receptive field to capture the semantics of large-scale portraits, the size of the minimum feature map of the restoration model is 1 / 32 of the input image patch size.

[0058] In one embodiment, the restoration model includes: a multi-order encoder and a decoder, and jump connections are formed between the encoders and decoders of the same order, wherein the order of the encoder and the decoder is greater than or equal to 5, so that the size of the minimum feature map of the restoration model is 1 / 32 of the input image block size.

[0059] Referring to Figure 4, a structural diagram of a repair model provided by an embodiment of the present application is shown. The left side of the figure includes a 5-order encoder encoder, and the right side includes a 5-order decoder decoder. The encoder and decoder of the same order are jump-connected, as shown in Figure 4 by the dotted arrow pointing from the encoder to the decoder.

[0060] The image blocks corresponding to the 5-order encoder are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image blocks, respectively. After the original degraded sample image is input into the restoration model, the encoder performs 5 convolution and downsampling operations. Each downsampling operation reduces the image block size to 1 / 2 of the original size. After that, the decoder performs 5 convolution and upsampling operations. Each upsampling operation reduces the image block size to twice the original size, so that the size of the restored image output by the decoder is the same as the input image size.

[0061] The solution provided in this embodiment obtains a sample face image and a sample non-face image, and fits the sample face image to the sample non-face image to obtain a target sample image and an image mask corresponding to the target sample image, wherein the image mask can be used to characterize the area corresponding to the sample face image in the target sample image; degrades the target sample image to obtain a degraded sample image; uses the degraded sample image to train a restoration model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask, so that the trained restoration model can not only improve the video clarity, but also perform adaptive restoration for portraits, without the need for a separate portrait restoration model and a compression removal model, thereby achieving lower model deployment and inference costs.

[0062] The solution provided in this embodiment can be executed by any electronic device with data processing capabilities, including but not limited to servers, PCs, tablets, mobile phones, etc.

[0063] 5 , which shows a flow chart of a video restoration method provided by an embodiment of the present application, as shown in the figure, includes:

[0064] S501, obtaining a video frame to be repaired;

[0065] In this embodiment, the video frames can be video frames obtained by any means, for example, video frames in the video before and after clipping, video frames before and after compression encoding, etc., all of which are within the protection scope of this application.

[0066] During video playback, the obtained video frames may be first repaired as video frames to be repaired, and then the repaired video frames may be played, so as to improve the quality of the video played to the user.

[0067] S502: Perform restoration processing on the video frame using a restoration model.

[0068] The repair model is trained using the method described above.

[0069] The video repair method provided in this embodiment can be applied to a variety of different scenarios, such as: storage and streaming of conventional videos or games. In one embodiment, the video can be repaired before encoding by the repair model provided in the embodiment of this application, and then encoded to form a corresponding video stream for storage and transmission in video streaming services or other similar applications; another example: low-latency scenarios such as video conferencing and live video. In one embodiment, the conference video data can be collected by a video acquisition device, and sent to the conference terminal after encoding, and then the video decoded by the conference terminal can be repaired by the repair model provided in the embodiment of this application, so as to obtain the played conference video picture; another example: virtual reality scene, the collected video data can be encoded to form a corresponding video stream, and sent to virtual reality related equipment (such as VR virtual glasses, etc.), and the video picture obtained by decoding the video stream through the VR device through the repair model provided in the embodiment of this application can be repaired, and the corresponding VR function can be implemented based on the repaired video picture, and so on.

[0070] It should also be noted that the solution provided in this embodiment is particularly suitable for narrowband, high-definition video playback scenarios. Generally speaking, from video capture to terminal playback, video content undergoes multiple video editing, processing, and re-encoding operations. Each processing / encoding operation affects the video quality, resulting in image quality loss. Even with high-quality video capture equipment, the quality of the final transmission to the user may be poor due to image quality loss during intermediate links.

[0071] To this end, the final transcoding process generally includes video repair processing to repair the image quality loss caused by the transmission process and improve the video quality without increasing the bandwidth of the transmitted video.

[0072] The solution provided in this embodiment can be used in the above scenarios to improve video quality with lower model deployment and inference costs.

[0073] According to the training scheme of the restoration model provided in the embodiment of the present application, by obtaining a sample face image and a sample non-face image, and fitting the sample face image to the sample non-face image, a target sample image and an image mask corresponding to the target sample image are obtained, and the image mask can be used to characterize the area corresponding to the sample face image in the target sample image; the target sample image is degraded to obtain a degraded sample image; the degraded sample image is used to train the restoration model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask, so that the trained restoration model can not only improve the video clarity, but also perform adaptive restoration for portraits, without the need for a separate portrait restoration model and a compression removal model, thereby achieving lower model deployment and inference costs.

[0074] 6 , a schematic structural diagram of an electronic device according to a fifth embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0075] As shown in FIG6 , the electronic device may include a processor 602 , a communications interface 604 , a memory 606 , and a communication bus 608 .

[0076] in:

[0077] The processor 602 , the communication interface 604 , and the memory 606 communicate with each other via a communication bus 608 .

[0078] The communication interface 604 is used to communicate with other electronic devices or servers.

[0079] The processor 602 is used to execute the program 610, and specifically can execute the training method of the above-mentioned restoration model or the relevant steps in the embodiment of the video restoration method.

[0080] In one embodiment, the program 610 may include program code, which includes computer operating instructions.

[0081] Processor 602 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or may be processors of different types, such as one or more CPUs and one or more ASICs.

[0082] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0083] The program 610 may include multiple computer instructions. Specifically, the program 610 may enable the processor 602 to execute operations corresponding to the training method of the restoration model or the video restoration method described in any of the aforementioned method embodiments through the multiple computer instructions.

[0084] The program 610 may specifically enable the processor 602 to execute, through a plurality of computer instructions, the following operations: obtaining a sample facial image and a sample non-facial image, and fitting the sample facial image to the sample non-facial image to obtain a target sample image and an image mask corresponding to the target sample image, wherein the image mask may be used to represent an area in the target sample image corresponding to the sample facial image;

[0085] An operation of degrading the target sample image to obtain a degraded sample image;

[0086] The degraded sample image is used to train a restoration model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

[0087] In one embodiment, the degrading the target sample image to obtain the degraded sample image includes: performing online degradation processing and / or offline degradation processing on the target sample image to obtain the degraded sample image;

[0088] The online degradation processing includes: performing blurring processing on the target sample image through an online process to obtain a degraded sample image;

[0089] The offline degradation processing includes: submitting the target sample image to an offline video compression encoding program, and obtaining an encoded image output by the offline video compression encoding program as the degraded sample image.

[0090] In one embodiment, the online process includes at least two stages of degradation links. In two adjacent stages of the degradation links, the output of the degradation link of the previous stage serves as the input of the degradation link of the next stage, and the output of the degradation link of the last stage is the degraded sample image.

[0091] In one embodiment, the degradation link includes a blur processing module for blurring the input target sample image; a sampling module for upsampling and / or downsampling the blurred target sample image; and a compression module for encoding and compressing the sampled target sample image to obtain the degraded target sample image.

[0092] In one embodiment, the blurring process performed on the input target sample image includes at least one of the following: Gaussian blurring and motion blurring.

[0093] In one embodiment, the size of the minimum feature map of the restoration model is 1 / 32 of the input image block size, and the input image block size of the restoration model is greater than or equal to 1 / 3 of the target sample image.

[0094] In one embodiment, the restoration model includes a multi-order encoder and decoder, and the encoders and decoders of the same order are jump-connected, wherein the order of the encoder and the decoder is greater than or equal to 5, so that the size of the minimum feature map of the restoration model is 1 / 32 of the input image block size.

[0095] The program 610 can also specifically enable the processor 602 to execute, through multiple computer instructions: an operation of obtaining a video frame to be repaired; and an operation of repairing the video frame using a repair model, wherein the repair model is trained using the method described above.

[0096] The specific implementation of each step in program 610 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0097] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the aforementioned method embodiments. The computer storage medium includes, but is not limited to, a compact disc read-only memory (CD-ROM), random access memory (RAM), a floppy disk, a hard disk, or a magneto-optical disk.

[0098] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any repair model training method or video repair method in the above-mentioned multiple method embodiments.

[0099] In addition, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used to train the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0100] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0101] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA)). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., random access memory (RAM), read-only memory (ROM), flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown here.

[0102] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0103] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A training method for a repair model, comprising: Acquire a sample face image and a sample non-face image, and fit the sample face image to the sample non-face image to obtain a target sample image and an image mask corresponding to the target sample image, wherein the image mask is used to characterize an area in the target sample image corresponding to the sample face image; Degrading the target sample image to obtain a degraded sample image; Using the degraded sample image, a restoration model is trained based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

2. The method according to claim 1, wherein: The step of degrading the target sample image to obtain a degraded sample image includes: Performing online degradation processing and / or offline degradation processing on the target sample image to obtain the degraded sample image; The online quality degradation process comprises: performing a fuzzy process on the target sample image through an online process to obtain a degraded sample image; The offline degradation processing includes: submitting the target sample image to an offline video compression encoding program, and obtaining an encoded image output by the offline video compression encoding program as the degraded sample image.

3. The method according to claim 2, wherein: The online process includes at least two-order degradation links. In two adjacent degradation links, the output of the last-order degradation link serves as the input of the next-order degradation link, and the output of the last-order degradation link is the degraded sample image.

4. The method according to claim 3, wherein: The degradation link includes a blur processing module, which is used to blur the input target sample image; a sampling module, which is used to upsample and / or downsample the target sample image after blur processing; The compression module is used to encode and compress the sampled target sample image to obtain the degraded target sample image.

5. The method according to claim 4, wherein: The blur processing performed on the input target sample image includes at least one of the following: Gaussian blur and motion blur.

6. The method according to claim 1, wherein: The size of the minimum feature map of the restoration model is 1 / 32 of the input image block size, and the input image block size of the restoration model is greater than or equal to 1 / 3 of the target sample image.

7. The method according to claim 6, wherein: The restoration model includes a multi-order encoder and decoder, and the encoders and decoders of the same order are jump-connected, wherein the order of the encoder and the decoder is greater than or equal to 5, so that the size of the minimum feature map of the restoration model is 1 / 32 of the input image block size.

8. A video repair method, comprising: Get the video frame to be repaired; The video frame is repaired by a repair model, and the repair model is trained by the method according to any one of claims 1 to 7.

9. An electronic device, comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform an operation corresponding to the method according to any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Image restoration method based on double-discrimination deep convolution generative adversarial network

    CN110570366A

  • Portrait restoration model training method and device, and electronic equipment

    CN113139915A

  • Image restoration method and device and storage medium

    CN113284073A

  • Demasked face recovery method and device, computer equipment and medium

    CN115223012A

  • Face map watermark shielding restoration method based on deep learning

    CN116152092A

Cited By

  • Image inpainting method and apparatus, and image inpainting model training method

    WO2026144301A1