Training method of repair model, video repair method, device and medium

Abstract

This application provides a method for training a restoration model, a video restoration method, an apparatus, and a medium. The method for training the restoration model includes: acquiring sample face images and sample non-face images, and pasting the sample face images onto the sample non-face images to obtain a target sample image and an image mask corresponding to the target sample image. The image mask is used to characterize the region in the target sample image corresponding to the sample face image; degrading the target sample image to obtain a degraded sample image; and using the degraded sample image, training a restoration model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for training a repair model, a video repair method, a device, and a medium. Background Technology

[0002] With the development of the multimedia industry, more and more users are watching videos, and their demands for video clarity and other qualities are also increasing. Therefore, general audio and video playback devices can enhance video quality during playback using restoration models.

[0003] However, existing video restoration solutions have relatively limited functionality. For example, some restoration models can remove compressed coding noise and imperfections from videos, while others can extract face images from video frames, restore faces separately, and then merge the restored faces back into the video frames.

[0004] However, existing restoration solutions require multiple models to work together if they need to remove blemishes and restore faces at the same time, resulting in high model deployment and inference costs during the restoration process. Summary of the Invention

[0005] In view of this, embodiments of this application provide a training method for a repair model and a video repair scheme to at least partially solve the above-mentioned problems.

[0006] According to a first aspect of the embodiments of this application, a method for training a repair model is provided, comprising:

[0007] Acquire sample face images and sample non-face images, and attach the sample face images to the sample non-face images to obtain a target sample image and an image mask corresponding to the target sample image. The image mask is used to characterize the region in the target sample image that corresponds to the sample face image.

[0008] The target sample image is degraded to obtain a degraded sample image;

[0009] Using the degraded sample image, a repair model is trained based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

[0010] Optionally, in any embodiment of this application, the step of degrading the target sample image to obtain a degraded sample image includes:

[0011] The target sample image is subjected to online and / or offline degradation processing to obtain the degraded sample image;

[0012] The online degradation processing includes: blurring the target sample image through an online process to obtain a degraded sample image;

[0013] The offline degradation processing includes: submitting the target sample image to an offline video compression encoding program, and obtaining the encoded image output by the offline video compression encoding program as the degraded sample image.

[0014] Optionally, in any embodiment of this application, the online process includes at least two degraded links, in which the output of the previous degraded link is used as the input of the next degraded link, and the output of the last degraded link is the degraded sample image.

[0015] Optionally, in any embodiment of this application, the degradation link includes a blurring module for blurring the input target sample image; a sampling module for upsampling and / or downsampling the blurred target sample image; and a compression module for encoding and compressing the sampled target sample image to obtain the degraded target sample image.

[0016] Optionally, in any embodiment of this application, the blurring processing performed on the input target sample image includes at least one of the following: Gaussian blur, motion blur.

[0017] Optionally, in any embodiment of this application, the size of the minimum feature map of the repair model is 1 / 32 of the size of the input image block, and the size of the input image block of the repair model is greater than or equal to 1 / 3 of the target sample image.

[0018] Optionally, in any embodiment of this application, the repair model includes a multi-order encoder and a decoder, with skip connections between encoders and decoders of the same order, wherein the order of the encoder and the decoder is greater than or equal to 5, so that the size of the minimum feature map of the repair model is 1 / 32 of the size of the input image patch.

[0019] According to a second aspect of the embodiments of this application, a video restoration method is provided, comprising:

[0020] Obtain the video frames to be repaired;

[0021] The video frames are repaired using a repair model, which is trained using the method described above.

[0022] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0023] The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to any of the methods described above.

[0024] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements any of the methods described above.

[0025] According to the training scheme of the restoration model provided in this application embodiment, sample face images and sample non-face images are acquired, and the sample face images are pasted onto the sample non-face images to obtain a target sample image and an image mask corresponding to the target sample image. The image mask can be used to characterize the region in the target sample image corresponding to the sample face image. The target sample image is degraded to obtain a degraded sample image. Using the degraded sample image, the restoration model is trained based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask. This allows the trained restoration model to not only improve video clarity but also perform adaptive restoration for portraits without the need for separate portrait restoration models and compression removal models, thereby achieving lower model deployment and inference costs. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0027] Figure 1 A schematic diagram of an exemplary system for training a repair model or a video repair method applicable to embodiments of this application;

[0028] Figure 2A This is a flowchart illustrating the steps of a training method for a repair model according to an embodiment of this application;

[0029] Figure 2B for Figure 2A A schematic diagram of a target sample image and its corresponding image mask in the illustrated embodiment;

[0030] Figure 3 This is a schematic diagram of a second-order degraded link according to an embodiment of this application;

[0031] Figure 4 This is a schematic diagram of the structure of a repair model provided in an embodiment of this application;

[0032] Figure 5 A flowchart illustrating a video restoration method provided in an embodiment of this application;

[0033] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0035] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0036] Figure 1 An exemplary system for training a repair model or a video repair method applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106. Figure 1 The example in the text shows multiple user devices.

[0037] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, cloud computing server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to store video. As an optional example, in some embodiments, the cloud server 102 can be used to train a repair model. As another example, in some embodiments, the cloud server 102 can be used to deploy a repair model.

[0038] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user equipment 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user equipment 106 and the cloud server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0039] User device 106 may include any one or more user devices suitable for presenting images or videos. In some embodiments, user device 106 may include any suitable type of device. For example, in some embodiments, user device 106 may include mobile devices, tablet computers, laptop computers, desktop computers, wearable computers, game consoles, media players, vehicle entertainment systems, and / or any other suitable type of user device.

[0040] Based on the above system, this application provides a method for training a repair model, which will be described below through several embodiments.

[0041] See Figure 2A The flowchart illustrates the steps of a training method for a repair model provided in an embodiment of this application, including:

[0042] S201. Obtain a sample face image and a sample non-face image, and attach the sample face image to the sample non-face image to obtain a target sample image and an image mask corresponding to the target sample image.

[0043] In this embodiment, the sample face image can be any image containing a face obtained in any way. For example, it can be face data including various poses obtained from an open source database, or it can be a face image collected by a face recognition model. All of these are within the protection scope of this application.

[0044] The sample non-face image can be an image that does not include a face, obtained in any way. Preferably, the sample non-face image is an image corresponding to a scene where a person may be present.

[0045] In this embodiment, a target sample image can be obtained by randomly superimposing a sample face image onto a sample non-face image. An image mask can be generated based on the position of the sample face image superimposed onto the sample non-face image. In this embodiment, the image mask is used to characterize the region in the target sample image corresponding to the sample face image.

[0046] See Figure 2B The left side shows the target sample image obtained by overlaying a sample face image onto a non-sample face image, and the right side shows the image mask corresponding to the target sample image. For example, the image mask can be a black and white binary image, where the area corresponding to the sample non-face image is black and the area corresponding to the sample face image is white.

[0047] S202. Degrade the target sample image to obtain a degraded sample image;

[0048] In this embodiment, any degradation processing scheme can be used to degrade the target sample image to obtain a degraded sample image.

[0049] Degradation processing: High-quality images are processed using methods such as blurring and compression to reduce image quality (e.g., reducing resolution, reducing image size, etc.) to obtain low-quality degraded sample images.

[0050] Optionally, in this embodiment, the target sample image can be subjected to online and / or offline degradation processing to obtain the degraded sample image.

[0051] The online degradation processing includes: performing degradation processing on the target sample image through an online process to obtain a degraded sample image.

[0052] The offline degradation processing includes: submitting the target sample image to an offline video compression encoding program, and obtaining the encoded image output by the offline video compression encoding program as the degraded sample image.

[0053] When both online and offline degradation processing are used simultaneously, the blurring and compression artifacts in real videos can be simulated as closely as possible. This allows the trained restoration model to better remove compression artifacts and improve video clarity. Online degradation processing can use fast compression schemes, such as JPEG, thus minimizing resource consumption and time. Offline degradation processing can use slower video compression encoding programs, such as H.264 and H.265, ensuring a wider variety of degraded sample images without significantly increasing the time and resources required for training.

[0054] Optionally, in this embodiment of the application, the online process includes at least two degraded links, in which the output of the previous degraded link is used as the input of the next degraded link, and the output of the last degraded link is the degraded sample image.

[0055] By using at least two degradation links, the degraded sample image obtained after degradation processing can be made closer to the blur level of the actual image of the restoration model, thereby improving the accuracy of the trained restoration model.

[0056] Specifically, in this embodiment, the degradation link includes a blurring module for blurring the input target sample image; a sampling module for upsampling and / or downsampling the blurred target sample image; and a compression module for encoding and compressing the sampled target sample image to obtain the degraded target sample image.

[0057] See Figure 3 The diagram illustrates a second-order degradation link structure, as shown below. Figure 3 As shown, each stage of the second-order degradation chain includes: a blurring module, a sampling module, and a compression module. Specifically, the compression module is a JPEG compression module.

[0058] The target sample image is input to the first-order degradation link. In the first-order degradation link, the blurring module blurs the input target sample image; the sampling module upsamples and / or downsamples the blurred target sample image; and the compression module encodes and compresses the sampled target sample image to obtain the degraded target sample image.

[0059] The target sample image, after being degraded by the first-order degradation link, is input into the second-order degradation link, where the above degradation process is repeated, and the final output is the degraded sample image.

[0060] It should be noted that this embodiment only uses a second-order degraded link as an example for illustration. Other solutions, such as third-order and above degraded links, are also within the scope of protection of this application.

[0061] Specifically, in this embodiment, in the degradation link, the blurring processing module performs blurring processing on the input target sample image, including at least one of the following: Gaussian blurring and motion blurring.

[0062] Gaussian blur is the convolution of an image with a probability density function of a Gaussian distribution. Gaussian blur can include isotropic Gaussian blur, anisotropic Gaussian blur, and generalized Gaussian blur, all of which are within the scope of protection of this application.

[0063] Motion blur, also known as dynamic blur, is a process that blurs or blurs fast-moving objects in an image.

[0064] By using Gaussian blur and motion blur, the blurred target sample image can be made closer to the actual video frame, thus improving the performance of the trained restoration model.

[0065] S203. Using the degraded sample image, train a repair model based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

[0066] In this embodiment, the global loss function is used to determine the loss value globally based on the degraded sample image, while the local loss function corresponding to the image mask is used to determine the loss value based on the region corresponding to the face in the degraded sample image (i.e., the region corresponding to the sample face image). The loss values ​​can then be summed, and the inpainting model can be adjusted based on the summed loss values ​​to train the inpainting model.

[0067] In this embodiment, the global loss function and the local loss function can be any loss function applicable to images, such as the cross-entropy loss function. The formulas for the global loss function and the local loss function can be the same or different, both of which are within the protection scope of this application.

[0068] In this embodiment, any training method can be used to train the repair model, such as supervised training, semi-supervised training, or unsupervised training, all of which are within the scope of protection of this application.

[0069] It should be noted that, in this embodiment of the application, in addition to the degraded sample images corresponding to image masking mentioned above, images that do not correspond to image masking can also be added as samples for training the repair model. In this case, the repair model can be trained based on the global loss function.

[0070] Optionally, in this embodiment, the size of the minimum feature map of the repair model is 1 / 32 of the size of the input image block, and the size of the input image block of the repair model is greater than or equal to 1 / 3 of the target sample image.

[0071] In real-world restoration scenarios, the resolution of the input video may be relatively large (e.g., 1920x1080), resulting in a larger scale of human figures. Therefore, to ensure consistency between training and testing, the input image patch size of the restoration model during training is greater than or equal to 1 / 3 of the target sample image. For example, the input image patch size can be set to 576. Simultaneously, to allow the network to have a larger receptive field to capture large-scale human figure semantics, the minimum feature map size of the restoration model is 1 / 32 of the input image patch size.

[0072] Specifically, the restoration model includes a multi-order encoder and a decoder, with skip connections between encoders and decoders of the same order, wherein the order of the encoder and the decoder is greater than or equal to 5, such that the size of the minimum feature map of the restoration model is 1 / 32 of the size of the input image patch.

[0073] See Figure 4 The diagram illustrates a structural schematic of a repair model provided in an embodiment of this application. The left side of the diagram includes a 5th-order encoder, and the right side includes a 5th-order decoder. There are skip connections between encoders and decoders of the same order. Figure 4 The image is shown by a dashed arrow pointing from the encoder to the decoder.

[0074] The image blocks corresponding to the 5th-order encoder are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the input image block, respectively. After the original degraded sample image is input into the restoration model, it undergoes 5 convolution and downsampling operations through the encoder. Each downsampling operation reduces the image block size to 1 / 2 of the original size. Then, it undergoes 5 convolution and upsampling operations through the decoder. Each upsampling operation doubles the image block size, so that the size of the restored image output by the decoder is the same as the size of the input image.

[0075] The solution provided in this embodiment acquires sample face images and sample non-face images, and then attaches the sample face images to the sample non-face images to obtain a target sample image and a corresponding image mask. The image mask can be used to characterize the region in the target sample image corresponding to the sample face image. The target sample image is then degraded to obtain a degraded sample image. Using the degraded sample image, a restoration model is trained based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask. This allows the trained restoration model to not only improve video clarity but also perform adaptive restoration for portraits, eliminating the need for separate portrait restoration and compression removal models, thereby achieving lower model deployment and inference costs.

[0076] The solution provided in this embodiment can be executed by any electronic device with data processing capabilities, including but not limited to servers, PCs, tablets, and mobile phones.

[0077] See Figure 5 The figure shows a flowchart illustrating a video restoration method provided in an embodiment of this application. As shown, it includes:

[0078] S501, Obtain the video frame to be repaired;

[0079] In this embodiment, the video frame can be a video frame obtained in any way, such as video frames in the video before and after editing, video frames before and after compression encoding, etc., all of which are within the protection scope of this application.

[0080] During video playback, the obtained video frames can be repaired first, and then the repaired video frames can be played to improve the quality of the video played to the user.

[0081] S502. The video frame is repaired using a repair model.

[0082] The repair model was trained using the method described above.

[0083] The video restoration method provided in this embodiment can be applied to various scenarios, such as: storage and streaming of regular videos or games. Specifically, the restoration model provided in this embodiment can be used to restore the video before encoding, and then encode it to form a corresponding video stream for storage and transmission in video streaming services or other similar applications. Another example is low-latency scenarios such as video conferencing and live video streaming. Specifically, video acquisition devices can be used to collect conference video data, which is then encoded and sent to the conference terminal. The restoration model provided in this embodiment can then be used to restore the video decoded by the conference terminal, thereby obtaining the conference video for playback. Yet another example is virtual reality scenarios. The collected video data can be encoded to form a corresponding video stream, which is then sent to virtual reality-related devices (such as VR glasses). The VR device can then use the restoration model provided in this embodiment to decode the video stream and restore the resulting video image, and implement corresponding VR functions based on the restored video image, and so on.

[0084] It should also be noted that the solution provided in this embodiment is particularly suitable for narrowband high-definition video playback scenarios. Generally speaking, video content undergoes multiple video editing, processing, and re-encoding operations from acquisition to terminal playback. Each processing / encoding operation affects the video quality, resulting in quality loss. Even with high-quality video acquisition equipment, the quality transmitted to the user may still be poor due to quality loss in intermediate stages.

[0085] Therefore, the final transcoding process usually includes video restoration processing to repair the image quality loss caused during transmission and improve video quality without increasing the bandwidth of the transmitted video.

[0086] The solution provided in this embodiment can be used in the above scenarios to improve video quality with lower model deployment and inference costs.

[0087] Reference Figure 6 The diagram shows a structural schematic of an electronic device according to Embodiment 5 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0088] like Figure 6 As shown, the electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0089] in:

[0090] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0091] Communication interface 604 is used for communication with other electronic devices or servers.

[0092] The processor 602 is used to execute program 610, which can specifically execute the training method of the above-mentioned repair model or the relevant steps in the video repair method embodiment.

[0093] Specifically, program 610 may include program code that includes computer operation instructions.

[0094] The processor 602 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0095] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0096] Program 610 may include multiple computer instructions. Specifically, program 610 may use multiple computer instructions to cause processor 602 to execute the operation corresponding to the training method of the repair model or the video repair method described in any of the foregoing multiple method embodiments.

[0097] Specifically, program 610 can be executed by processor 602 through multiple computer instructions: acquiring sample face images and sample non-face images, and attaching the sample face images to the sample non-face images to obtain a target sample image and an image mask corresponding to the target sample image. The image mask can be used to characterize the region in the target sample image corresponding to the sample face image.

[0098] The operation of degrading the target sample image to obtain a degraded sample image;

[0099] The operation of training a repair model using the degraded sample image based on a preset loss function, wherein the loss function includes a global loss function and a local loss function corresponding to the image mask.

[0100] Optionally, in any embodiment of this application, the step of degrading the target sample image to obtain a degraded sample image includes: performing online and / or offline degrading processing on the target sample image to obtain the degraded sample image;

[0101] The online degradation processing includes: blurring the target sample image through an online process to obtain a degraded sample image;

[0102] The offline degradation processing includes: submitting the target sample image to an offline video compression encoding program, and obtaining the encoded image output by the offline video compression encoding program as the degraded sample image.

[0103] Optionally, in any embodiment of this application, the online process includes at least two degraded links, in which the output of the previous degraded link is used as the input of the next degraded link, and the output of the last degraded link is the degraded sample image.

[0104] Optionally, in any embodiment of this application, the degradation link includes a blurring module for blurring the input target sample image; a sampling module for upsampling and / or downsampling the blurred target sample image; and a compression module for encoding and compressing the sampled target sample image to obtain the degraded target sample image.

[0105] Optionally, in any embodiment of this application, the blurring processing performed on the input target sample image includes at least one of the following: Gaussian blur, motion blur.

[0106] Optionally, in any embodiment of this application, the size of the minimum feature map of the repair model is 1 / 32 of the size of the input image block, and the size of the input image block of the repair model is greater than or equal to 1 / 3 of the target sample image.

[0107] Optionally, in any embodiment of this application, the repair model includes a multi-order encoder and a decoder, with skip connections between encoders and decoders of the same order, wherein the order of the encoder and the decoder is greater than or equal to 5, so that the size of the minimum feature map of the repair model is 1 / 32 of the size of the input image patch.

[0108] Specifically, program 610 can also cause processor 602 to execute the following operations through multiple computer instructions: acquiring the video frame to be repaired; and repairing the video frame using a repair model, wherein the repair model is trained using the method described above.

[0109] The specific implementation of each step in program 610 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0110] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0111] This application also provides a computer program product, including computer instructions that instruct a computing device to perform the operation corresponding to the training method of any of the repair models or the video repair method in the above-described multiple method embodiments.

[0112] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0113] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0114] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded over a network. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0115] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0116] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A training method of a inpainting model, comprising: obtaining a sample face image and a sample non-face image, and pasting the sample face image into the sample non-face image to obtain a target sample image and an image mask corresponding to the target sample image, the image mask being used to represent a region in the target sample image corresponding to the sample face image; performing degradation processing on the target sample image to obtain a degraded sample image, wherein the degradation processing comprises performing online degradation processing on the target sample image through an online process, the online process comprising at least two stages of degradation links, in adjacent two stages of the degradation links, an output of a previous stage of the degradation links being used as an input of a next stage of the degradation links, and an output of a last stage of the degradation links being the degraded sample image; training an inpainting model based on a preset loss function using the degraded sample image, wherein the loss function comprises a global loss function and a local loss function corresponding to the image mask.

2. The method of claim 1, wherein, The degradation processing on the target sample image to obtain the degraded sample image further comprises: performing offline degradation processing on the target sample image to obtain the degraded sample image. The online degradation processing comprises performing blurring processing on the target sample image through the online process to obtain the degraded sample image. The offline degradation processing comprises submitting the target sample image to an offline video compression encoding program, and obtaining an encoded image output by the offline video compression encoding program as the degraded sample image.

3. The method of claim 1, wherein, The degradation link comprises a blurring processing module configured to perform blurring processing on the input target sample image, a sampling module configured to perform up-sampling and / or down-sampling on the target sample image after the blurring processing, and a compression module configured to perform encoding compression on the target sample image after the sampling to obtain the degraded target sample image.

4. The method of claim 3, wherein, The blurring processing performed on the input target sample image comprises at least one of Gaussian blurring and motion blurring.

5. The method of claim 1, wherein, A minimum feature map of the inpainting model has a size of 1 / 32 of an input image block size, and the input image block size of the inpainting model is greater than or equal to 1 / 3 of the target sample image.

6. The method of claim 5, wherein, The inpainting model comprises a multi-stage encoder and a decoder, and the same-stage encoder and decoder are connected by a skip connection, wherein the number of stages of the encoder and the decoder is greater than or equal to 5, so that the minimum feature map of the inpainting model has a size of 1 / 32 of the input image block size.

7. A video inpainting method, comprising: obtaining a video frame to be inpainted; performing inpainting processing on the video frame by an inpainting model trained by the method of any one of claims 1-6.

8. An electronic device comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface being in communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method of any one of claims 1-6.

7. A video inpainting method, comprising: obtaining a video frame to be inpainted; performing inpainting processing on the video frame by an inpainting model trained by the method of any one of claims 1-6. a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface being in communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method of any one of claims 1-6.

9. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of claims 1-6.

Images