Model training method, image restoration method and related device

Abstract

The embodiment of the invention discloses a model training method, an image restoration method and a related device, and the method comprises the steps: obtaining a to-be-restored image which is a face image comprising a shielding object; performing feature extraction based on the to-be-restored image to obtain image features of the to-be-restored image; determining shelter indication information used for indicating a shelter in the to-be-repaired image according to the to-be-repaired image through a shelter detection model; performing feature extraction based on the shielding object indication information to obtain shielding object features of the to-be-restored image; according to the shielding object indication information, determining restoration strategy parameters of the image restoration model; and generating a repaired image corresponding to the to-be-repaired image according to the image features and the shelter features by adopting the repair strategy parameters through the image repair model. According to the method, the corresponding high-quality image can be reliably recovered for the low-quality image of which the human face is shielded.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a model training method, an image restoration method, and related devices, equipment, storage media, and program products. Background Art

[0002] Nowadays, many film and television works have various image quality problems (such as noise, blur, distortion, etc.) due to the long shooting time and aging storage media. These image quality problems will seriously affect the user's viewing experience of film and television works. In order to solve these image quality problems, digital restoration technology is currently usually used to restore the image screen in film and television works.

[0003] In film and television works, the human face is one of the most important elements, and it plays a vital role in promoting the plot and shaping the character's personality. Correspondingly, in digital restoration, face restoration is also a very important link. However, in many cases, the faces in film and television works may be blocked by occluders (such as masks, hats, beards, arms, etc.). For such images with blocked faces, the image restoration models in related technologies are often unable to effectively extract the features of the occluders, and thus it is difficult to reliably restore the corresponding high-quality images. Summary of the invention

[0004] The embodiments of the present application provide a model training method, an image restoration method and related devices, which can reliably restore a corresponding high-quality image for a low-quality image with an occluded face.

[0005] In view of this, the first aspect of the present application provides a model training method, the method comprising:

[0006] Acquire a training sample; the training sample includes a training image to be repaired and a corresponding training repaired image, wherein the training image to be repaired is a face image including an occluded object;

[0007] Performing feature extraction based on the training image to be repaired to obtain image features of the training image to be repaired;

[0008] Determining, by means of an occlusion detection model, occlusion indication information for indicating the occlusion in the training image to be repaired according to the training image to be repaired; performing feature extraction based on the occlusion indication information to obtain occlusion features of the training image to be repaired;

[0009] Determining, according to the occlusion indication information, a restoration strategy parameter of the image restoration model to be trained;

[0010] The image restoration model is trained using the restoration strategy parameters according to the image features, the occlusion features and the training restored image.

[0011] A second aspect of the present application provides an image restoration method, the method comprising:

[0012] Acquire an image to be restored; the image to be restored is a face image including an occluder;

[0013] Performing feature extraction based on the image to be repaired to obtain image features of the image to be repaired;

[0014] Determining, by means of an occlusion detection model, occlusion indication information for indicating the occlusion in the image to be repaired according to the image to be repaired; performing feature extraction based on the occlusion indication information to obtain occlusion features of the image to be repaired;

[0015] Determining a restoration strategy parameter of an image restoration model according to the occlusion indication information;

[0016] Through the image restoration model, the restoration strategy parameters are adopted to generate a restored image corresponding to the image to be restored according to the image features and the occlusion features; the image restoration model is trained by the above-mentioned model training method, and the image quality of the restored image is higher than the image quality of the image to be restored.

[0017] A third aspect of the present application provides a model training device, the device comprising:

[0018] A sample acquisition module, used to acquire training samples; the training samples include training images to be repaired and corresponding training images after repair, and the training images to be repaired are human face images including occluders;

[0019] A first feature extraction module, used for performing feature extraction based on the training image to be repaired, to obtain image features of the training image to be repaired;

[0020] A second feature extraction module is used to determine, through the occlusion detection model, occlusion indication information used to indicate the occlusion in the training image to be repaired according to the training image to be repaired; and perform feature extraction based on the occlusion indication information to obtain the occlusion feature of the training image to be repaired;

[0021] A first strategy determination module, used to determine the restoration strategy parameters of the image restoration model to be trained according to the occlusion indication information;

[0022] The model training module is used to adopt the restoration strategy parameters to train the image restoration model according to the image features, the occlusion features and the trained restoration image.

[0023] A fourth aspect of the present application provides an image restoration device, the device comprising:

[0024] An image acquisition module, used to acquire an image to be restored; the image to be restored is a face image including an occluder;

[0025] A third feature extraction module, used to extract features based on the image to be repaired to obtain image features of the image to be repaired;

[0026] A fourth feature extraction module is used to determine, based on the image to be repaired, occluder indication information for indicating the occluder in the image to be repaired by using an occluder detection model; and perform feature extraction based on the occluder indication information to obtain occluder features of the image to be repaired;

[0027] A second strategy determination module, used to determine the restoration strategy parameters of the image restoration model according to the occlusion indication information;

[0028] An image restoration module is used to generate a restored image corresponding to the image to be restored through the image restoration model and the restoration strategy parameters according to the image features and the occlusion features; the image restoration model is trained by the above-mentioned model training method, and the image quality of the restored image is higher than the image quality of the image to be restored.

[0029] A fifth aspect of the present application provides an electronic device, the electronic device comprising a processor and a memory;

[0030] The memory is used to store computer programs;

[0031] The processor is used to execute the model training method described in the first aspect or the image restoration method described in the second aspect according to the computer program.

[0032] The sixth aspect of the present application provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a computer device, it implements the model training method described in the first aspect above, or the image restoration method described in the second aspect above.

[0033] The seventh aspect of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the model training method described in the first aspect above, or the image restoration method described in the second aspect above.

[0034] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0035] The embodiment of the present application provides an image restoration method, which includes: obtaining an image to be restored, which is a face image including an occlusion. Then, based on the image to be restored, feature extraction is performed to obtain image features of the image to be restored, and the image features can reflect the image content (such as face, occlusions, etc.) in the image to be restored. At the same time, through the occlusion detection model, according to the image to be restored, the occlusion indication information used to indicate the occlusion therein is determined, that is, the occlusion in the image to be restored is detected, and the position and type of the occlusion in the image to be restored are determined; and, based on the occlusion indication information, feature extraction is performed to obtain the occlusion feature of the image to be restored, and the occlusion feature is specifically used to reflect the occlusion in the image to be restored. So far, through the above steps, multi-dimensional feature extraction of the image to be restored is realized, that is, the extraction of features for the image content of the image to be restored and the occlusions therein is realized, and the extraction of relevant features of the occlusion in the image to be restored is focused. Next, according to the above-mentioned occlusion indication information, the repair strategy parameters of the image repair model are determined, that is, according to the occlusion in the image to be repaired, the repair strategy of the image repair model is adaptively adjusted so that the repair strategy can be adapted to high-quality repair of the image including the occlusion. Furthermore, through the image repair model, the above-mentioned repair strategy parameters are adopted to generate a repaired image corresponding to the image to be repaired according to the image features and occlusion features of the image to be repaired, and the image quality of the repaired image is higher than that of the image to be repaired; in this way, the image repair model can simultaneously refer to the overall image content of the image to be repaired and the occlusion in the image to be repaired during the process of repairing the image, and repair the image to be repaired using a repair strategy adapted to the occlusion, thereby generating a high-quality, real and credible repaired image, and realizing reliable repair of the image to be repaired with occluded faces. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of an application scenario of the image restoration method provided in an embodiment of the present application;

[0037] Figure 2 A schematic diagram of the process of the image restoration method provided in the embodiment of the present application;

[0038] Figure 3 A schematic diagram of the implementation architecture of the image restoration method provided in the embodiment of the present application;

[0039] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0040] Figure 5A flowchart of another model training method provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of the structure of an image restoration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] In the related art, for images with occluded faces, image restoration models often have difficulty reliably restoring corresponding high-quality images. The specific reason is that for images with occluded faces, image restoration models often have difficulty effectively extracting the features of the occluded objects, and thus cannot reasonably consider the occluded objects in the image, making it difficult to reliably restore the corresponding high-quality images.

[0046] In order to solve the above problems, the embodiment of the present application provides an image restoration method. On the one hand, the method uses an occlusion detection model to detect occlusions in the image to be restored, obtains corresponding occlusion indication information, and performs feature extraction based on the occlusion indication information to obtain occlusion features; thereby, occlusion features specifically used to characterize the occlusions in the image to be restored are obtained, and the occlusion features are subsequently input into the image restoration model, so that the image restoration model can clearly know the occlusion situation in the image to be restored during the image restoration process, and is conducive to reliably restoring the image to be restored including the occlusion. On the other hand, the method determines the restoration strategy parameters of the image restoration model according to the above occlusion indication information, and enables the image restoration model to perform corresponding image restoration according to the restoration strategy parameters; thereby, the restoration strategy is adaptively adjusted according to the occlusions in the image to be restored, so that the restoration strategy adopted by the image restoration model is adapted to the occlusion situation in the image to be restored.

[0047] Specifically, the image restoration method provided in the embodiment of the present application includes: obtaining an image to be restored, which is a face image including an occlusion. Then, feature extraction is performed based on the image to be restored to obtain image features of the image to be restored, and the image features can reflect the image content (such as face, occlusions, etc.) in the image to be restored. At the same time, through the occlusion detection model, according to the image to be restored, the occlusion indication information used to indicate the occlusion therein is determined, that is, the occlusion in the image to be restored is detected, and the position and type of the occlusion in the image to be restored are determined; and, based on the occlusion indication information, feature extraction is performed to obtain the occlusion feature of the image to be restored, and the occlusion feature is specifically used to reflect the occlusion in the image to be restored. So far, through the above steps, multi-dimensional feature extraction of the image to be restored is realized, that is, the extraction of features for the image content of the image to be restored and the occlusions therein is realized, and the extraction of relevant features of the occlusion in the image to be restored is focused. Next, according to the above-mentioned occlusion indication information, the repair strategy parameters of the image repair model are determined, that is, according to the occlusion in the image to be repaired, the repair strategy of the image repair model is adaptively adjusted so that the repair strategy can be adapted to high-quality repair of the image including the occlusion. Furthermore, through the image repair model, the above-mentioned repair strategy parameters are adopted to generate a repaired image corresponding to the image to be repaired according to the image features and occlusion features of the image to be repaired, and the image quality of the repaired image is higher than that of the image to be repaired; in this way, the image repair model can simultaneously refer to the overall image content of the image to be repaired and the occlusion in the image to be repaired during the process of repairing the image, and repair the image to be repaired using a repair strategy adapted to the occlusion, thereby generating a high-quality, real and credible repaired image, and realizing reliable repair of the image to be repaired with occluded faces.

[0048] It should be noted that the image restoration method and model training method provided in the embodiments of the present application can be executed by a computer device, which can be a terminal device or a server. Among them, the terminal device can be a computer or other device with image processing function; the server can be an independent server or a cluster composed of multiple servers. In addition, in the method provided in the embodiments of the present application, the collection, use and processing of relevant data shall comply with the relevant laws, regulations and standards of relevant countries and regions.

[0049] To facilitate understanding of the image restoration method provided in the embodiment of the present application, the following uses a server as an example in which the execution subject of the image restoration method is used to exemplify the application scenario of the image restoration method.

[0050] See also Figure 1 , Figure 1 Schematic diagram of application scenarios of the image restoration method provided in the embodiment of the present application. Figure 1 As shown, the application scenario includes a server 110 and a terminal device 120, and the server 110 and the terminal device 120 can communicate through a network. The server 110 is used to execute the image restoration method provided in the embodiment of the present application, and the terminal device 120 is used to provide the server 110 with the image to be restored.

[0051] In actual applications, the terminal device 120 can transmit the image to be repaired to the server 110; for example, an application with an image repair function can be running on the terminal device 120, and the terminal device 120 can respond to the user's use of the image repair function in the application and transmit the image to be repaired (i.e., the image to be repaired) selected by the user to the server 110. The image to be repaired can be a face image including an occlusion.

[0052] After receiving the image to be repaired transmitted from the terminal device 120, the server 110 can perform feature extraction based on the image to be repaired to obtain the image features of the image to be repaired. At the same time, the server 110 can determine the occlusion indication information used to indicate the occlusion in the image to be repaired according to the image to be repaired through the occlusion detection model; and perform feature extraction based on the occlusion indication information to obtain the occlusion features of the image to be repaired. Then, the server 110 can determine the repair strategy parameters of the image repair model according to the occlusion indication information. Furthermore, the server 110 can use the image repair model, adopt the determined repair strategy parameters, and generate the repaired image corresponding to the image to be repaired according to the image features and occlusion features of the above-mentioned image to be repaired. The image quality of the repaired image is higher than that of the image to be repaired. Finally, the server 110 can transmit the generated repaired image to the terminal device 120 so that the repaired image can be displayed by the terminal device 120.

[0053] It should be understood that Figure 1 The application scenarios shown are only examples. In actual applications, the image restoration method provided in the embodiments of the present application can also be applied to other scenarios, and the embodiments of the present application do not impose any limitations on this.

[0054] The image restoration method and model training method provided in the embodiments of the present application are introduced in detail below through method examples.

[0055] See also Figure 2 , Figure 2 The following is a flow chart of the image restoration method provided in the embodiment of the present application. Figure 2 As shown, the method comprises the following steps:

[0056] Step 201: Acquire an image to be restored; the image to be restored is a face image including an occlusion.

[0057] The image to be repaired is an image that needs to be further repaired in the embodiment of the present application, and its image quality is usually low, such as low clarity, low resolution, etc. Since the embodiment of the present application is intended to repair an image with face occlusion, the image to be repaired in the embodiment of the present application is usually a face image including an occlusion, that is, an image in which the face is partially occluded by the occlusion.

[0058] As an example, when the execution subject of the image restoration method provided in the embodiment of the present application is a server, the server can obtain the image to be restored from the terminal device. For example, an application with an image restoration function is running in the terminal device. When a user uses the image restoration function in the application through the terminal device, the terminal device can transmit the image to be restored selected by the user to the server in response to the user's use of the image restoration function. Alternatively, the server can also obtain the image to be restored from a database. For example, the server can extract the video to be restored from the database, and then use the video frame in the video as the image to be restored.

[0059] Of course, in practical applications, the image to be restored may also be obtained by other methods, and the embodiment of the present application does not impose any limitation on the method for obtaining the image to be restored.

[0060] Step 202: performing feature extraction based on the image to be repaired to obtain image features of the image to be repaired.

[0061] After obtaining the image to be repaired, feature extraction processing can be performed based on the image to be repaired to obtain image features of the image to be repaired. The image features are features that can reflect the overall content of the image to be repaired. Specifically, the image features can reflect the face, occlusions, background, etc. in the image to be repaired.

[0062] As an example, any one or more neural network models capable of extracting image features may be used to perform feature extraction processing on the image to be repaired, thereby obtaining image features of the image to be repaired. The neural network models used may include, for example, but are not limited to, convolutional neural networks (CNN), constrastive language-image pre-training (CLIP) models using contrastive learning, and the like.

[0063] As an example, the image features of the image to be repaired may specifically include at least one of features extracted based on the image to be repaired itself, features extracted based on the description text of the image to be repaired, features extracted based on the depth information of the image to be repaired, and features extracted based on physical elements in the image to be repaired.

[0064] In a possible implementation, the above step 202 may include the following steps 2021 to 2024 (not shown in the figure):

[0065] Step 2021: extract features of the image to be repaired using an image feature extraction model to obtain a first image feature of the image to be repaired.

[0066] The image feature extraction model may be any neural network model that can extract image features, and the image feature extraction model may be, for example, a CNN model.

[0067] Exemplarily, the image to be repaired can be input into the image feature extraction model. Accordingly, the image feature extraction model can perform feature extraction processing on the image to be repaired to obtain a first image feature for characterizing the image content in the image to be repaired. The first image feature can specifically be a feature matrix of h*w dimensions, where h and w are both integers greater than or equal to 1, which can represent the image content (such as faces, occlusions, backgrounds, etc.) in the image to be repaired in the form of feature data.

[0068] Step 2022: Generate a description text corresponding to the image to be repaired based on the image to be repaired through a multimodal processing model; perform feature extraction on the description text to obtain a second image feature of the image to be repaired.

[0069] The multimodal processing model may be any neural network model that supports processing data in both image and text modes. The multimodal processing model may be, for example, a CLIP model.

[0070] Exemplarily, the image to be repaired can be input into the image coding network in the CLIP model, and the image coding network can obtain the corresponding image coding features by encoding the features of the image to be repaired. Since the CLIP model is usually trained based on training samples including images and texts with corresponding relationships, the CLIP model has the ability to align image coding features and text coding features, that is, the image coding features encoded by the image coding network in the CLIP model can reflect the corresponding text. It can also be understood that the information expressed by the image coding features is the same as the information expressed by a certain text, and the text expressing the same information can be restored based on the image coding features. Therefore, by performing text decoding processing on the image coding features output by the image coding network in the CLIP model, text reflecting the same information as the image coding features can be obtained, that is, the description text corresponding to the image to be repaired can be obtained.

[0071] Furthermore, the description text may be subjected to feature extraction processing, for example, by any text encoding network (such as a recurrent neural network (RNN), a Bert model, a Transformer model, etc.), thereby obtaining text features corresponding to the description text. Since the text features can reflect the image content of the image to be repaired to a certain extent, the text features can be used as the second image features of the image to be repaired.

[0072] Step 2023: Determine the depth information corresponding to the image to be repaired through a depth detection model; perform feature extraction on the depth information to obtain a third image feature of the image to be repaired.

[0073] The depth detection model may be any neural network model capable of detecting depth information of an image, and the depth detection model may be, for example, a regional convolutional neural network (R-CNN).

[0074] Exemplarily, the image to be repaired can be input into a depth detection model, and the depth detection model can determine the depth information corresponding to each pixel in the image to be repaired by analyzing and processing the image to be repaired; it should be understood that the depth information corresponding to each pixel in the image to be repaired can reflect the distance relationship between each element included in the image to be repaired and the camera, such as the distance relationship between the facial features on a person's face and the camera, the distance relationship between an obstruction and the camera, etc. Furthermore, feature extraction can be performed on the depth information corresponding to each pixel in the image to be repaired to convert the depth information into a corresponding feature representation, which is the third image feature of the image to be repaired, that is, the third feature is used to reflect the depth information of the image to be repaired.

[0075] Step 2024: Determine each entity element region included in the image to be repaired through an image segmentation model; perform feature extraction on each entity element region to obtain a fourth image feature of the image to be repaired.

[0076] The image segmentation model can be any neural network model that can detect and identify each entity element included in the image. The image segmentation model can be, for example, Unet, YOLO (You Only Look Once) model, etc. The entity element here can be any recognizable entity, such as a face, facial features, occlusions, objects in the image background, etc.

[0077] Exemplarily, the image to be repaired can be input into an image segmentation model, and the image segmentation model can output an image segmentation result corresponding to the image to be repaired by performing segmentation processing on the image to be repaired. The image segmentation result can specifically be each entity element region included in the image to be repaired, which is marked on the basis of the image to be repaired, and the type of entity element included in each entity element region is marked. Furthermore, feature extraction can be performed on each entity element region separately, for example, by using any neural network model with image feature extraction capability to extract features of the entity element region, so as to obtain the regional features corresponding to each entity element region; the regional features corresponding to each entity element region can be used to obtain the fourth image feature of the image to be repaired, and the fourth image feature can characterize the image content of the image to be repaired from the perspective of the entity element.

[0078] Figure 3 Schematic diagram of the implementation architecture of the image restoration method provided in the embodiment of the present application. Figure 3As shown, for an image to be repaired with face occlusion, the first image feature corresponding to the image to be repaired can be obtained through the image feature extraction model; the description text corresponding to the image to be repaired can be determined through the CLIP model, and the second image feature corresponding to the image to be repaired can be obtained based on the description text; the depth information of the image to be repaired can be determined through the depth detection model, and the third image feature corresponding to the image to be repaired can be obtained based on the depth information; the various entity element regions included in the image to be repaired can be determined through the image segmentation model, and the fourth image feature corresponding to the image to be repaired can be obtained based on the various entity element regions. The above-mentioned first image feature, second image feature, third image feature and fourth image feature will be provided to the image repair model later, so that it can be used as a reference when repairing the image.

[0079] In this way, through the above method, by extracting image features of different dimensions and modalities for the image to be repaired, a full-dimensional expression of the image content of the image to be repaired can be achieved. Accordingly, the image repair model is subsequently used to perform image repair based on the above-mentioned first image feature, second image feature, third image feature and fourth image feature at the same time, which can comprehensively take into account the content of each part of the image to be repaired, avoid loss of details, and help improve the quality of the generated repaired image.

[0080] Step 203: Determine occluder indication information for indicating the occluder in the image to be repaired according to the image to be repaired through the occluder detection model; perform feature extraction based on the occluder indication information to obtain occluder features of the image to be repaired.

[0081] The occluder detection model is a neural network model for detecting occluders that obstruct the face in an image. The occluder detection model can specifically detect whether there is an occluder in the image, and when it is detected that there is an occluder, determine the position of the occluder in the image and the type of the occluder. It should be noted that the occluder detection model is a model specifically used for detecting occluders proposed in the embodiments of the present application, and its model structure can be any neural network model that supports the execution of target detection tasks, such as a convolutional neural network model, etc. The embodiments of the present application do not specifically limit this; in addition, the embodiments of the present application also propose a corresponding training mechanism for the occluder detection model, which will be discussed below. Figure 5 The illustrated embodiment provides a detailed introduction to the training method of the occlusion detection model.

[0082] The occluder indication information is information used to indicate the occluder in the image to be repaired. The occluder indication information may specifically include the position and type of the occluder determined by the occluder detection model, and may also include other derivative information determined according to the position and / or type of the occluder, such as the size of the occluder, the occlusion ratio of the occluder to the face in the image to be repaired, and the like.

[0083] For example, Figure 3 As shown, the image to be repaired can be directly input into a pre-trained occlusion detection model, and the occlusion detection model can output the occlusion detection result corresponding to the image to be repaired by analyzing and processing the image to be repaired; if the image to be repaired includes an occlusion, the position and type of the occlusion are marked in the image to be repaired as the occlusion detection result corresponding to the image to be repaired. Furthermore, according to the position and type of the occlusion in the image to be repaired, the occlusion indication information for indicating the occlusion can be determined, such as determining the position and type of the occlusion as the occlusion indication information, and other relevant information determined according to the position and / or type of the occlusion is also determined as the occlusion indication information.

[0084] Furthermore, feature extraction (which can also be understood as feature encoding) may be performed on the above-mentioned occluder indication information to obtain corresponding occluder features, where the occluder features are feature data representing the occluder in the image to be repaired.

[0085] It should be noted that, in actual applications, the above steps 202 and 203 can be executed simultaneously, or step 202 can be executed first and then step 203, or step 203 can be executed first and then step 202. This application does not impose any limitation on the execution order of the above steps 202 and 203.

[0086] Step 204: Determine the restoration strategy parameters of the image restoration model according to the occlusion indication information.

[0087] The image restoration model is a neural network model used to perform image restoration tasks in the embodiments of the present application. Through the image restoration model, low-quality images can be restored to corresponding high-quality images. For example, low-definition images can be restored to high-definition images while restoring blurred details in the image. The image restoration model can be any neural network model that supports performing image restoration tasks, such as a large mask inpainting (LaMa) model, etc. The embodiments of the present application do not specifically limit the model structure of the image restoration model; in addition, the embodiments of the present application also propose a corresponding training mechanism for the image restoration model, which will be discussed below. Figure 4 The illustrated embodiment provides a detailed introduction to the training method of the image restoration model.

[0088] The restoration strategy parameter is a parameter that the image restoration model refers to when performing image restoration tasks. This parameter can determine the restoration strategy adopted by the image restoration model. For example, it can determine the restoration granularity of the image by the image restoration model, the image area that the image restoration model focuses on when performing image restoration tasks, etc.

[0089] For example, Figure 3 As shown, the restoration strategy parameters used by the image restoration model when restoring the image to be restored can be determined according to the predetermined restoration strategy parameter setting rules and the determined occlusion indication information; the restoration strategy parameter setting rules can specifically record the correspondence between multiple occlusion indication information and restoration strategy parameters. Furthermore, the determined restoration strategy parameters can be transmitted to the image restoration model so that it can perform the image restoration task for the image to be restored accordingly.

[0090] In a possible implementation, the above step 204 may include the following steps 2041 to 2042:

[0091] Step 2041: Determine the restoration parameters of the image restoration model according to the size of the occluder indicated by the occluder indication information.

[0092] The occluder indication information may specifically indicate the size of the occluder included in the image to be repaired; for example, the size of the occluder may be determined according to the position of the occluder in the image to be repaired output by the occluder detection model.

[0093] The repair strategy parameters of the image repair model may include a repair parameter, which may determine the repair strength of the input image of the image repair model. It should be understood that the repair strength and the repair parameter may be positively correlated, that is, the larger the repair parameter, the greater the repair strength.

[0094] Specifically, when determining the restoration parameters of the image restoration model according to the size of the occluder indicated by the occluder indication information, the restoration parameters corresponding to the size of the occluder indicated by the occluder indication information can be searched in the correspondence between the pre-set occluder size and the restoration parameters. It should be understood that the restoration parameters and the occluder size are usually positively correlated, that is, the larger the occluder size, the larger the restoration parameters; thereby, the purpose of taking a greater restoration force to restore the image to be restored including the larger occluder is achieved.

[0095] Step 2042: Determine the occlusion relationship between the occlusion and the face in the image to be repaired based on the occlusion indication information, and determine the focus of attention of the image repair model based on the occlusion relationship.

[0096] When the occluder indication information includes the position information of the occluder in the image to be repaired, the occlusion relationship between the occluder and the face in the image to be repaired can be further determined based on the position information. The occlusion relationship can, for example, indicate the occlusion ratio of the occluder to the face.

[0097] The repair strategy parameters of the image repair model may also include an attention focus parameter, which may determine the size of the image area that the image repair model focuses on. It should be understood that the size of the image area that is focused on may be positively correlated with the attention focus parameter, that is, the larger the attention focus parameter, the larger the image area that is focused on.

[0098] Specifically, when determining the focus parameter of the image restoration model according to the occlusion ratio of the occluder to the face in the image to be restored, the focus parameter corresponding to the occlusion ratio calculated according to the occluder indication information can be found in the correspondence between the preset occlusion ratio and the focus parameter. It should be understood that the focus parameter and the occlusion ratio are usually positively correlated, that is, the larger the occlusion ratio, the larger the focus parameter; thereby, the purpose of expanding the key restoration area for the image to be restored where the face is more occluded is achieved.

[0099] In this way, through the above method, the repair parameters used to determine the repair strength of the image repair model are determined according to the size of the occlusion, and the attention focus parameters used to determine the size of the key focus area of ​​the image repair model are determined according to the occlusion ratio of the occlusion to the face. This can achieve adaptive adjustment of the image repair strategy adopted by the image repair model, so that the image repair strategy is adapted to the occlusion situation in the current image to be repaired, which is conducive to the image repair model to better repair the image to be repaired.

[0100] Step 205: Generate a restored image corresponding to the image to be restored by using the image restoration model and the restoration strategy parameters according to the image features and the occlusion features; the image quality of the restored image is higher than the image quality of the image to be restored.

[0101] Finally, the image restoration model can determine the image restoration strategy for the image to be restored based on the restoration strategy parameters determined by step 204, and adopt the image restoration strategy to perform corresponding image restoration processing on the image to be restored according to the image features extracted by step 202 and the occlusion features extracted by step 203, and then output the restored image corresponding to the image to be restored.

[0102] It should be understood that the output restored image has higher image quality than the image to be restored, such as higher definition, clearer presentation of details in the image, etc.

[0103] In a possible implementation, if the first image feature, the second image feature, the third image feature and the fourth image feature are extracted simultaneously when the image feature is extracted through step 202, then the above step 205 may specifically include step 2051 and step 2052 (not shown in the figure):

[0104] Step 2051: Fusing the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature to obtain a fused feature.

[0105] For example, Figure 3 As shown, before the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature are input into the image restoration model, the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature can be fused to obtain corresponding fused features. The fused features can comprehensively represent the overall image content of the image to be restored, the descriptive text content corresponding to the image to be restored, the depth information of the image to be restored, the various entity elements in the image to be restored, and the occlusions in the image to be restored.

[0106] As an example, the fused feature may be generated through a feature fusion network according to the first image feature, the second image feature, the third image feature, the fourth image feature, and the occlusion feature.

[0107] Specifically, a feature fusion network (such as concatenation, etc.) with feature fusion function can be used to fuse the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature, that is, the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature are all input into the feature fusion network, and the feature fusion network can output the fused fusion feature accordingly. Since such feature fusion network has the function of reasonably fusing features of different modalities, the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature that reflect different information can be fully and effectively fused, thereby ensuring that the obtained fusion feature can retain the effective information reflected by each feature, while deleting the redundant information in each feature.

[0108] As an example, based on the attention mechanism, the attention weight of the combined feature composed of the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature can be determined, and the combined feature can be weighted according to the attention weight to obtain the fused feature.

[0109] Specifically, the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature with the same feature dimension can be combined to obtain a combined feature. For example, the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature can be combined into a large matrix as a combined feature.

[0110] Then, the combined feature can be processed by the attention mechanism to determine the corresponding attention weight; it should be understood that the process of determining the attention weight by the attention mechanism here is essentially the process of determining the importance of each dimension of the combined feature. The more important data in the combined feature will correspond to a higher weight value in the attention weight, while the less important data in the combined feature will correspond to a lower weight value in the attention weight. Furthermore, the determined attention weight can be used to perform weighted processing on the combined feature to strengthen the more important data in the combined feature and weaken the less important data in the combined feature, thereby obtaining the corresponding fusion feature.

[0111] Therefore, the attention mechanism is used to strengthen the important information in the first image features, the second image features, the third image features, the fourth image features and the occluder features, and weaken the unimportant information, so that the obtained fusion features can specifically reflect the information expressed by the first image features, the second image features, the third image features, the fourth image features and the occluder features.

[0112] Of course, in actual applications, other methods can also be used to fuse the first image features, the second image features, the third image features, the fourth image features and the occlusion features, for example, bitwise superposition processing of the first image features, the second image features, the third image features, the fourth image features and the occlusion features, splicing processing of the first image features, the second image features, the third image features, the fourth image features and the occlusion features, etc. The embodiments of the present application do not specifically limit the fusion processing method adopted.

[0113] Step 2052: Generate a restored image corresponding to the image to be restored by using the image restoration model and the restoration strategy parameters according to the fusion features.

[0114] For example, Figure 3 As shown, after the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature are fused and processed to obtain the corresponding fused feature, the fused feature can be input into the image restoration model. Accordingly, the image restoration model can adopt the image restoration strategy determined based on the image restoration parameter to analyze and process the fused feature, and then generate a restored image corresponding to the image to be restored.

[0115] In the image restoration method provided in the embodiment of the present application, an image to be restored is first obtained, and the image to be restored is a face image including an occlusion. Then, feature extraction is performed based on the image to be restored to obtain the image features of the image to be restored, and the image features can reflect the image content (such as face, occlusions, etc.) in the image to be restored. At the same time, through the occlusion detection model, according to the image to be restored, the occlusion indication information used to indicate the occlusion therein is determined, that is, the detection of the occlusion in the image to be restored is realized, and the position and type of the occlusion in the image to be restored are determined; and, based on the occlusion indication information, feature extraction is performed to obtain the occlusion features of the image to be restored, and the occlusion features are specifically used to reflect the occlusion in the image to be restored. So far, through the above steps, multi-dimensional feature extraction of the image to be restored is realized, that is, the extraction of features for the image content of the image to be restored and the occlusions therein is realized, and the extraction of relevant features of the occlusions in the image to be restored is focused. Next, according to the above-mentioned occlusion indication information, the repair strategy parameters of the image repair model are determined, that is, according to the occlusion in the image to be repaired, the repair strategy of the image repair model is adaptively adjusted so that the repair strategy can be adapted to high-quality repair of the image including the occlusion. Furthermore, through the image repair model, the above-mentioned repair strategy parameters are adopted to generate a repaired image corresponding to the image to be repaired according to the image features and occlusion features of the image to be repaired, and the image quality of the repaired image is higher than that of the image to be repaired; in this way, the image repair model can simultaneously refer to the overall image content of the image to be repaired and the occlusion in the image to be repaired during the process of repairing the image, and repair the image to be repaired using a repair strategy adapted to the occlusion, thereby generating a high-quality, real and credible repaired image, and realizing reliable repair of the image to be repaired with occluded faces.

[0116] See also Figure 4 , Figure 4 A flow chart of a model training method provided in an embodiment of the present application, wherein the model training method is used to train the above Figure 2 The image restoration model used in the method shown. Figure 4 As shown, the method comprises the following steps:

[0117] Step 401: Acquire training samples; the training samples include training images to be repaired and corresponding training images after repair, and the training images to be repaired are face images including occluders.

[0118] In an embodiment of the present application, before training an image restoration model, it is necessary to first obtain a plurality of training samples for training the image restoration model. For example, the training samples may be retrieved from a relevant database.

[0119] Each training sample includes two images, namely, a training image to be repaired and its corresponding training image after repair. The training image to be repaired is the image that needs to be processed by the image repair model to be trained during the model training process, that is, the image repair model needs to repair the training image to be repaired, and continuously improve its own image repair ability during the repair process; in the embodiment of the present application, the training image to be repaired is a face image including an occluder, that is, an image in which the face is obscured by the occluder. Such training images to be repaired can be obtained by sampling, such as collecting images in which the face is obscured by the occluder from an open source database as training images to be repaired. Such training images to be repaired can also be generated using a specific image generation strategy, such as synthesizing the occluder image and the face image to generate a training image to be repaired. The embodiment of the present application does not impose any limitation on such training images to be repaired. The training repaired image is an image used as a training reference. When training an image repair model, it is necessary to compare the image obtained by repairing the training image to be repaired in the training sample with the image repaired image, and then train the image repair model based on the difference obtained by the comparison, so as to improve the performance of the image repair model and make the output repaired image gradually approach the training repaired image; the training repaired image can be obtained by manually repairing the training image to be repaired.

[0120] It should be understood that when training an image restoration model, the training samples used may include other types of training images to be restored in addition to the training samples of the training images to be restored including occluded objects, that is, the training samples used are not limited to images with face occlusions, thereby enabling the image restoration model to learn the ability to restore various types of images and enhance the generalization of the image restoration model. However, since the embodiments of the present application focus on the restoration of images with face occlusions, when introducing the model training scheme, only the training scheme based on training samples including images with face occlusions is introduced, but this does not represent a limitation on the training samples actually used when training the image restoration model.

[0121] Step 402: performing feature extraction based on the training image to be repaired to obtain image features of the training image to be repaired.

[0122] Step 403: Determine, through the occlusion detection model, occlusion indication information for indicating the occlusion in the training image to be repaired according to the training image to be repaired; perform feature extraction based on the occlusion indication information to obtain the occlusion feature of the training image to be repaired.

[0123] Step 404: Determine the restoration strategy parameters of the image restoration model to be trained according to the occlusion indication information.

[0124] Step 405: adopting the restoration strategy parameters, and training the image restoration model according to the image features, the occlusion features and the training restored image.

[0125] After the repair strategy parameters are determined through step 404, the image repair model to be trained can be used to repair the training repair image based on the repair strategy parameters. That is, the image repair model can be used to process the image features determined through step 402 and the occlusion features determined through step 403 based on the repair strategy parameters to obtain the corresponding output results. Then, the image repair model can be trained based on the difference between the output result and the training repaired image corresponding to the training repaired image.

[0126] In a possible implementation, step 405 may include the following steps 4051 and 4052:

[0127] Step 4051: Generate a predicted restored image corresponding to the training image to be restored by using the image restoration model and the restoration strategy parameters according to the image features and the occlusion features.

[0128] The processing of the above steps 402 to 4051 is Figure 2 The processing of step 202 to step 205 in the illustrated embodiment is similar, the only difference is that the processing of step 402 to step 4051 is performed in the model training stage, while the processing of step 202 to step 205 is performed in the model application stage. However, the basic implementation principles of the two processing flows are the same, so the detailed implementation of the processing of step 402 to step 405 can refer to the introduction of the processing of step 202 to step 205 above, and will not be repeated here.

[0129] It should be understood that in order to distinguish the data in the model training phase from the data in the model application phase, in steps 402 to 4051, the image to be inpainted by the image inpainting model is called a training image to be inpainted, which is essentially the same as the training image. Figure 2 The image to be repaired in the embodiment shown is the same. And the repaired image output by the image repair model is called the predicted repaired image, which is essentially the same as Figure 2 The restored images in the illustrated embodiments are the same; however, due to Figure 4 The illustrated embodiment introduces the training process of the image restoration model. At this stage, the performance of the image restoration model has not yet reached the standard. Therefore, the predicted restored image output by the image restoration model at this stage is usually not very accurate, and there is much room for improvement.

[0130] Step 4052: Determine a first loss based on the predicted restored image and the trained restored image; and train the image restoration model based on the first loss.

[0131] After the image to be repaired in the training sample is repaired by the image repair model and the corresponding predicted repaired image is obtained, the model loss used to train the image repair model, i.e., the first loss, can be determined according to the difference between the predicted repaired image and the training repaired image in the training sample. Then, the back propagation algorithm is used to update the model parameters of the image repair model according to the first loss to achieve the purpose of training the image repair model.

[0132] It should be noted that the image restoration model can specifically be any neural network model that can perform image restoration tasks, such as a Large Mask inpainting (LaMa) model.

[0133] In a possible implementation, the above step 4052 may include steps 4061 to 4065 (not shown in the figure):

[0134] Step 4061: Determine the reconstruction loss based on the difference between the predicted restored image and the trained restored image.

[0135] Specifically, the difference between the predicted restored image output by the image restoration model and the trained restored image in the training sample can be directly calculated as the reconstruction loss. The reconstruction loss can characterize the gap between the current image restoration capability of the image restoration model and the standard image restoration capability. The specific calculation formula of the reconstruction loss is shown in the following formula (1):

[0136]

[0137] Among them, L mse is the reconstruction loss, x i is the training image to be repaired in the i-th training sample, G(x i ) is the image restoration model trained on the image to be restored x i The predicted restored image obtained after restoration, is the training restored image in the i-th training sample.

[0138] Step 4062: Perform feature extraction on the predicted restored image and the trained restored image respectively through a deep convolutional neural network to obtain a first depth image feature corresponding to the predicted restored image and a second depth image feature corresponding to the trained restored image; determine the perceptual loss based on the difference between the first depth image feature and the second depth image feature.

[0139] Specifically, the predicted restored image and the training restored image having a corresponding relationship can be respectively input into a deep convolutional neural network to extract deep features, thereby obtaining a first deep image feature corresponding to the predicted restored image and a second deep image feature corresponding to the training restored image. The deep convolutional neural network here can be, for example, a Visual Geometry Group (VGG) model.

[0140] Then, the difference between the first depth image feature and the second depth image feature is calculated as the perceptual loss, which can characterize the gap between the current image restoration capability of the image restoration model and the standard image restoration capability from the dimension of deeper features. The specific calculation formula of the perceptual loss is shown in the following formula (2):

[0141]

[0142] Among them, L perceptual is the perceived loss; i =G(x i ), represents the training image to be repaired x in the i-th training sample i The predicted restored image obtained after being restored by the image restoration model, is the training restored image in the i-th training sample; Φ j Represents the processing of a deep convolutional neural network.

[0143] Step 4063: Determine the adversarial loss based on the predicted restored image through the discriminator network.

[0144] The discriminator network is a discriminator in a generative adversarial network (GAN). In the embodiment of the present application, the image restoration model is equivalent to the generator in the generative adversarial network. The discriminator network is used to determine the probability that the predicted restored image output by the image restoration model is a true restored image. In the embodiment of the present application, the performance of the image restoration model is improved by confusing the discriminator network so that it cannot accurately distinguish whether the image output by the image restoration model is a true restored image.

[0145] When determining the adversarial loss, the predicted restored image output by the image restoration model can be directly input into the discriminator network, and then the predicted probability value output by the discriminator network is used as the adversarial loss. The adversarial loss is used to characterize the probability that the predicted restored image is the true restored image. The specific calculation formula of the adversarial loss is shown in the following formula (3):

[0146] L adv = -log(D(G(x i )))(3)

[0147] Among them, L adv To counter the loss, D represents the processing of the discriminator network, G represents the generator network, that is, the image restoration model, G(x i ) represents the training image to be repaired x in the i-th training sample i The predicted restored image obtained after being restored by the image restoration model.

[0148] Step 4064: Generate a first description text corresponding to the predicted restored image and a second description text corresponding to the training restored image through a multimodal processing model; determine the description consistency loss based on the difference between a first text feature corresponding to the first description text and a second text feature corresponding to the second description text.

[0149] The multimodal processing model has been described above. Figure 2 In the embodiment shown, it is introduced that it is a neural network model that can convert an image into a corresponding description text, for example, it can be a CLIP model. In an embodiment of the present application, the predicted repaired image and the training repaired image can be respectively input into the CLIP model to obtain a first description text corresponding to the predicted repaired image and a second description text corresponding to the training repaired image. Then, the first description text and the second description text can be respectively subjected to text encoding processing to obtain a first text feature corresponding to the first description text and a second text feature corresponding to the second description text. Furthermore, the difference between the above-mentioned first text feature and the second text feature can be calculated as a description consistency loss, and the description consistency loss is used to characterize the difference between the description text corresponding to the predicted repaired image and the description text corresponding to the training repaired image. The specific calculation formula for the description consistency loss is shown in the following formula (4):

[0150]

[0151] Among them, L clip To describe the consistency loss, CLIPimg(y i ) is the first text feature of the first description text corresponding to the predicted restored image, The second text feature of the second description text corresponding to the training restored image.

[0152] It should be understood that in actual applications, the above steps 4061, 4062, 4063, and 4064 can be executed simultaneously or in series according to a preset sequence. The embodiment of the present application does not impose any limitation on the execution order of steps 4061, 4062, 4063, and 4064.

[0153] Step 4065: Determine the first loss according to the reconstruction loss, the perceptual loss, the adversarial loss and the description consistency loss.

[0154] Finally, the calculated reconstruction loss, perception loss, adversarial loss and description consistency loss can be weighted and summed according to the preset loss weights to obtain the corresponding first loss.

[0155] In this way, through the above method, different loss calculation methods are used to calculate the reconstruction loss, perception loss, adversarial loss and description consistency loss used to characterize different information from different dimensions, and these losses are combined to calculate the first loss. This can ensure that the calculated first loss can comprehensively reflect the current performance of the image restoration model, and based on this, the image restoration model is comprehensively optimized so that it can fully learn the corresponding restoration methods in all dimensions.

[0156] Based on the acquired training samples, multiple rounds of steps 402 to 406 are iteratively performed until an image restoration model that meets the training end condition is obtained. It should be understood that the training end condition here can be a requirement condition for model performance, such as requiring the accuracy of image restoration to reach a preset accuracy threshold, or it can be a restriction condition for the number of training rounds, such as the number of training rounds reaching a preset number of training rounds, etc. The embodiment of the present application does not impose any limitation on the training end condition.

[0157] In the model training method provided in the embodiment of the present application, the trained image restoration model refers to the overall image content of the training image to be restored and the occlusion in the training image to be restored during the image restoration process, and uses a restoration strategy that matches the occlusion to restore the training image to be restored, thereby generating a corresponding predicted restored image, so that the image restoration model learns the restoration method for face occluded images, improves its restoration performance for face occluded images, and enables it to reliably restore face occluded images. In addition, the image restoration model can be jointly trained with multiple losses, and the image restoration model can be trained from multiple dimensions to achieve an all-round improvement in the capabilities of the image restoration model.

[0158] See also Figure 5 , Figure 5 A flow chart of another model training method provided in an embodiment of the present application, which is used to train the above Figure 2 and Figure 4 The occlusion detection model used in the method shown. Figure 5 As shown, the method comprises the following steps:

[0159] Step 501: Acquire a training image; the training image has a corresponding label, and the label is used to indicate the marked position and marking type of the occlusion in the training image.

[0160] In the embodiment of the present application, before training the occluder detection model, a large number of training images need to be acquired, and the acquired training images are the images used when training the occluder detection model.

[0161] The acquired training images may include images with occluders (i.e., images in which the face is obstructed by the occluders), and may also include images without occluders (e.g., images in which the face is completely exposed). For training images with occluders, the corresponding labels will be used to indicate the presence of occluders in the training images, and may also indicate the position and type of the occluders in the training images. The indicated position is referred to as the annotation position, and the indicated type is referred to as the annotation type. For images without occluders, the corresponding labels only need to indicate that there are no occluders in the training images.

[0162] It should be understood that the above-mentioned training images can be obtained by sampling, such as collecting images of faces obstructed by occluders and images of faces fully exposed from an open source database as training images; the above-mentioned training images can also be generated using a specific image generation strategy, such as synthesizing occluder images and face images to generate training images with occluders; the embodiments of the present application do not impose any restrictions on the method of obtaining training images. In addition, the labels corresponding to the training images can be manually annotated or automatically annotated when the training images are generated, such as automatically annotating the position of the occluder in the synthesized image and the type of the occluder when synthesizing the occluder image and the face image. The embodiments of the present application do not impose any restrictions on the method of obtaining the labels corresponding to the training images.

[0163] Step 502: Determine the occluder detection result corresponding to the training image according to the training image by using the occluder detection model to be trained.

[0164] Then, each training image can be input into the occlusion detection model to be trained, and the occlusion detection model performs detection processing on the training image to determine the occlusion detection result corresponding to the training image. It should be understood that the occlusion detection result can represent whether the input training image includes an occlusion. In the case where the occlusion detection result represents that the input training image includes an occlusion, the occlusion detection result can further represent the position and type of the occlusion in the training image. The position represented by the occlusion detection result is called the predicted position, and the type represented by the occlusion detection result is called the predicted type.

[0165] It should be noted that the above-mentioned occlusion detection model can specifically be any neural network model that supports executing image detection tasks, such as a convolutional neural network model.

[0166] Step 503: Determine a second loss according to the difference between the occlusion detection result corresponding to the training image and the label corresponding to the training image; and train the occlusion detection model based on the second loss.

[0167] Furthermore, the model loss of the occlusion detection model, i.e., the second loss, can be determined according to the difference between the occlusion detection result and the label corresponding to each training image. The back propagation algorithm is used to update the model parameters of the occlusion detection model according to the second loss, thereby achieving the purpose of training the occlusion detection.

[0168] Based on the acquired training samples, multiple rounds of steps 502 to 503 are iteratively performed until an occlusion detection model that satisfies the training end condition is obtained. It should be understood that the training end condition here can be a requirement condition for model performance, such as requiring the detection accuracy to reach a preset accuracy threshold, or a restriction condition for the number of training rounds, such as the number of training rounds reaching a preset number of training rounds, etc. The present embodiment of the application does not impose any limitation on the training end condition.

[0169] Thus, through the above method, based on a wealth of training images including and excluding occluders, the occluder detection model can be fully trained to acquire accurate occluder detection capabilities. Accordingly, the occluder detection model is applied to the restoration process of face occluded images, and based on the occluder information accurately detected by the occluder detection model, the image restoration model can perform better restoration on the image including occluders, thereby further improving the image restoration quality.

[0170] The present application also provides a model training device, see Figure 6 , Figure 6 A schematic diagram of the structure of the model training device provided in the embodiment of the present application, such as Figure 6 As shown, the model training device includes:

[0171] The sample acquisition module 601 is used to acquire training samples; the training samples include training images to be repaired and corresponding training images after repair, and the training images to be repaired are human face images including occluders;

[0172] A first feature extraction module 602, configured to extract features based on the training image to be repaired, and obtain image features of the training image to be repaired;

[0173] A second feature extraction module 603 is used to determine, through an occlusion detection model and according to the training image to be repaired, occlusion indication information used to indicate the occlusion in the training image to be repaired; and perform feature extraction based on the occlusion indication information to obtain occlusion features of the training image to be repaired;

[0174] A first strategy determination module 604, configured to determine a restoration strategy parameter of an image restoration model to be trained according to the occlusion indication information;

[0175] The model training module 605 is used to train the image restoration model using the restoration strategy parameters according to the image features, the occlusion features and the training restored image.

[0176] Optionally, the model training module 605 is specifically used for:

[0177] By using the image restoration model and the restoration strategy parameters, a predicted restored image corresponding to the training image to be restored is generated according to the image features and the occluder features;

[0178] Determining a first loss according to the predicted restored image and the training restored image;

[0179] Based on the first loss, the image restoration model is trained.

[0180] Optionally, the model training module 605 is specifically used for:

[0181] Determining a reconstruction loss according to a difference between the predicted restored image and the training restored image;

[0182] By using a deep convolutional neural network, feature extraction is performed on the predicted restored image and the training restored image respectively to obtain a first depth image feature corresponding to the predicted restored image and a second depth image feature corresponding to the training restored image; and a perceptual loss is determined according to a difference between the first depth image feature and the second depth image feature;

[0183] Determining, by a discriminator network, an adversarial loss according to the predicted restored image;

[0184] Generate a first description text corresponding to the predicted restored image and a second description text corresponding to the training restored image through a multimodal processing model; determine a description consistency loss according to a difference between a first text feature corresponding to the first description text and a second text feature corresponding to the second description text;

[0185] The first loss is determined according to the reconstruction loss, the perceptual loss, the adversarial loss and the description consistency loss.

[0186] The present application also provides an image restoration device. Figure 7 , Figure 7 A schematic diagram of the structure of the image restoration device provided in the embodiment of the present application is shown in FIG. Figure 7 As shown, the image restoration device comprises:

[0187] The image acquisition module 701 is used to acquire the image to be restored; the image to be restored is a face image including an occluder;

[0188] A third feature extraction module 702 is used to extract features based on the image to be repaired to obtain image features of the image to be repaired;

[0189] The fourth feature extraction module 703 is used to determine, according to the image to be repaired, occluder indication information for indicating the occluder in the image to be repaired by using the occluder detection model; and perform feature extraction based on the occluder indication information to obtain occluder features of the image to be repaired;

[0190] A second strategy determination module 704, configured to determine a restoration strategy parameter of an image restoration model according to the occlusion indication information;

[0191] The image restoration module 705 is used to generate a restored image corresponding to the image to be restored according to the image features and the occluder features by using the image restoration model and the restoration strategy parameters; the image restoration model is based on Figure 4 The image quality of the restored image obtained by training with the model training method shown is higher than the image quality of the image to be restored.

[0192] Optionally, the third feature extraction module 702 is specifically used for:

[0193] Performing feature extraction on the image to be repaired by using an image feature extraction model to obtain a first image feature of the image to be repaired;

[0194] Generate a description text corresponding to the image to be repaired according to the image to be repaired through a multimodal processing model; extract features from the description text to obtain a second image feature of the image to be repaired;

[0195] Determine the depth information corresponding to the image to be repaired by using a depth detection model; extract features from the depth information to obtain a third image feature of the image to be repaired;

[0196] The various entity element regions included in the image to be repaired are determined by using an image segmentation model; and feature extraction is performed on each of the entity element regions to obtain a fourth image feature of the image to be repaired.

[0197] Optionally, the image restoration module 705 is specifically used for:

[0198] fusing the first image feature, the second image feature, the third image feature, the fourth image feature, and the occlusion feature to obtain a fused feature;

[0199] The image restoration model is used to generate a restored image corresponding to the image to be restored according to the fusion features by adopting the restoration strategy parameters.

[0200] Optionally, the image restoration module 705 is specifically used for:

[0201] generating the fused feature according to the first image feature, the second image feature, the third image feature, the fourth image feature and the occluder feature through a feature fusion network;

[0202] or,

[0203] Based on the attention mechanism, the attention weight of the combined feature composed of the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature is determined, and the combined feature is weighted according to the attention weight to obtain the fused feature.

[0204] Optionally, the second strategy determination module 704 is specifically configured to:

[0205] Determining a restoration parameter of the image restoration model according to the size of the occluder indicated by the occluder indication information;

[0206] According to the occlusion indication information, the occlusion relationship between the occlusion and the face in the image to be repaired is determined, and the attention focus of the image repair model is determined according to the occlusion relationship to serve as a repair strategy parameter of the image repair model.

[0207] Optionally, the device further includes an occlusion detection model training module, which is used to:

[0208] Acquire a training image; the training image has a corresponding label, and the label is used to indicate the marked position and marking type of the occlusion in the training image;

[0209] Determining, by means of the occlusion detection model to be trained and according to the training image, an occlusion detection result corresponding to the training image;

[0210] Determine a second loss according to a difference between an occluder detection result corresponding to the training image and a label corresponding to the training image; and train the occluder detection model based on the second loss.

[0211] An embodiment of the present application also provides an electronic device, which includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the image restoration method or model training method described in the above method embodiment according to the computer program.

[0212] An embodiment of the present application also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a computer device, it implements the image restoration method or model training method described in the above method embodiment.

[0213] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the image restoration method or model training method described in the above method embodiment.

[0214] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0215] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0216] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0217] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0218] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store computer programs.

[0219] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0220] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a training sample; the training sample includes a training image to be repaired and a corresponding training repaired image, wherein the training image to be repaired is a face image including an occluded object; Performing feature extraction based on the training image to be repaired to obtain image features of the training image to be repaired; Determining, by means of an occlusion detection model, occlusion indication information for indicating the occlusion in the training image to be repaired according to the training image to be repaired; performing feature extraction based on the occlusion indication information to obtain occlusion features of the training image to be repaired; Determining, according to the occlusion indication information, a restoration strategy parameter of the image restoration model to be trained; The image restoration model is trained using the restoration strategy parameters according to the image features, the occlusion features and the training restored image.

2. The method according to claim 1, characterized in that The adopting the restoration strategy parameters, training the image restoration model according to the image features, the occlusion features and the training restored image, comprises: By using the image restoration model and the restoration strategy parameters, a predicted restored image corresponding to the training image to be restored is generated according to the image features and the occluder features; A first loss is determined according to the predicted restored image and the trained restored image; and the image restoration model is trained based on the first loss.

3. The method according to claim 2, characterized in that The determining of a first loss according to the predicted restored image and the training restored image comprises: Determining a reconstruction loss according to a difference between the predicted restored image and the training restored image; By using a deep convolutional neural network, feature extraction is performed on the predicted restored image and the training restored image respectively to obtain a first depth image feature corresponding to the predicted restored image and a second depth image feature corresponding to the training restored image; and a perceptual loss is determined according to a difference between the first depth image feature and the second depth image feature; Determining, by a discriminator network, an adversarial loss according to the predicted restored image; Generate a first description text corresponding to the predicted restored image and a second description text corresponding to the training restored image through a multimodal processing model; determine a description consistency loss according to a difference between a first text feature corresponding to the first description text and a second text feature corresponding to the second description text; The first loss is determined according to the reconstruction loss, the perceptual loss, the adversarial loss and the description consistency loss.

4. An image restoration method, characterized in that: The method comprises: Acquire an image to be restored; the image to be restored is a face image including an occluder; Performing feature extraction based on the image to be repaired to obtain image features of the image to be repaired; Determining, by means of an occlusion detection model, occlusion indication information for indicating the occlusion in the image to be repaired according to the image to be repaired; performing feature extraction based on the occlusion indication information to obtain occlusion features of the image to be repaired; Determining a restoration strategy parameter of an image restoration model according to the occlusion indication information; Through the image restoration model, the restoration strategy parameters are adopted to generate a restored image corresponding to the image to be restored according to the image features and the occlusion features; the image restoration model is trained by the model training method described in any one of claims 1 to 3, and the image quality of the restored image is higher than the image quality of the image to be restored.

5. The method according to claim 4, characterized in that The extracting features based on the image to be repaired to obtain image features of the image to be repaired includes: Performing feature extraction on the image to be repaired by using an image feature extraction model to obtain a first image feature of the image to be repaired; Generate a description text corresponding to the image to be repaired according to the image to be repaired through a multimodal processing model; extract features from the description text to obtain a second image feature of the image to be repaired; Determine the depth information corresponding to the image to be repaired by using a depth detection model; extract features from the depth information to obtain a third image feature of the image to be repaired; The various entity element regions included in the image to be repaired are determined by using an image segmentation model; and feature extraction is performed on each of the entity element regions to obtain a fourth image feature of the image to be repaired.

6. The method according to claim 5, characterized in that The method of generating a repaired image corresponding to the image to be repaired by using the image repair model and the repair strategy parameters according to the image features and the occlusion features includes: fusing the first image feature, the second image feature, the third image feature, the fourth image feature, and the occlusion feature to obtain a fused feature; The image restoration model is used to generate a restored image corresponding to the image to be restored according to the fusion features by adopting the restoration strategy parameters.

7. The method according to claim 6, characterized in that The fusing the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature to obtain a fused feature includes: generating the fused feature according to the first image feature, the second image feature, the third image feature, the fourth image feature and the occluder feature through a feature fusion network; or, Based on the attention mechanism, the attention weight of the combined feature composed of the first image feature, the second image feature, the third image feature, the fourth image feature and the occlusion feature is determined, and the combined feature is weighted according to the attention weight to obtain the fused feature.

8. The method according to claim 4, characterized in that Determining the restoration strategy parameters of the image restoration model according to the occlusion indication information includes: Determining a restoration parameter of the image restoration model according to the size of the occluder indicated by the occluder indication information; According to the occlusion indication information, the occlusion relationship between the occlusion and the face in the image to be repaired is determined, and the attention focus of the image repair model is determined according to the occlusion relationship to serve as a repair strategy parameter of the image repair model.

9. The method according to claim 1 or 4, characterized in that: The occlusion detection model is trained in the following way: Acquire a training image; the training image has a corresponding label, and the label is used to indicate the marked position and marking type of the occlusion in the training image; Determining, by means of the occlusion detection model to be trained and according to the training image, an occlusion detection result corresponding to the training image; Determine a second loss according to a difference between an occlusion detection result corresponding to the training image and a label corresponding to the training image; and train the occlusion detection model based on the second loss.

10. A model training device, characterized in that: The device comprises: A sample acquisition module, used to acquire training samples; the training samples include training images to be repaired and corresponding training images after repair, and the training images to be repaired are human face images including occluders; A first feature extraction module, used for performing feature extraction based on the training image to be repaired, to obtain image features of the training image to be repaired; A second feature extraction module is used to determine, through the occlusion detection model, occlusion indication information used to indicate the occlusion in the training image to be repaired according to the training image to be repaired; and perform feature extraction based on the occlusion indication information to obtain the occlusion feature of the training image to be repaired; A first strategy determination module, used to determine the restoration strategy parameters of the image restoration model to be trained according to the occlusion indication information; The model training module is used to adopt the restoration strategy parameters to train the image restoration model according to the image features, the occlusion features and the trained restoration image.

11. An image restoration device, characterized in that: The device comprises: An image acquisition module, used to acquire an image to be restored; the image to be restored is a face image including an occluder; A third feature extraction module, used to extract features based on the image to be repaired to obtain image features of the image to be repaired; A fourth feature extraction module is used to determine, based on the image to be repaired, occluder indication information for indicating the occluder in the image to be repaired by using an occluder detection model; and perform feature extraction based on the occluder indication information to obtain occluder features of the image to be repaired; A second strategy determination module, used to determine the restoration strategy parameters of the image restoration model according to the occlusion indication information; An image restoration module is used to generate a restored image corresponding to the image to be restored through the image restoration model and the restoration strategy parameters according to the image features and the occlusion features; the image restoration model is trained by the model training method described in any one of claims 1 to 3, and the image quality of the restored image is higher than the image quality of the image to be restored.

12. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The memory is used to store computer programs; The processor is used to execute the model training method described in any one of claims 1 to 3, or the image restoration method described in any one of claims 4 to 9 according to the computer program.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a computer device, it implements the model training method described in any one of claims 1 to 3, or the image restoration method described in claims 4 to 9.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method described in any one of claims 1 to 3 or the image restoration method described in claims 4 to 9 is implemented.

Images