Method, device, medium and program product for generating image restoration model
By acquiring the image to be repaired and the images they have been repaired, using image feature point synthesis and generative adversarial model training, the image repair model is generated, which solves the problem of manual intervention in image repair in the existing technology, realizes automatic repair and improves Image repair efficiency and quality.
Patent Information
- Application Number
- CN202110475219.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-04-29
AI Technical Summary
The prior art requires manual intervention in the image repair process, and the repair effect is not satisfactory, making it difficult to automatically and efficiently repair damaged or stained images.
By obtaining the image to be repaired and the images they have been repaired, using image feature point synthesis and generative adversarial models for training, an image repair model is generated, and automated image repair is achieved.
Automatic repair of images is realized, the efficiency and quality of image repair is improved, manual intervention is reduced, and noise, scratches and missing in images can be effectively repaired.
Smart Images

Figure CN113177892B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the computer field, specifically to artificial intelligence fields such as deep learning and computer vision, and more particularly to a method, device, medium and program product for generating an image restoration model. Background Art
[0002] In the era when digital cameras and digital storage devices were not popular, people would develop the photos after taking them to record the beautiful moments. However, due to the shortcomings of the photographic paper itself, scratches, fading, stains, etc. would easily appear during the preservation process, seriously affecting the visual quality of the photos.
[0003] Currently, the image to be restored is repaired manually using professional software to complete the image restoration. Summary of the invention
[0004] The embodiments of the present disclosure provide a method, device, medium and program product for generating an image restoration model.
[0005] In a first aspect, an embodiment of the present disclosure proposes a method for generating an image restoration model, comprising: acquiring a first image and a second image, wherein the second image is an image after the first image is restored; synthesizing an image corresponding to a feature point of the first image with the first image to obtain a synthesized image; and training using the second image and the synthesized image to obtain an image restoration model.
[0006] In the second aspect, an embodiment of the present disclosure proposes a device for generating an image restoration model, comprising: an image acquisition module, configured to acquire a first image and a second image, wherein the second image is an image after the first image is restored; an image synthesis module, configured to synthesize an image corresponding to a feature point of the first image with the first image to obtain a synthesized image; and a model training module, configured to perform training using the second image and the synthesized image to obtain an image restoration model.
[0007] On the third aspect, the embodiment of this announcement proposes an image restoration method, including: obtaining an image to be restored; inputting the image to be restored into a pre-trained image restoration model to obtain a restored image.
[0008] In a fourth aspect, an embodiment of the present announcement proposes an image restoration device, comprising: an image acquisition module, configured to acquire an image to be restored; and an image restoration module, configured to input the image to be restored into a pre-trained image restoration model to obtain a restored image.
[0009] In a fifth aspect, an embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect or the second aspect.
[0010] In a sixth aspect, an embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, where the computer instructions are used to cause a computer to execute the method described in the first aspect or the second aspect.
[0011] In a seventh aspect, an embodiment of the present disclosure proposes a computer program product, including a computer program, which implements the method described in the first aspect or the second aspect when executed by a processor.
[0012] The method, device, medium and program product for generating an image restoration model provided by the embodiments of the present disclosure first obtain a first image and a second image, wherein the second image is an image after restoration of the first image; then the image corresponding to the feature points of the first image is synthesized with the first image to obtain a synthesized image; finally, the second image and the synthesized image are used for training to obtain an image restoration model. The image restoration model can be obtained by synthesizing the synthesized image obtained by synthesizing the first image and the image corresponding to the feature points of the object in the first image, and the second image for model training, so as to realize image restoration.
[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Other features, purposes and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Among them:
[0015] Figure 1 is an exemplary system architecture diagram in which the present disclosure may be applied;
[0016] Figure 2 is a flowchart of an embodiment of a method for generating an image restoration model according to the present disclosure;
[0017] Figure 3 is a flowchart of another embodiment of a method for generating an image restoration model according to the present disclosure;
[0018] Figure 4is a flow chart of an embodiment of an image restoration method according to the present disclosure;
[0019] Figure 5 is an application scenario diagram of the image restoration method disclosed herein;
[0020] Figure 6 is a structural schematic diagram of an embodiment of a device for generating an image restoration model according to the present disclosure;
[0021] Figure 7 is a structural schematic diagram of an embodiment of an image restoration device according to the present disclosure;
[0022] Figure 8 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION
[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0025] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method for generating an image restoration model or an apparatus for generating an image restoration model of the present disclosure can be applied.
[0026] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0027] The user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send video frames, etc. The terminal devices 101, 102, 103 can be installed with various client applications, intelligent interactive applications, such as image processing applications, etc.
[0028] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, the terminal devices can be electronic products that interact with users through one or more methods such as keyboards, touch panels, touch screens, remote controls, voice interactions, or handwriting devices, such as PCs (Personal Computers), mobile phones, smart phones, PDAs (Personal Digital Assistants), wearable devices, PPCs (Pocket PCs), tablet computers, smart car machines, smart TVs, smart speakers, tablet computers, laptop computers, and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. It can be implemented as multiple software or software modules, or it can be implemented as a single software or software module. No specific limitation is made here.
[0029] The server 105 can provide various services. For example, the server 105 can analyze and process the video displayed on the terminal devices 101, 102, and 103, and generate processing results.
[0030] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.
[0031] It should be noted that the method for generating an image restoration model provided in the embodiments of the present disclosure is generally executed by the server 105 , and accordingly, the device for generating an image restoration model is generally disposed in the server 105 .
[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0033] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a method for generating an image restoration model according to the present disclosure. The method for generating an image restoration model may include the following steps:
[0034] Step 201, obtaining a first image and a second image, wherein the second image is an image after the first image is restored.
[0035] In this embodiment, the execution subject (eg Figure 1The terminal devices 101, 102, 103 shown in the figure can obtain the first image and the second image through a shooting device, and the shooting device can be a camera of the terminal device or a camera outside the terminal device; or, the execution subject of the method for generating an image restoration model (for example Figure 1 The server 105 shown in FIG. 105 ) receives data from a terminal device (eg Figure 1 The first image and the second image are obtained from the terminal devices 101, 102, and 103 shown in the figure. The first image may be an image to be repaired or one or more frames of images to be repaired in a video stream. The first image may include one or more areas to be repaired, and the second image may be an image obtained by repairing the area to be repaired in the first image.
[0036] In this embodiment, acquiring the first image and the second image may include: acquiring the second image; and generating the first image according to the second image.
[0037] Wherein, generating the first image according to the second image may include:
[0038] (1) The second image is damaged by using a preset mask image to generate a first image, wherein the preset mask image may be various randomly generated noises.
[0039] In one example, a mask of the same size as the first image is applied to the second image to obtain the first image.
[0040] (2) The second image is multiplied by a binary mask to obtain the first image.
[0041] (3) The first image may be an image obtained by adding noise to the second image.
[0042] In this embodiment, when the number of first images is small, the first image can be obtained by processing the second image to increase the training samples of the training image restoration model, thereby improving the image restoration accuracy of the image restoration model.
[0043] In this embodiment, after obtaining the first image, the method for generating an image restoration model may further include: determining a to-be-restored area of the first image.
[0044] Correspondingly, in this example, determining the area to be repaired in the first image may include: using a model to identify the first image to determine the area to be repaired in the first image; or determining the area to be repaired in the first image by manual annotation.
[0045] Among them, the model mainly uses artificial intelligence (AI), that is, a neural network model. The neural network model can specifically identify the area to be repaired in the first image based on algorithms such as target detection algorithms, such as R-FCN, Faster R-CNN, SSD, YOLO V3 and other algorithms. These neural network models can be obtained by marking the area to be repaired in the first image and training the initial neural network model.
[0046] Here, the second image may be a restored image of the first image.
[0047] It should be noted that image restoration refers to repairing and reconstructing a damaged image or removing redundant objects in the image.
[0048] The image restoration technology in the embodiments of the present disclosure is a type of image processing technology. The image restoration technology aims to restore the lost or blocked parts of an image according to the image context. The image restoration task requires that the restored image as a whole is as natural as possible and as close to the original image as possible. Through the image restoration technology, some noise, scratches, missing parts, and occlusions in the image can be removed to improve the image quality.
[0049] Step 202 : synthesize the image corresponding to the feature points of the first image with the first image to obtain a synthesized image.
[0050] In this embodiment, the execution entity may synthesize the image corresponding to the feature point of the first image with the first image to obtain a synthesized image.
[0051] Specifically, the first image may be firstly subjected to target detection; then, the object in the first image may be determined; then, feature point detection may be performed on the object in the first image to obtain feature points of the object; then, the feature points may be segmented from the first image to obtain an image corresponding to the feature points; then, the image corresponding to the feature points may be synthesized with the first image to obtain a synthesized image; for example, the number of channels of the image corresponding to the feature points may be synthesized with the number of channels of the first image to obtain a synthesized image; or, the target feature points in the image corresponding to the specific points may be spliced with the target feature points in the first image, wherein the target feature points in the image corresponding to the feature points have the same position as the target feature points in the first image. The above feature points may be used to characterize the features of the object, and the target feature points may be one or more features among all the features characterizing the object. The above object may be a target in the first image, such as a face, a car, a background, text, and the like.
[0052] In a specific example, the first image may be an image containing a face; target detection is performed on the first image; then, the category of the object in the first image is determined to be a face, and the position of the face in the first image is determined; then, key point detection is performed on the face in the first image to obtain key points of the face, such as facial features (i.e., eyes, eyebrows, mouth, nose, etc.), contours, etc.; then, the key points of the face in the first image are segmented to obtain an image corresponding to the key points of the face; then, the image corresponding to the key points of the face is synthesized with the feature points at the same position in the first image to obtain a synthesized image, for example, the left eye (i.e., the image corresponding to the key points of the face) is spliced with the left eye in the first image.
[0053] Correspondingly, in this example, performing target detection on the first image may include: performing target detection on the first image using an image recognition model to obtain the category of the target object and the position of the target object in the first image. The above-mentioned image recognition model can take a sample image in a training sample in a training sample set as input, and a label corresponding to the input sample image (for example, the position of the object in the sample image in the sample image, and the category label of the object) as output, and train a neural network to obtain a target recognition model. Among them, the target recognition model can be used to determine the position and / or category of the object in the first image.
[0054] After determining the area to be repaired in the first image in step 201, synthesizing the image corresponding to the feature points of the first image with the first image to obtain a synthesized image may include: synthesizing the image corresponding to the feature points of the target area to be repaired in the first image with the first image to obtain a synthesized image.
[0055] Step 203: training using the second image and the synthesized image to obtain an image restoration model.
[0056] In this embodiment, the above-mentioned execution subject can use the second image and the synthesized image for training to obtain an image restoration model.
[0057] Specifically, the above-mentioned execution entity can use the synthesized image as the input of the image restoration model, use the second image as the output of the image restoration model, train the initial model, and obtain the image restoration model.
[0058] In this embodiment, after obtaining the synthetic image and the second image, the execution subject can use the synthetic image and the second image to train the initial model to obtain the image restoration model. During training, the execution subject can use the synthetic image as the input of the image restoration model, and use the second image corresponding to the input as the expected output to obtain the image restoration model. The initial model can be a neural network model in the existing technology or future development technology. For example, the neural network model can include any of the following: Generative Adversarial Networks (GAN), Cycle GAN, Pix2pixGAN, Dual GAN, DiscoGAN, and DCGAN. Among them, GAN can include a generator and a discriminator. The discriminator is used to distinguish between the first image and the second image. Under the supervision of the discriminator, the generator will try its best to generate results close to the real photo to confuse the discriminator and reduce the loss, so that we can get a model that can automatically repair the first image (i.e., the image with defective areas).
[0059] It should be noted that the generator may be a convolutional neural network (e.g., various convolutional neural network structures including convolutional layers, pooling layers, depooling layers, and deconvolutional layers, which may be downsampled and upsampled in sequence); the discriminator may also be a convolutional neural network (e.g., various convolutional neural network structures including fully connected layers, wherein the fully connected layers may implement classification functions). In addition, the discriminator may also be other model structures that may be used to implement classification functions, such as a support vector machine (SVM).
[0060] The method for generating an image restoration model provided by the embodiment of the present disclosure first obtains a first image and a second image, wherein the second image is an image after restoration of the first image; then an image corresponding to a feature point of the first image is synthesized with the first image to obtain a synthesized image; finally, the second image and the synthesized image are used for training to obtain an image restoration model. The image restoration model can be obtained by performing model training with the second image using a synthesized image obtained by synthesizing the first image and an image corresponding to a feature point of an object in the first image, thereby enabling restoration of the image.
[0061] In some optional implementations of this embodiment, the image corresponding to the feature points of the first image is synthesized with the first image to obtain a synthesized image, including: synthesizing the number of channels of the image corresponding to the feature points of the first image with the number of channels of the first image to obtain a synthesized image.
[0062] In this implementation, the execution subject may obtain a composite image according to the sum of the number of channels of the image corresponding to the feature points of the first image and the number of channels of the first image.
[0063] In this implementation, synthesis may be performed based on the number of channels of the image corresponding to the feature point and the number of channels of the first image to obtain a synthesized image.
[0064] In some optional implementations of this embodiment, if the feature points of the first image may include feature points of the first target area to be repaired in the first image.
[0065] In this implementation, after obtaining the first target area to be repaired of the first image, the method for generating an image repair model may further include:
[0066] The number of channels of the image corresponding to the feature points of the first target area to be repaired in the first image is synthesized with the number of channels of the first image to obtain a synthesized image. The first target area to be repaired may be one or more areas to be repaired in the first image.
[0067] It should be noted that the feature points of the first target area to be repaired may be all feature points of the first target area to be repaired; the feature points of the first target area to be repaired may also be relatively key feature points in the first target area to be repaired, such as facial features, facial contours, etc.
[0068] In this implementation, image synthesis can be performed on the feature points of the first target area to be repaired and the first image. While obtaining the synthesized image, the resource consumption caused by synthesizing other feature points (for example, features other than the feature points of the first target area to be repaired) can be reduced.
[0069] In some optional implementations of this embodiment, the image restoration model is a generative adversarial model, wherein the generative adversarial model may include a discriminator and a generator.
[0070] In this implementation, the generative adversarial model may include a generator G and a discriminator D. The generator G may be used to adjust the resolution of an input image (e.g., a synthetic image) and output the adjusted image, and the discriminator D may be used to determine whether the input image is the image output by the generator G. The generative adversarial model trains the generator G and the discriminator D simultaneously through a continuous adversarial process. The training process is a cross-optimization process of the generator G and the discriminator D. The generator G is trained to generate a fake image to deceive the discriminator D, and the discriminator D is trained to distinguish between a real image and a fake image generated by the generator G. The generator G is used to generate an initial repair image based on the synthetic image; then, the discriminator D determines whether the initial repair image is consistent with the real image (the repaired image, i.e., the second image); if not, the parameters of the generative adversarial model are continuously adjusted until the initial repair image is consistent with the real image, and then the adjustment of the model parameters is stopped, and the final model is determined as the image repair model.
[0071] In this implementation, image restoration can be achieved based on a generative adversarial model including a discriminator and a generator.
[0072] Further references Figure 3 , Figure 3 A process 300 of another embodiment of a method for generating an image restoration model according to the present disclosure is shown. The method for generating an image restoration model may include the following steps:
[0073] Step 301: Acquire a first image and a second image, wherein the second image is an image after the first image is restored.
[0074] Step 302: synthesize the number of channels of the image corresponding to the feature points of the first image with the number of channels of the first image to obtain a synthesized image.
[0075] In this embodiment, the execution subject (eg Figure 1 The terminal device 101, 102, 103 or the server 105 shown in the figure can synthesize the number of channels of the image corresponding to the feature point of the first image with the number of channels of the first image to obtain a synthesized image, and the number of channels of the synthesized image is the sum of the number of channels of the image corresponding to the feature point and the number of channels of the first image. The above number of channels can be used to characterize the features of multiple dimensions of the image, and the number of channels of the first image can be obtained together when the first image is obtained.
[0076] Step 303: Use the second image and the synthesized image for training to obtain an image restoration model.
[0077] In this embodiment, the specific operations of steps 301 and 303 are respectively Figure 2In the illustrated embodiment, steps 201 and 203 are described in detail and will not be repeated here.
[0078] from Figure 3 It can be seen that Figure 2 Compared with the corresponding embodiment, the method for generating an image restoration model in this embodiment highlights the step of synthesizing an image. Therefore, the solution described in this embodiment synthesizes the number of channels of the image corresponding to the feature points of the first image with the number of channels of the first image to obtain a synthesized image.
[0079] Further references Figure 4 , Figure 4 A process 400 of an embodiment of an image restoration method according to the present disclosure is shown. The image restoration method may include the following steps:
[0080] Step 401: Acquire an image to be restored.
[0081] In this embodiment, the execution subject of the image restoration method may be the same as or different from the execution subject of the method for generating the image restoration model. If they are the same, the execution subject of the method for generating the image restoration model may store the model structure information of the trained image restoration model and the parameter values of the model parameters locally after the image restoration model is trained. If they are different, the execution subject of the method for generating the image restoration model may send the model structure information of the trained image restoration model and the parameter values of the model parameters to the execution subject of the image restoration method after the image restoration model is trained.
[0082] In this embodiment, the execution subject of the image restoration method can obtain the image to be restored in a variety of ways. For example, the image to be restored can be obtained through a terminal device (such as Figure 1 The terminal devices 101, 102, and 103 shown in the figure are used to obtain the image to be repaired. The above-mentioned image to be repaired may be an image with a region to be repaired.
[0083] Step 402: input the image to be repaired into a pre-trained image repair model to obtain a repaired image.
[0084] In this embodiment, the execution subject may input the image to be repaired into a pre-trained image repair model to obtain a repaired image. The image repair model may be a model trained by a method for generating an image repair model, for example Figure 2 and Figure 3 The model obtained by training the corresponding embodiment.
[0085] The method provided in the embodiment of the present disclosure can realize the restoration of the image to be restored based on a pre-trained image restoration model.
[0086] In some optional implementations of this embodiment, before executing step 402, the image restoration method may further include: determining a second target region to be restored of the image to be restored; and segmenting an image corresponding to the second target region to be restored from the image to be restored.
[0087] It should be noted that the description of determining the second target region to be repaired in the image to be repaired may refer to the description of determining the region to be repaired in the first image. The second target region to be repaired may be one or more regions to be repaired in the image to be repaired.
[0088] After the second target area to be repaired is determined, step 402 may include: inputting an image corresponding to the second target area to be repaired into a pre-trained image repair model to obtain a repaired image.
[0089] In this implementation, the second target area to be repaired in the image to be repaired can be repaired to reduce the repair operation on the entire image to be repaired, thereby improving the efficiency of image repair.
[0090] In some optional implementations of this embodiment, if the image to be repaired is a face image, after obtaining the repaired image, the image repair method may further include: identifying the repaired image to obtain a recognition result; and performing identity authentication based on the recognition result.
[0091] In this implementation, face recognition can be performed on the repaired image to obtain a face recognition result; then, based on the face recognition result, the face recognition result is matched with the standard image to perform identity authentication; if the face recognition result matches the standard image, the identity authentication is determined to be successful; if the face recognition result does not match the standard image, the identity authentication is confirmed. The standard image can be an image uploaded by the user in advance, and the standard image can be used to accurately determine whether the user is a legitimate user.
[0092] It should be noted that when the user performs identity authentication, since the user is in a situation where it is inconvenient to take pictures (for example, in a fast-moving vehicle), the user may capture a not very clear image (that is, the image to be repaired) through the terminal device. At this time, the user needs to perform identity authentication, and the captured image can be repaired through the image repair model; after obtaining the repaired image, identity authentication is performed based on the repaired image, so as to realize identity authentication in a scenario where it is inconvenient to take pictures.
[0093] In this implementation, after the user is authenticated, subsequent operations related to the information of the repaired image may be performed based on the repaired image, for example, recommendations based on the information of the repaired image (for example, image search scenarios) and resource transfers based on the information of the repaired image.
[0094] In a specific example, a facial image of a person to be transferred and a facial image preset for the account to be transferred (i.e., a standard image) are obtained; the facial image of the person to be transferred is input into an image repair model, and the facial image of the person to be transferred is repaired by the image repair model to obtain a repaired facial image; facial recognition is performed on the repaired facial image to obtain an identity recognition result of the facial image; if the identity recognition result indicates that the repaired facial image matches the facial image preset for the account to be transferred, the resource transfer is performed.
[0095] It should be noted that resource transfer may refer to a change in the ownership of a resource; for example, a resource is transferred from location A (or device A, or user A) to location B (or device B, or user B).
[0096] In this implementation, after the image to be repaired is repaired by using the image repair model to obtain the repaired image, the repaired image can be recognized to perform identity authentication based on the recognition result.
[0097] For ease of understanding, the following provides application scenarios that can implement the image restoration method of the embodiment of the present application. Figure 5 As shown, taking a face image as an example, the terminal device 501 (eg Figure 1 Taking the terminal devices 101, 102, and 103 shown in the figure as an example, the terminal device first obtains a first image 51; then, performs key point detection 52 on the first image to obtain key points (i.e., mask) 53 of the first image; then, the number of channels of the image corresponding to the key points of the first image and the number of channels of the first image are input into a pre-trained image restoration model 54 to obtain a restoration result 55 (e.g., a second image).
[0098] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for generating an image restoration model. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0099] like Figure 6 As shown, the apparatus 600 for generating an image restoration model in this embodiment may include: an image acquisition module 601, an image synthesis module 602, and a model training module 603. The image acquisition module 601 is configured to acquire a first image and a second image, wherein the second image is an image after restoration of the first image; the image synthesis module 602 is configured to synthesize an image corresponding to a feature point of the first image with the first image to obtain a synthesized image; and the model training module 603 is configured to perform training using the second image and the synthesized image to obtain an image restoration model.
[0100] In the apparatus 600 for generating an image restoration model, the specific processing of the image acquisition module 601, the image synthesis module 602, and the model training module 603 and the technical effects thereof can be referred to in the respective Figure 2 The relevant descriptions of steps 201 - 203 in the corresponding embodiment are not repeated here.
[0101] In some optional implementations of this embodiment, the image synthesis module 602 is further configured to: synthesize the number of channels of the image corresponding to the feature points of the first image with the number of channels of the first image to obtain a synthesized image.
[0102] In some optional implementations of this embodiment, the feature points of the first image are feature points of the first target area to be repaired in the first image.
[0103] In some optional implementations of this embodiment, the image restoration model is a generative adversarial model.
[0104] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image restoration device. Figure 4 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0105] like Figure 7 As shown, the image restoration device 700 of this embodiment may include: an image acquisition module 701 and an image restoration module 702. The image acquisition module 701 is configured to acquire an image to be restored; the image restoration module 702 is configured to input the image to be restored into a pre-trained image restoration model to obtain a restored image.
[0106] In this embodiment, in the image restoration device 700, the specific processing of the image acquisition module 701 and the image restoration module 702 and the technical effects thereof can be referred to in Figure 4 The relevant descriptions of steps 401 - 402 in the corresponding embodiment are not repeated here.
[0107] In some optional implementations of this embodiment, the image restoration device also includes: a region determination module (not shown in the figure), configured to determine a second target region to be restored in the image to be restored; the image restoration module 702 is further configured to: input the image corresponding to the second target region to be restored into a pre-trained image restoration model to obtain a restored image.
[0108] In some optional implementations of this embodiment, if the image to be repaired is a facial image to be repaired, the image repair device also includes: an image recognition module (not shown in the figure), configured to identify the repaired image and obtain a recognition result; an identity authentication module (not shown in the figure), configured to perform identity authentication based on the recognition result.
[0109] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0110] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0111] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0112] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0113] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a method for generating an image repair model or an image repair method. For example, in some embodiments, the method for generating an image repair model or an image repair method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the method for generating an image repair model or an image repair method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the method for generating an image restoration model or the image restoration method in any other appropriate manner (for example, by means of firmware).
[0114] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0115] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0116] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0118] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0119] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0120] Artificial intelligence is a discipline that studies computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural speech processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.
[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions mentioned in this disclosure can be achieved, and this document does not limit this.
[0122] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating an image restoration model, comprising: Acquire a first image and a second image, wherein the first image is an image to be repaired, the first image includes a first area and at least one area to be repaired, the first area is an area outside the at least one area to be repaired in the first image, and the second image is an image obtained by repairing the area to be repaired in the first image; Synthesizing the image corresponding to the feature points of the first image with the first image to obtain a synthesized image, including: splicing target feature points in the image corresponding to feature points of the first target area to be repaired in the first image with target feature points in the first image to obtain a synthesized image, wherein the target feature points in the image corresponding to the feature points of the first image have the same position as the target feature points in the first image, and the first target area to be repaired is one or more areas to be repaired in the first image; The image restoration model is obtained by training with the second image and the composite image, including: a generator of a generative adversarial model generates an initial restoration image based on the composite image; a discriminator of the generative adversarial model determines whether the initial restoration image is consistent with the second image; if not, continuing to adjust the parameters of the generative adversarial model until the initial restoration image is consistent with the second image, then stopping the adjustment of the model parameters and determining the final model as the image restoration model.
2. The method according to claim 1, wherein: The step of synthesizing the image corresponding to the feature points of the first image with the first image to obtain a synthesized image includes: The number of channels of the image corresponding to the feature points of the first image is synthesized with the number of channels of the first image to obtain a synthesized image.
3. An image restoration method, comprising: Obtain the image to be repaired; The image to be repaired is input into the image repair model as described in claim 1 or 2 to obtain a repaired image.
4. The method according to claim 3, further comprising: Determining a second target to-be-repaired area in the to-be-repaired image; The step of inputting the image to be repaired into the image repair model as claimed in claim 1 or 2 to obtain a repaired image comprises: The image corresponding to the second target area to be repaired is input into the image repair model as described in claim 1 or 2 to obtain a repaired image.
5. The method according to claim 3 or 4, wherein: If the image to be restored is a face image to be restored, the method further includes: Recognize the repaired image to obtain a recognition result; According to the recognition result, identity authentication is performed.
6. A device for generating an image restoration model, comprising: an image acquisition module, configured to acquire a first image and a second image, wherein the first image is an image to be repaired, the first image includes a first area and at least one area to be repaired, the first area is an area outside the at least one area to be repaired in the first image, and the second image is an image obtained by repairing the area to be repaired in the first image; an image synthesis module, configured to synthesize an image corresponding to a feature point of the first image with the first image to obtain a synthesized image, and further configured to: splice target feature points in the image corresponding to feature points of a first target area to be repaired in the first image with target feature points in the first image to obtain a synthesized image, wherein the target feature points in the image corresponding to the feature points of the first image have the same position as the target feature points in the first image, and the first target area to be repaired is one or more areas to be repaired in the first image; The model training module is configured to use the second image and the composite image for training to obtain an image restoration model, and is further configured to generate an initial restoration image based on the composite image by a generator of a generative adversarial model; determine whether the initial restoration image is consistent with the second image by a discriminator of the generative adversarial model; if not, continue to adjust the parameters of the generative adversarial model until the initial restoration image is consistent with the second image, then stop adjusting the model parameters and determine the final model as the image restoration model.
7. The device according to claim 6, wherein: The image synthesis module is further configured to: The number of channels of the image corresponding to the feature points of the first image is synthesized with the number of channels of the first image to obtain a synthesized image.
8. An image restoration device, comprising: An image acquisition module is configured to acquire an image to be restored; The image restoration module is configured to input the image to be restored into the image restoration model as described in claim 1 or 2 to obtain a restored image.
9. The device according to claim 8, further comprising: A region determination module is configured to determine a second target to-be-repaired region in the to-be-repaired image; The image restoration module is further configured to: The image corresponding to the second target area to be repaired is input into the image repair model as described in claim 1 or 2 to obtain a repaired image.
10. The device according to claim 8 or 9, wherein: If the image to be restored is a face image to be restored, the device further includes: An image recognition module is configured to recognize the repaired image and obtain a recognition result; The identity authentication module is configured to perform identity authentication according to the recognition result.
11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.
12. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.
