Image restoration method, apparatus and electronic device

Abstract

This invention provides an image restoration method, apparatus, and electronic device, applicable to the field of image processing technology. The method includes: determining the location and object labels of different objects in an image to be restored; using a pre-set image restoration method for each object label, restoring the image regions corresponding to the locations of each object label in the image to be restored, thereby obtaining a restored image; wherein, the location corresponding to each object label is the location of the object to which that label belongs in the image to be restored. This solution can improve the restoration quality of image restoration.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image restoration method, apparatus, and electronic device. Background Technology

[0002] Image restoration refers to the process of restoring or reconstructing damaged, missing, or noisy digital images through a series of processing methods to improve their quality and information usability. Image restoration can be applied to various fields, such as digital images, remote sensing images, medical images, etc., to improve the display effect and accuracy of images.

[0003] In related technologies, most of them use a single image restoration algorithm to process the entire image. For example, they use an image restoration model to restore the entire image. However, an image often contains different types of objects. Using a single image restoration algorithm to process the entire image often lacks targeted restoration, resulting in poor restoration quality. Summary of the Invention

[0004] The purpose of this invention is to provide an image restoration method, apparatus, and electronic device to improve the restoration quality of images. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of the present invention provide an image restoration method, the method comprising:

[0006] Determine the location and label of different objects in the image to be repaired;

[0007] Using a pre-set image restoration method for each object label, the image regions corresponding to each object label in the image to be restored are restored to obtain the restored image;

[0008] Among them, the region location corresponding to each object label is the region location of the object to which the object label belongs in the image to be repaired.

[0009] Optionally, the step of using a pre-set image restoration method for each object label to restore the image region corresponding to each object label in the image to be restored, thereby obtaining the restored image, includes:

[0010] Feature extraction is performed on the image to be repaired to obtain shallow image features of the image to be repaired;

[0011] From the shallow image features, determine the region features located at the region position corresponding to each object label, and use them as the region features corresponding to each object label.

[0012] Based on the deep feature extraction method pre-set for each object label, deep feature extraction is performed on the region features corresponding to each object label to obtain the depth image features corresponding to that object label.

[0013] Feature reconstruction is performed on the depth image features corresponding to each object label to obtain the repaired image.

[0014] Optionally, the step of performing deep feature extraction on the region features corresponding to each object label based on a pre-set deep feature extraction method for each object label to obtain the depth image features corresponding to that object label includes:

[0015] Determine the deep feature extraction parameters corresponding to each object label;

[0016] For each object label, based on the deep feature extraction parameters corresponding to that object label, deep feature extraction is performed on the region features corresponding to that object label to obtain the depth image features corresponding to that object label.

[0017] Optionally, determining the deep feature extraction parameters corresponding to each object label includes:

[0018] Based on the pre-defined correspondence between object labels and deep feature extraction parameters, deep feature extraction parameters corresponding to each object label in the image to be repaired are selected from the pre-defined deep feature extraction parameters.

[0019] And / or,

[0020] For each object label, feature extraction is performed on the image region corresponding to the object label in the image to be repaired to obtain the image features of the region corresponding to the object label; based on the image features of the region corresponding to the object label, the deep feature extraction parameters corresponding to the object label are determined.

[0021] Optionally, the step of extracting features from the image region corresponding to the object label in the image to be repaired, to obtain the image features of the region corresponding to the object label, includes:

[0022] Using a feature recognition network, feature extraction is performed on the image region at the location of the object label in the image to be repaired, and the image features of the region corresponding to the object label are obtained.

[0023] Wherein, the network structure of the feature recognition network is the network structure of the network used to extract features in the first object recognition model; the first object recognition model is a neural network model used to perform object recognition.

[0024] Optionally, determining the deep feature extraction parameters corresponding to the object label based on the region image features corresponding to the object label includes:

[0025] The image features of the region corresponding to the object label are input into a pre-defined fully connected layer corresponding to the object label to obtain the deep feature extraction parameters corresponding to the object label.

[0026] Optionally, the deep feature extraction parameters are network parameter values ​​of specified network parameters in the deep feature extraction network;

[0027] For each object label, based on the deep feature extraction parameters corresponding to that object label, deep feature extraction is performed on the region features corresponding to that object label to obtain the depth image features corresponding to that object label, including:

[0028] For each object label, the network parameter value corresponding to the object label is loaded into the specified network parameter in the deep feature extraction network; using the deep feature extraction network with the changed network parameter value, the region feature corresponding to the object label is extracted to obtain the depth image feature corresponding to the object label.

[0029] Optionally, the specified network parameters may be all the network parameters included in the deep feature extraction network, or the network parameters of a specified convolutional layer in the deep feature extraction network.

[0030] Optionally, determining the location and object label of different objects in the image to be repaired includes:

[0031] Perform semantic segmentation on the image to be repaired to obtain the region locations and object labels of different objects in the image; and / or,

[0032] The image to be repaired is segmented to determine the location of different objects in the image; object recognition is performed on the image regions at each location in the image to be repaired to obtain object labels corresponding to each location; and / or,

[0033] Key point identification is performed on the image to be repaired to obtain each key point in the image to be repaired; based on each key point, the regional location and object label of different objects in the image to be repaired are determined.

[0034] Optionally, the method is applied to an image restoration model;

[0035] The training methods for the image restoration model include:

[0036] Obtain the sample image and the corresponding ground truth image;

[0037] The sample image is input into the image restoration model to be trained, so that the image restoration model to be trained can determine the location of different objects in the sample image and the object label; the image restoration method to be trained is set for each object label, and the image region at the location corresponding to each object label in the sample image is restored to obtain the restored image;

[0038] Based on the restored image and the ground truth image, determine the loss value of the image restoration model to be trained;

[0039] Based on the obtained loss value, the model parameters in the image restoration model to be trained are adjusted.

[0040] Optionally, determining the loss value of the image inpainting model to be trained based on the restored image and the ground truth image includes:

[0041] Calculate the difference in pixel values ​​between the restored image and the ground truth image, using this difference as the first loss value; and / or,

[0042] The restored image and the ground truth image are respectively input into a second object recognition model; wherein, the second object recognition model is a pre-trained neural network model containing multiple sequentially connected feature extraction layers; the image features of the restored image and the image features of the ground truth image output by at least one feature extraction layer are respectively obtained; the feature difference between the image features of the restored image and the image features of the ground truth image is calculated as a second loss value; and / or,

[0043] The repaired image is input into a preset image discriminator to obtain a first evaluation score that the repaired image is a real image, as output by the image discriminator; the first difference between the first evaluation score and the first preset score value is calculated as a third loss value; wherein, the first preset score value is a score value that characterizes the real image;

[0044] Based on the first loss value, the second loss value, and / or the third loss value, the loss value of the image inpainting model to be trained is determined.

[0045] Optionally, before obtaining the image features of the repaired image and the image features of the ground truth image output by at least one of the plurality of feature extraction layers, the method further includes:

[0046] The location of different objects in the sample image and their labels are input into the second object recognition model;

[0047] The step of obtaining the image features of the repaired image and the image features of the ground truth image output by at least one feature extraction layer among the plurality of feature extraction layers includes:

[0048] For each input object label, the region features of the repaired image corresponding to the object label and the region features of the ground truth image corresponding to the object label are obtained respectively; wherein, the region features of any image corresponding to each object label are: the region features output by at least one feature extraction layer among the plurality of feature extraction layers for the image region at the region position corresponding to the object label in the image;

[0049] The calculation of the feature difference between the image features of the restored image and the image features of the ground truth image includes:

[0050] For each input object label, calculate the feature difference between the region features of the repaired image corresponding to the object label and the region features of the ground truth image corresponding to the object label, and use the feature difference size corresponding to the object label as the feature difference size.

[0051] Based on the feature difference magnitude corresponding to each input object label, the feature difference magnitude between the image features of the repaired image and the image features of the ground truth image is determined.

[0052] Optionally, the step of inputting the restored image into a preset image discriminator to obtain a first evaluation score output by the image discriminator indicating that the restored image is a real image includes:

[0053] The regions and object labels of different objects in the restored image and the sample image are input into a preset image discriminator to obtain the region evaluation score of the image region at the region location corresponding to each input object label in the restored image, output by the image discriminator.

[0054] Based on the obtained evaluation scores for each region, the first evaluation score is used to determine whether the restored image is a true image.

[0055] Optionally, after inputting the restored image into a preset image discriminator and obtaining a first evaluation score from the image discriminator indicating that the restored image is a real image, the method further includes:

[0056] The ground truth image is input into the image discriminator to obtain a second evaluation score predicted by the image discriminator as a real image;

[0057] Calculate a second difference between the second evaluation score and the first preset score, and calculate a third difference between the first evaluation score and the second preset score; wherein the second preset score is a score that characterizes a fake image;

[0058] The parameters of the image discriminator are updated based on the second difference and the third difference.

[0059] Secondly, embodiments of the present invention provide an image restoration apparatus, the apparatus comprising:

[0060] The information determination module is used to determine the location and object label of different objects in the image to be repaired;

[0061] The image restoration module is used to restore the image regions corresponding to each object label in the image to be restored by using a pre-set image restoration method for each object label, so as to obtain the restored image; wherein, the region corresponding to each object label is the region location of the object to which the object label belongs in the image to be restored.

[0062] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0063] Memory, used to store computer programs;

[0064] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0065] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0066] Fifthly, embodiments of the present invention provide a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the methods described in the first aspect.

[0067] Beneficial effects of the embodiments of the present invention:

[0068] In the image restoration method provided by this invention, the location and object labels of different objects in the image to be restored can be determined. Then, an image restoration method pre-set for each object label is used to restore the image regions corresponding to the locations of each object label in the image to be restored, resulting in a restored image. The location of each object label refers to the location of the object to which that label belongs in the image to be restored. Because an image restoration method pre-set for each object label is used to restore the image regions corresponding to the locations of each object label in the image to be restored, this means that the image regions containing different objects in the image to be restored are restored using image restoration methods corresponding to each object. This allows each image restoration method to more specifically restore local areas of the image to be restored, thereby improving the restoration quality.

[0069] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0071] Figure 1 A flowchart of the first image restoration method provided in the embodiments of the present invention;

[0072] Figure 2a A schematic diagram provided for an embodiment of the present invention;

[0073] Figure 2b This is an image illustration of a determined region location and object label provided in an embodiment of the present invention;

[0074] Figure 3 This is a schematic diagram of the first image restoration model provided in an embodiment of the present invention;

[0075] Figure 4 This is a flowchart of the second image restoration method provided in the embodiments of the present invention;

[0076] Figure 5 This is a schematic diagram of the second image restoration model provided in an embodiment of the present invention;

[0077] Figure 6 This is a schematic diagram of the third image restoration model provided in an embodiment of the present invention;

[0078] Figure 7 This is a schematic diagram of the fourth image restoration model provided in an embodiment of the present invention;

[0079] Figure 8 This is a schematic diagram of the fifth image restoration model provided in an embodiment of the present invention;

[0080] Figure 9 This is a schematic diagram of the sixth image restoration model provided in an embodiment of the present invention;

[0081] Figure 10 This is a flowchart of the third image restoration method provided in the embodiments of the present invention;

[0082] Figure 11 This is a schematic diagram illustrating feature extraction for an object recognition model provided in an embodiment of the present invention;

[0083] Figure 12 This is a schematic diagram of the training framework of the first image restoration model provided in an embodiment of the present invention;

[0084] Figure 13 This is a schematic diagram of the training framework of the second image restoration model provided in an embodiment of the present invention;

[0085] Figure 14 This is a schematic diagram of the training framework of the third image restoration model provided in the embodiments of the present invention;

[0086] Figure 15 This is a schematic diagram of the training framework of the fourth image restoration model provided in the embodiments of the present invention;

[0087] Figure 16 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of the present invention;

[0088] Figure 17 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0089] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0090] To improve the quality of image restoration, embodiments of the present invention provide an image restoration method, apparatus, and electronic device.

[0091] It should be noted that, in specific applications, the embodiments of the present invention can be applied to various electronic devices, such as personal computers, servers, mobile phones, and other devices with data processing capabilities. Furthermore, the image restoration method provided by the embodiments of the present invention can be implemented through software, hardware, or a combination of both.

[0092] In one embodiment, the present invention can be applied to a repair device, repair terminal, or repair platform for image quality restoration of images, videos, etc. Specifically, it can be deployed offline or online to repair images that need image restoration.

[0093] The image restoration method provided in this embodiment of the invention may include:

[0094] Determine the location and label of different objects in the image to be repaired;

[0095] Using a pre-defined image restoration method for each object label, the image regions corresponding to each object label in the image to be restored are restored to obtain the restored image.

[0096] Among them, the region location corresponding to each object label is the region location of the object to which the object label belongs in the image to be repaired.

[0097] In the above-described scheme of the present invention, since the image restoration method is pre-set for each object label, the image regions corresponding to each object label in the image to be restored are restored respectively. This means that in this scheme, the image regions where different objects are located in the image to be restored are restored using the image restoration method corresponding to each object. This allows each image restoration method to restore the local image of the image to be restored in a more targeted manner, thereby improving the restoration quality of the image restoration.

[0098] The image restoration method provided by the embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0099] like Figure 1 As shown, this embodiment of the invention provides an image restoration method, including steps S101-S102, wherein:

[0100] S101, Determine the location and object label of different objects in the image to be repaired;

[0101] In this embodiment of the invention, image restoration is used to achieve functions such as improving image quality, filling in missing parts, and repairing cracks. Improving image quality includes operations such as noise removal; filling in missing parts refers to repairing missing or damaged portions of an image; and repairing cracks refers to repairing linear or crack-like defects in an image caused by human or natural factors.

[0102] In this step, the image to be repaired can be an image or video frame. Optionally, the image to be repaired can be a portrait image, a face image, an oil painting image, a multi-target image, a medical image, etc. Optionally, the image to be repaired can be a picture / video captured by a terminal, which can include mobile phones, tablets, small cameras, large camcorders, etc. It is also applicable to surveillance videos, such as traffic monitoring and indoor venue monitoring screen processing. The content of the image to be repaired can include movie images, TV series images, variety show images, documentary images, sports event images, surveillance screens, animations, artworks, etc.

[0103] Before performing this step, the image to be repaired can be identified. Optionally, in one approach, an image or video uploaded by a user can be received, and at least one frame from the received image or video can be used as the image to be repaired. In another approach, an image or video can be acquired from an image acquisition device or video recording device, and at least one frame from the acquired image or video can be used as the image to be repaired. Alternatively, the image or video to be repaired can be obtained from an image database, which is also possible.

[0104] After acquiring the image to be repaired, the location and labels of different objects within the image can be determined. It should be noted that when multiple images or video frames are acquired, they can be processed sequentially.

[0105] like Figure 2a As shown, this embodiment of the invention provides an image schematic diagram, after acquiring... Figure 2a After viewing the image shown, the location and labels of different objects in the image to be repaired can be determined. For example, as shown... Figure 2b As shown in the figure, this embodiment of the invention provides an image diagram with a determined region location and object label. Figure 2b Different regions in China Figure 2a The regions where different objects reside, and thus the location of different objects within a region, is the location of the region where the object resides. For example... Figure 2b The region where the object label is 1 represents Figure 2a The region where the sky is located, and the region with the object label 1, is the region position of the sky object. Figure 2b The region with the object label 2 represents Figure 2a The location of the building in the area where the object tag is 2 is the location of the building object. Figure 2b The region with the object label 3 represents Figure 2a The region where the tree is located, and the region with the object label 3, is the region location of the tree object. Figure 2b The region with the object label 4 represents Figure 2a The location of the grassland object is the region where the grassland is located, specifically the region with the object label 4.

[0106] Optionally, in one implementation, the location of each object in the image to be repaired can be a mask for that object. The mask indicates the region in the image belonging to that object. Specifically, the mask is a data matrix with specific values ​​of the same size as the image to be repaired. Different values ​​in this matrix distinguish between object regions and non-object regions. Thus, the location of the object in the image to be repaired can be determined using the mask, thereby identifying the object region. Alternatively, the location of each object in the image to be repaired can also be a set of pixels within the region where the object is located in the image to be repaired. This pixel set allows for the identification of which regions in the image to be repaired contain the object.

[0107] Optionally, in one implementation, the object label for each object can be pre-defined to distinguish different objects, such as the one described above. Figure 2b In the image, the object tag for the sky object is 1, the object tag for the building object is 2, the object tag for the tree object is 3, and the object tag for the grass object is 4.

[0108] In this step, there are multiple ways to determine the location of different objects and their labels in the image to be repaired. Optional methods include at least one of the following three:

[0109] The first method involves semantic segmentation of the image to be repaired to obtain the location of different objects in the image and their labels.

[0110] Semantic segmentation can divide different objects in the image to be repaired into pixel-level segments, and then assign object labels to pixels within the region where each object is located, generating a mask for that object. Therefore, after obtaining the image to be repaired, semantic segmentation can be performed to obtain the location and object labels of different objects in the image.

[0111] In one implementation, an independently trained semantic segmentation model can be used to perform semantic segmentation on the image to be repaired. That is, the image to be repaired is input into the semantic segmentation model to obtain the location of different objects in the image and the object labels.

[0112] In another implementation, when the embodiment of the present invention is applied to an image restoration model, the image restoration model may include a semantic segmentation layer, thereby enabling semantic segmentation of the image to be restored. For example, as shown... Figure 3 As shown in the figure, an embodiment of the present invention provides a schematic diagram of an image restoration model, wherein the semantic segmentation layer is used to perform semantic segmentation on the input image.

[0113] When this invention is applied to facial image restoration, because the similarity between facial features is high, semantic segmentation models or layers trained with ordinary data often cannot accurately segment facial images. Therefore, to improve segmentation accuracy, the aforementioned semantic segmentation model or layer can be a face segmentation model, which can be trained using sample facial data. When this invention is applied to animation image restoration, because animation scenes differ from images in real-world scenes, the aforementioned semantic segmentation model or layer needs to be adapted to animation images. Similarly, artworks also require corresponding semantic segmentation models or layers. When this invention is applied to traffic scene image restoration, the objects segmented by the semantic segmentation model or layer will tend to be pedestrians, motor vehicles, bicycles, cars, buses, buildings, roads, etc.

[0114] The second method involves image segmentation of the image to be repaired to determine the location of different objects in the image; then object recognition is performed on the image regions at each location in the image to be repaired to obtain the corresponding object labels.

[0115] In this method, after acquiring the image to be repaired, image segmentation can be performed first to determine the location of different objects in the image. Further, object recognition, object classification, and object detection techniques are used to identify objects in each region of the image to be repaired, obtaining object labels for each region. Optionally, the image to be repaired can be segmented using an image segmentation algorithm, and then object recognition, object classification, and / or object detection models can be used to identify the segmented image and obtain object labels for each region.

[0116] When this invention is applied to facial image restoration, because facial features are highly similar, object segmentation models, object recognition models, object classification models, and object detection models trained with ordinary data often cannot accurately segment or recognize facial images. Therefore, to improve the accuracy of segmentation and recognition, the aforementioned object segmentation models, object recognition models, object classification models, and object detection models can be segmentation models, recognition models, classification models, or detection models specifically for faces. When this invention is applied to animation image restoration, because animation scenes differ from images in real-world scenes, the aforementioned object segmentation models, object recognition models, object classification models, and object detection models need to be adapted to animation images. Similarly, artworks also require corresponding object segmentation models, object recognition models, object classification models, and object detection models. When this invention is applied to traffic scene image restoration, the objects segmented by the object segmentation models, object recognition models, object classification models, and object detection models will tend to be biased towards pedestrians, motor vehicles, bicycles, cars, buses, buildings, roads, etc.

[0117] The third method involves identifying key points in the image to be repaired to obtain each key point in the image; based on each key point, determining the location of different objects in the image to be repaired and their object labels.

[0118] When this invention is applied to facial image restoration, due to the high similarity between facial features, directly performing semantic segmentation or image segmentation on the face often yields inaccurate results. In this case, keypoint recognition can be performed on the image to be restored to obtain various keypoints of the facial image, such as the left eye keypoint, right eye keypoint, nose tip keypoint, and mouth keypoint. Then, based on these keypoints, the facial image to be restored is segmented, and the object label for each segmented region is determined based on the type of keypoints contained within that region.

[0119] S102, using a pre-set image restoration method for each object label, the image regions corresponding to each object label in the image to be restored are restored to obtain the restored image; wherein, the region corresponding to each object label is the region location of the object to which the object label belongs in the image to be restored.

[0120] After obtaining the location and labels of different objects in the image to be repaired, a pre-defined image inpainting method can be used to repair the image region corresponding to each object label, resulting in a repaired image. In simpler terms, after obtaining the location and labels of different objects in the image to be repaired, a pre-defined image inpainting method can be used for each object label to repair the image region corresponding to that object label. This allows the object regions containing different objects to be repaired using an image inpainting method tailored to the object's label.

[0121] In this step, the image restoration methods mentioned above can include at least one of interpolation-based restoration, texture synthesis-based restoration, and deep learning-based restoration. For different objects in the image to be restored, the restoration methods corresponding to the object labels of different objects can be of the same type, such as all being deep learning methods, or they can be of different types. For example, the image restoration method corresponding to object label 1 can be interpolation-based restoration, while the image restoration method corresponding to object label 2 can be texture synthesis-based restoration. It is important to emphasize that although the image restoration methods corresponding to the object labels of different objects can be of the same type, the restoration algorithms and parameters of the restoration models used for the restoration methods corresponding to different object labels can be different.

[0122] In one implementation, the image restoration methods corresponding to different object labels in the embodiments of the present invention are all based on deep learning. In the deep learning-based restoration methods corresponding to different object labels, the model parameters used by the deep learning restoration are set independently for different objects.

[0123] Optionally, when applied to an image restoration model in this embodiment of the invention, the image restoration model may include an image restoration layer, for example, such as... Figure 3 The image inpainting model shown has an image inpainting layer, which is independently configured for each object label. Figure 3 The model parameters in the image inpainting layer are shown, so that for each object label, the corresponding image inpainting method is determined. That is, the image inpainting layer uses the model parameters set for the object label to repair the image region corresponding to the object label in the image to be repaired.

[0124] In the above-described scheme of the present invention, since the image restoration method is pre-set for each object label, the image regions corresponding to each object label in the image to be restored are restored respectively. This means that in this scheme, the image regions where different objects are located in the image to be restored are restored using the image restoration method corresponding to each object. This allows each image restoration method to restore the local image of the image to be restored in a more targeted manner, thereby improving the restoration quality of the image restoration.

[0125] based on Figure 1 Examples, such as Figure 4 As shown, in another embodiment of the image restoration method provided by the present invention, the above-mentioned S102 may include steps S401-S404:

[0126] S401, Perform feature extraction on the image to be repaired to obtain shallow image features of the image to be repaired;

[0127] In this step, a preset convolution kernel can be used to perform convolution processing on the image to be repaired, thereby realizing feature extraction of the image to be repaired and obtaining shallow image features of the image to be repaired.

[0128] Optionally, in one implementation, when the embodiments of the present invention are applied to an image restoration model, based on Figure 3 The image restoration model shown is as follows: Figure 5 As shown, this embodiment of the invention provides another image restoration model, wherein the image restoration layer includes a shallow feature extraction layer, a deep feature extraction layer, and a feature reconstruction layer. In this case, the shallow feature extraction layer can be used to extract features from the image to be restored, obtaining the shallow image features of the image to be restored.

[0129] S402, From the shallow image features, determine the region features located at the region position corresponding to each object label, and use them as the region features corresponding to each object label;

[0130] In order to repair the image regions corresponding to each object label in the image to be repaired, the shallow image features can be used to determine the region features located at the region corresponding to each object label.

[0131] Optionally, when the feature map size of the shallow image features is the same as the image size of the image to be repaired, for each object label, the regional features within the region corresponding to the object label in the shallow image features can be determined as the regional features at the region corresponding to the object label. If the region location of each object is a mask for that object, for each object label, the product of the shallow image features and the object mask can be calculated to obtain the regional features within the region corresponding to the object label.

[0132] Optionally, when the feature map size of the shallow image features is inconsistent with the image size of the image to be repaired, the scaling ratio between the feature map size of the shallow image features and the image size of the image to be repaired can be predetermined. Then, for each object label, the scaled area of ​​the shallow image features and the area corresponding to the object label can be determined according to the scaling ratio. The area features within the area range can then be used as the area features at the area corresponding to the object label.

[0133] In one implementation, when the embodiments of the present invention are applied to an image restoration model, it can be based on Figure 5 The deep feature extraction layer in the image restoration model shown performs the step of determining the region features located at the region corresponding to each object label from the shallow image features, and using these as the region features corresponding to each object label.

[0134] S403, Based on the deep feature extraction method pre-set for each object label, perform deep feature extraction on the region features corresponding to each object label to obtain the depth image features corresponding to the object label.

[0135] After obtaining the region features corresponding to each object label, it is necessary to use the deep feature extraction method set for each object label to perform deep feature extraction on the region features corresponding to each object label, so as to obtain the depth image features corresponding to the object label. In short, for each object label, it is necessary to use the deep feature extraction method set for that object label to perform further feature processing on the region features corresponding to that object label, so as to obtain the depth image features corresponding to that object label.

[0136] The deep feature extraction method set for each object label can be at least one convolution kernel pre-set for that object label, which can then be used to perform convolution processing on the region features corresponding to that object label to obtain the depth image features corresponding to that object label.

[0137] Optionally, the deployment structure of at least one convolutional kernel set for each object label can be the same, but the parameters within the convolutional kernel can be set independently for each object label. Alternatively, the deployment structure can be the same, but at least one parameter in the convolutional kernel corresponds to the object label, such as the weight of the input channel of the convolutional kernel. For ease of description, the relevant parameters set independently for the object label in this embodiment of the invention are collectively referred to as deep feature extraction parameters.

[0138] In this case, optionally, deep feature extraction parameters corresponding to each object label can be determined, and then for each object label, deep feature extraction can be performed on the region features corresponding to the object label based on the deep feature extraction parameters corresponding to the object label to obtain the depth image features corresponding to the object label.

[0139] Optionally, there can be many ways to determine the deep feature extraction parameters corresponding to each object label. For example, at least one of the following two methods can be used:

[0140] The first method is based on the correspondence between preset object labels and deep feature extraction parameters, and selects deep feature extraction parameters that correspond to each object label in the image to be repaired from the preset deep feature extraction parameters.

[0141] In this scenario, one implementation involves pre-constructing a deep feature extraction parameter pool, which contains deep feature extraction parameters corresponding to each object label. Optionally, the deep feature extraction parameters corresponding to each object label in the deep feature extraction parameter pool can be all or part of the parameters from a feature extraction algorithm pre-set for the object to which that label belongs. The setting method can be determined through experience or training. The aforementioned feature extraction algorithm can be a deep feature extraction network.

[0142] After constructing the deep feature extraction parameter pool, once the region features corresponding to each object label in the image to be repaired are obtained, the deep feature extraction parameters corresponding to each object label in the image to be repaired can be selected from the deep feature extraction parameters contained in the deep feature extraction parameter pool based on the preset correspondence between the object labels and the deep feature extraction parameters.

[0143] When the deep feature extraction parameters are network parameter values ​​specified in the deep feature extraction network, the network parameter values ​​corresponding to each object label in the image to be repaired can be selected from the preset network parameter values ​​based on the correspondence between preset object labels and network parameter values.

[0144] In this case, for each object label, the network parameter value corresponding to the object label can be loaded into the specified network parameters in the deep feature extraction network, and then the deep feature extraction network with the changed network parameter value can be used to extract the region features corresponding to the object label to obtain the depth image features corresponding to the object label.

[0145] The aforementioned deep feature extraction network can contain multiple convolutional layers. In one approach, the specified network parameters can be the network parameters of all convolutional layers contained in the deep feature extraction network. That is, the deep feature extraction parameters corresponding to each object label can be the network parameter values ​​of all network parameters contained in the deep feature extraction network.

[0146] In this scenario, a network parameter pool corresponding to the deep feature extraction network can be pre-constructed. This pool contains the network parameter values ​​of all network parameters in the deep feature extraction network when performing feature extraction on objects with different object labels. Then, based on the pre-defined correspondence between object labels and network parameter values, the network parameter values ​​corresponding to each object label in the image to be repaired can be selected from the network parameter pool. All acquired network parameter values ​​are then loaded into the specified network parameters of the deep feature extraction network. Finally, the deep feature extraction network with the modified network parameter values ​​is used to extract features from the region corresponding to the object label, obtaining the depth image features corresponding to that object label.

[0147] In the case of applying the above-mentioned deep feature extraction network to an image restoration model in the embodiments of the present invention, it can be as follows: Figure 5 The image restoration model shown has a deep feature extraction layer. In one implementation, based on... Figure 5 The image restoration model shown is as follows: Figure 6 As shown, this embodiment of the invention provides another image inpainting model, including a network parameter pool for providing network parameter values ​​for the deep feature extraction layer. The network parameter pool can be deployed inside the image inpainting model (not shown in the figure) or outside the image inpainting model; both are possible. In another implementation, based on... Figure 5 The image restoration model shown is as follows: Figure 7As shown, this embodiment of the invention provides another image restoration model. The deep feature extraction layer includes different branches set for different object labels. The network parameter values ​​set for different object labels in the parameter network pool are the network parameter values ​​contained in the branches. The process of selecting and loading the network parameter values ​​is the process of selecting branches. The structures of different branches in the deep feature extraction layer can be the same. Each branch only performs deep feature extraction on the region features corresponding to the object label of that branch.

[0148] The second approach involves extracting features from the image region corresponding to the object label in the image to be repaired for each object label, thereby obtaining the image features of the region corresponding to the object label; and determining the deep feature extraction parameters corresponding to the object label based on the image features of the region corresponding to the object label.

[0149] In addition to the above-mentioned method of pre-generating and filtering based on correspondence, the deep feature extraction parameters corresponding to each object label can be generated in real time. In this case, feature extraction can be performed on the image region at the location corresponding to each object label in the image to be repaired to obtain the image features of the region corresponding to each object label. Then, based on the image features of the region corresponding to each object label, the deep feature extraction parameters corresponding to each object label can be determined.

[0150] In this case, after determining the location and label of different objects in the image to be repaired, feature extraction can be performed on the image to be repaired based on the location and label of different objects. Optionally, for each object label, feature extraction can be performed on the image region corresponding to that object label in the image to be repaired, obtaining the image features of the region corresponding to that object label. Figure 2a The example shown illustrates this point and can be used to... Figure 2a Feature extraction is performed on the sky region where the object is located to obtain the regional image features of the sky region, and then targeted at... Figure 2a Feature extraction is performed on the tree region where the tree object is located to obtain the regional image features of the tree region.

[0151] Optionally, in one implementation, for each object label, a feature recognition network can be used to extract features from the image region corresponding to the object label in the image to be repaired, thereby obtaining the image features of the region corresponding to the object label.

[0152] The aforementioned feature recognition network may contain at least one convolutional kernel, and then use the at least one convolutional kernel to extract features from the image region corresponding to each object label in the image to be repaired, thereby obtaining the image features of the region corresponding to each object label.

[0153] To better extract regional image features of different objects, the network structure of the aforementioned feature recognition network is the same as that of the network used for feature extraction in the first object recognition model, which is a neural network model for object recognition. Optionally, this first object recognition model can be a neural network model capable of multi-object recognition, which can output object labels for all objects in the image from a single input image.

[0154] In the case of applying the embodiments of the present invention to an image restoration model, based on Figure 5 The image restoration model shown is as follows: Figure 8 As shown, this embodiment of the invention provides another image restoration model, which may include a parameter determination layer. The parameter determination layer includes the feature extraction network described above and is used to extract features from the image to be restored by region using the mask and semantic labels output by the semantic segmentation layer, to obtain the region image features corresponding to each object label, and then, for each object label, to determine the deep feature extraction parameters corresponding to the object label based on the region image features corresponding to the object label.

[0155] Optionally, after obtaining the region image features, in order to generate deep feature extraction parameters for each object label in real time, one approach is to input the region image features corresponding to the object label into a preset fully connected layer corresponding to the object label to obtain the deep feature extraction parameters corresponding to the object label.

[0156] By using fully connected layers, the features of a region image can be converted into a parameter vector, which can then be used as the parameters for extracting deep features corresponding to the object label.

[0157] In one implementation, the deep feature extraction parameters are the network parameter values ​​of specified network parameters in the deep feature extraction network. In this case, the network parameter values ​​of the specified network parameters can be determined based on the extracted parameter vector. In another implementation, when the specified network parameters are all the network parameters included in the deep feature extraction network, the parameter value of each network parameter can be determined based on the parameter vector. For example, the value at a specified position in the parameter vector can be used as the parameter value of that network parameter. In yet another implementation, when the specified network parameters are the network parameters of a specified convolutional layer in the deep feature extraction network, the parameter vector can be used as the channel weights of the input channels of the specified convolutional layer. In this case, if the specified convolutional layer has N input channels, the parameter vector can be an N-dimensional parameter vector, where N is a positive integer, for example... Figure 8The weight vector of the N-dimensional channels of a specified convolutional layer in a medium-depth feature extraction layer. In this case, for each object label, the region image features corresponding to that object label can be input into a pre-defined fully connected layer corresponding to that object label to obtain the deep feature extraction parameters corresponding to that object label.

[0158] The fully connected layer is used to process the region image features corresponding to the object label to obtain an N-dimensional vector, which is used as the weight vector of the specified convolutional layer. Then, for each object label, the determined weight vector is loaded into the deep feature extraction network, and the deep feature extraction network with the weight vector loaded is used to extract the region features corresponding to the object label to obtain the depth image features corresponding to the object label.

[0159] In the case of applying the embodiments of the present invention to an image restoration model, based on Figure 8 The image restoration model shown is as follows: Figure 9 As shown, this embodiment of the invention provides another image restoration model. The parameter determination layer includes a feature recognition layer and a fully connected layer (FC) for each object label. The feature recognition layer includes the aforementioned feature recognition network. In this case, the feature recognition layer is used to determine the regional image features of each object label. Then, for each object label, the FC set for that object label is used to process the regional image features corresponding to that object label to obtain an N-dimensional feature vector, which serves as the weight vector of the specified convolutional layer corresponding to that object label. Then, when the deep feature extraction layer processes the regional features corresponding to that object label, it can use the weight vector corresponding to that object label as the weight vector of the input channel of the specified convolutional layer in the deep feature extraction layer, and perform feature extraction in one step to obtain the deep image features corresponding to that object label.

[0160] S404: Perform feature reconstruction on the depth image features corresponding to each object label to obtain the repaired image.

[0161] After obtaining the depth image features of each object label, feature reconstruction can be performed on the depth image features corresponding to each object label to obtain the repaired image. Feature reconstruction refers to regenerating an image that is similar to, but not completely identical to, the image to be repaired by manipulating the depth image features; that is, the repaired image.

[0162] Optionally, after obtaining the depth image features of each object label, the depth image features of each object label can be merged to obtain complete depth image features, and then feature reconstruction can be performed on the depth image features.

[0163] When applied to an image restoration model in the embodiments of the present invention, it is possible to utilize Figures 5-9In any image restoration model, the feature reconstruction layer performs feature reconstruction on the depth image features corresponding to each object label to obtain the restored image.

[0164] The above-described scheme in this embodiment of the invention can improve the restoration quality of image restoration. Furthermore, by employing different deep feature extraction methods, the depth image features of different object labels are determined, and then feature reconstruction is performed on the depth image features of each object label to obtain the restored image. This provides a targeted restoration method, laying the foundation for improving the restoration quality of image restoration.

[0165] In one embodiment, the image restoration method described above provided by the present invention can be applied to an image restoration model.

[0166] Based on this, such as Figure 10 As shown, this embodiment of the invention also provides an image restoration method, wherein the training steps of the above-mentioned image restoration model include S1001-S1004:

[0167] S1001, Obtain the sample image and the corresponding ground image;

[0168] The sample images can be of lower quality, while the corresponding ground truth images can be of higher quality. The sample images and their corresponding ground truth images can be collected in advance; this embodiment of the invention does not impose specific limitations on this.

[0169] S1002, Input the sample image into the image restoration model to be trained so that the image restoration model to be trained can determine the location of different objects in the sample image and the object label; Use the image restoration method to be trained for each object label to restore the image region at the location corresponding to each object label in the sample image to obtain the restored image;

[0170] After acquiring the sample image, it can be input into the image inpainting model to be trained, so that the image inpainting model to be trained can determine the location of different objects in the sample image and the object label; using the image inpainting method set for each object label, the image region at the location corresponding to each object label in the sample image is repaired to obtain the repaired image.

[0171] The image restoration model to be trained can be a model structure and Figure 3 , Figures 5-9 The image restoration model shown is the same, and its processing procedure for the sample image is the same. Figure 1 , Figure 4 The process of processing the image to be repaired in the embodiments shown is the same or similar, and will not be described again here.

[0172] S1003, Based on the restored image and the ground truth image, determine the loss value of the image restoration model to be trained;

[0173] After obtaining the restored image, the loss value of the image restoration model to be trained can be determined based on the restored image and the ground truth image.

[0174] Optionally, the loss value of the image inpainting model to be trained can be calculated in various ways. For example, it can include at least one of the following methods:

[0175] The first method of loss calculation calculates the difference in pixel values ​​between the restored image and the ground truth image, using this difference as the first loss value.

[0176] Optionally, at least one of the following loss functions can be used to calculate the difference between the pixel values ​​of the restored image and the ground truth image: L1 loss (a mean absolute error loss function), MSE loss (mean squared error loss function), Charbonnier loss (an image reconstruction loss function).

[0177] Taking L1 loss as an example, the calculation of l fidelity =1(O,G), where l fidelity To correct the difference in pixel values ​​between the original image and the ground truth image, O represents the corrected image, G represents the ground truth image, and L1 represents the calculation of L1 loss.

[0178] The second method for calculating the loss value involves inputting the restored image and the ground truth image into a second object recognition model. The second object recognition model is a pre-trained neural network model containing multiple sequentially connected feature extraction layers. The image features of the restored image and the image features of the ground truth image are obtained from at least one of the multiple feature extraction layers. The feature difference between the image features of the restored image and the image features of the ground truth image is calculated as the second loss value.

[0179] The second object recognition model is a separate network model, pre-trained independently. Its training data can utilize open-source, pre-annotated object recognition datasets or manually annotated private datasets. This second object recognition model is capable of object recognition; once trained, it can be used to calculate the loss value for the image restoration model.

[0180] like Figure 11The diagram illustrates feature extraction using a second object recognition model provided in an embodiment of the present invention. This second object recognition model includes M feature extraction layers. After the repaired image is input into the second object recognition model, image features of the repaired image are output at least one feature extraction layer (shown in the diagram as the current image features are output after feature extraction layers 1 and 3, respectively). Then, after the ground truth image is input into the second object recognition model, image features of the repaired image are output at least one feature extraction layer (shown in the diagram as the current image features are output after feature extraction layers 1 and 3, respectively). Finally, the feature difference between the image features of the repaired image and the image features of the ground truth image is calculated. Optionally, the feature difference between the image features of the restored image and the image features of the ground truth image output by the same feature extraction layer can be calculated independently. For example, the feature difference between f1(O) and f1(G) and the feature difference between f2(O) and f2(G) can be calculated. Here, f1(O) is the image feature of the restored image output by feature extraction layer 1, f2(O) is the image feature of the restored image output by feature extraction layer 3, f1(G) is the image feature of the ground truth image output by feature extraction layer 1, and f2(G) is the image feature of the ground truth image output by feature extraction layer 3. Then, the feature difference between the image features output by each feature extraction layer is combined as the second loss value.

[0181] In one implementation, before acquiring the image features of the restored image and the ground truth image output by at least one feature extraction layer from multiple feature extraction layers, the regional locations and object labels of different objects in the sample image can be input into a second object recognition model. Then, for each input object label, the regional features of the restored image and the regional features of the ground truth image corresponding to that object label are acquired. Furthermore, for each input object label, the feature difference between the regional features of the restored image and the regional features of the ground truth image corresponding to that object label is calculated and used as the feature difference magnitude corresponding to that object label. Based on the feature difference magnitudes corresponding to each input object label, the feature difference magnitude between the image features of the restored image and the image features of the ground truth image is determined. The regional features of any image corresponding to each object label are: the regional features output by at least one feature extraction layer from multiple feature extraction layers for the image region at the location corresponding to the object label in that image.

[0182] Based on the location and label of different objects in the sample image, a corresponding object recognition module is selected for each object. The object recognition module extracts a specified number (e.g., 3) of features from its feature extraction layers, outputting the region features of the restored image and the ground truth image corresponding to each object label. Then, for each input object label, the feature difference between the region features of the restored image and the ground truth image corresponding to that object label is calculated. This difference is used as the feature difference magnitude corresponding to that object label. Finally, based on the feature difference magnitudes of each input object label, the feature difference magnitude between the image features of the restored image and the image features of the ground truth image is determined.

[0183] For example, using L1 loss as the loss function to calculate the feature difference, the following formula can be used for calculation:

[0184]

[0185] Among them, l perceptual The second loss value is denoted by i, where i is the number of object labels in the sample image, and M is the number of labels in the sample image. i f represents the mask corresponding to the i-th object label. j (O) represents the image features of the repaired image output by the j-th feature extraction layer, f j (G) represents the image features of the ground truth image output by the j-th feature extraction layer, and K represents the number of feature extraction layers for the image features of the output restored image.

[0186] like Figure 12 The diagram shown is a training framework diagram of an image restoration model provided by an embodiment of the present invention. In this model, a second object recognition model is used to extract the feature difference between the restored image and the ground truth image to obtain the second loss value of the image restoration model to be trained.

[0187] The third method of loss value calculation involves inputting the repaired image into a preset image discriminator to obtain a first evaluation score for the repaired image as a real image, which is output by the image discriminator; calculating the first difference between the first evaluation score and the first preset score value as the third loss value; wherein, the first preset score value is the score value that represents the real image.

[0188] The image discriminator can be a pre-trained discriminator or an image discriminator to be trained. In the case of an image discriminator to be trained, adversarial training can be formed between the image restoration model to be trained and the image discriminator to be trained.

[0189] Optional, such as Figure 13The diagram shown illustrates a training framework for an image restoration model according to an embodiment of the present invention. The image discriminator outputs a first evaluation score indicating that the restored image is a real image. Then, a first difference between the first evaluation score and a first preset score value is calculated as a third loss value. The first preset score value represents the score of the real image. Optionally, the first preset score value can be 1.

[0190] In one implementation, the location and labels of different objects in the restored image and the sample image are input into a preset image discriminator. The image discriminator outputs a region evaluation score for each image region corresponding to the input object label in the restored image. Based on these region evaluation scores, a first evaluation score is determined to identify the restored image as a true image. By calculating different losses for different object regions, the targeted restoration capability of the trained image restoration model can be further improved.

[0191] In situations where adversarial training is possible between the image inpainting model to be trained and the image discriminator, the parameters of the image discriminator to be trained can be further updated, such as... Figure 14 The diagram illustrates a training framework for an image restoration model according to an embodiment of the present invention. A ground truth image is input into an image discriminator to be trained, yielding a second evaluation score indicating that the ground truth image is a real image. A second difference between the second evaluation score and a first preset score is then calculated, as well as a third difference between the first evaluation score and the second preset score. The second preset score represents a score indicating a fake image. Based on the second and third differences, the parameters of the image discriminator are updated. The second preset score can be 0. After obtaining the second and third differences, an adversarial loss can be calculated, and the parameters of the image discriminator are updated based on this adversarial loss.

[0192] After obtaining the first loss value, the second loss value, and / or the third loss value, the loss value of the image inpainting model to be trained can be determined based on the first loss value, the second loss value, and / or the third loss value.

[0193] Optionally, the loss value of the image inpainting model to be trained can be calculated using the following formulas, including:

[0194] Loss = w l *l fidelity +w p *l perceptual +w g * gan

[0195] Among them, l fidelity For the first loss value, l perceptual For the second loss value, lgan The third loss value, w l ,w p ,w g These are the weighting coefficients for the three loss values, and they must be greater than or equal to 0 (0 indicates that this loss value is not used).

[0196] like Figure 15 The diagram illustrates a training framework for an image restoration model according to an embodiment of the present invention. It includes an image restoration model to be trained, an image discriminator to be trained, and a second object recognition model. The image restoration model calculates a first loss value using the restored image and the ground truth image, calculates a second loss value using the first evaluation score output by the image discriminator to be trained, and calculates a third loss value using the image features of the restored image and the ground truth image output by the second object recognition model. The model parameters of the image restoration model to be trained are then adjusted by combining the first, second, and third loss values. The image discriminator to be trained determines the second and third evaluation scores using the ground truth image and the restored image, respectively, and determines the discrimination loss of the image discriminator based on the second and third evaluation scores. The parameters of the image discriminator to be trained are then adjusted based on the discrimination loss.

[0197] S1004, Based on the obtained loss value, adjust the model parameters in the image restoration model to be trained.

[0198] After obtaining the loss value, the model parameters in the image restoration model to be trained can be adjusted based on the obtained loss value. Optionally, gradient descent, optimizers, or other methods can be used to adjust the model parameters in the image restoration model to be trained.

[0199] The above-described solutions in this embodiment of the invention can improve the quality of image restoration. Furthermore, they provide a method for training image restoration models, thus providing a foundation for generating image restoration models with targeted restoration capabilities.

[0200] Corresponding to the image restoration method provided in the above embodiments of the present invention, such as Figure 16 As shown, this embodiment of the invention also provides an image restoration device, the device comprising:

[0201] The information determination module 1601 is used to determine the location of different objects in the image to be repaired and the object labels;

[0202] The image restoration module 1602 is used to restore the image regions corresponding to each object label in the image to be restored by using an image restoration method pre-set for each object label, so as to obtain the restored image; wherein, the region corresponding to each object label is the region location of the object to which the object label belongs in the image to be restored.

[0203] Optionally, the image restoration module includes:

[0204] The first feature extraction submodule is used to extract features from the image to be repaired to obtain shallow image features of the image to be repaired.

[0205] The feature determination submodule is used to determine the region features located at the region position corresponding to each object label from the shallow image features, and use them as the region features corresponding to each object label.

[0206] The second feature extraction submodule is used to perform deep feature extraction on the region features corresponding to each object label based on the deep feature extraction method set in advance for each object label, so as to obtain the depth image features corresponding to the object label.

[0207] The feature reconstruction submodule is used to reconstruct the features of the depth image corresponding to each object label to obtain the repaired image.

[0208] Optionally, the second feature extraction submodule includes:

[0209] The parameter determination unit is used to determine the deep feature extraction parameters corresponding to each object label;

[0210] The feature extraction unit is used to extract deep features from the region corresponding to each object label based on the deep feature extraction parameters corresponding to that object label, so as to obtain the depth image features corresponding to that object label.

[0211] Optionally, the parameter determining unit includes:

[0212] The filtering subunit is used to filter deep feature extraction parameters corresponding to each object label in the image to be repaired from the preset deep feature extraction parameters based on the correspondence between preset object labels and deep feature extraction parameters; and / or, the determining subunit is used to perform feature extraction on the image region at the location corresponding to the object label in the image to be repaired for each object label, to obtain the image features of the region corresponding to the object label; and to determine the deep feature extraction parameters corresponding to the object label based on the image features of the region corresponding to the object label.

[0213] Optionally, the determining subunit is specifically used to extract features from the image region at the location corresponding to the object label in the image to be repaired using a feature recognition network, thereby obtaining the image features of the region corresponding to the object label; wherein, the network structure of the feature recognition network is the network structure of the network used for feature extraction in the first object recognition model; the first object recognition model is a neural network model used for object recognition.

[0214] Optionally, the determining subunit is specifically used to input the region image features corresponding to the object label into a preset fully connected layer corresponding to the object label to obtain the deep feature extraction parameters corresponding to the object label.

[0215] Optionally, the deep feature extraction parameters are network parameter values ​​of specified network parameters in the deep feature extraction network;

[0216] The feature extraction unit is specifically used to load the network parameter value corresponding to each object label into the specified network parameters in the deep feature extraction network for each object label; and to extract the region features corresponding to the object label using the deep feature extraction network with the changed network parameter values ​​to obtain the depth image features corresponding to the object label.

[0217] Optionally, the specified network parameters may be all the network parameters included in the deep feature extraction network, or the network parameters of a specified convolutional layer in the deep feature extraction network.

[0218] Optionally, the information determination module is specifically used to perform semantic segmentation on the image to be repaired to obtain the region location and object label of different objects in the image to be repaired; and / or, perform image segmentation on the image to be repaired to determine the region location of different objects in the image to be repaired; perform object recognition on the image regions at each region location in the image to be repaired to obtain the object label corresponding to each region location; and / or, perform key point recognition on the image to be repaired to obtain each key point in the image to be repaired; and determine the region location and object label of different objects in the image to be repaired based on each key point.

[0219] Optionally, the device is applied to an image restoration model;

[0220] The image restoration model is trained using the following modules:

[0221] The image acquisition module is used to acquire a sample image and the ground truth image corresponding to the sample image;

[0222] The image input module is used to input the sample image into the image restoration model to be trained, so that the image restoration model to be trained can determine the region location and object label of different objects in the sample image; and to repair the image region at the region location corresponding to each object label in the sample image by using the image restoration method set for each object label, so as to obtain the restored image.

[0223] The loss value determination module is used to determine the loss value of the image restoration model to be trained based on the restored image and the ground truth image.

[0224] The parameter adjustment module is used to adjust the model parameters in the image restoration model to be trained based on the obtained loss value.

[0225] Optionally, the loss value determination module includes:

[0226] The first determining submodule is used to calculate the difference in pixel values ​​between the repaired image and the ground image, as a first loss value; and / or,

[0227] The second determining submodule is used to input the restored image and the ground truth image into a second object recognition model, respectively; wherein the second object recognition model is a pre-trained neural network model containing multiple sequentially connected feature extraction layers; to obtain the image features of the restored image and the image features of the ground truth image output by at least one feature extraction layer among the multiple feature extraction layers; to calculate the feature difference between the image features of the restored image and the image features of the ground truth image as a second loss value; and / or,

[0228] The third determining submodule is used to input the repaired image into a preset image discriminator to obtain a first evaluation score that the repaired image output by the image discriminator is a real image; calculate a first difference between the first evaluation score and a first preset score value as a third loss value; wherein, the first preset score value is a score value that characterizes the real image;

[0229] The loss value determination submodule is used to determine the loss value of the image inpainting model to be trained based on the first loss value, the second loss value, and / or the third loss value.

[0230] Optionally, the device further includes:

[0231] An information input module is used to input the region location and object label of different objects in the sample image into the object recognition model before the second determining submodule performs the step of obtaining the image features of the repaired image and the image features of the ground truth image output by at least one feature extraction layer among the plurality of feature extraction layers respectively.

[0232] The second determining submodule is specifically used to obtain, for each input object label, the region features of the repaired image corresponding to that object label and the region features of the ground truth image corresponding to that object label; wherein, the region features of any image corresponding to each object label are: the region features output by at least one feature extraction layer among the plurality of feature extraction layers for the image region at the region position corresponding to that object label in the image; for each input object label, the feature difference between the region features of the repaired image corresponding to that object label and the region features of the ground truth image corresponding to that object label is calculated, and this difference is used as the feature difference size corresponding to that object label; based on the feature difference sizes corresponding to each input object label, the feature difference size between the image features of the repaired image and the image features of the ground truth image is determined.

[0233] Optionally, the third determining submodule includes:

[0234] The information input unit is used to input the location and object label of different objects in the repaired image and the sample image into a preset image discriminator to obtain the region evaluation score of the image region at the location corresponding to each input object label in the repaired image, output by the image discriminator.

[0235] The evaluation determination unit is used to determine the first evaluation score of the restored image as a real image based on the obtained evaluation scores of each region.

[0236] Optionally, the device further includes:

[0237] The discriminator update unit is configured to, after the third determining submodule performs the step of inputting the repaired image into a preset image discriminator to obtain a first evaluation score output by the image discriminator indicating that the repaired image is a real image, input the ground truth image into the image discriminator to obtain a second evaluation score predicted by the image discriminator indicating that the ground truth image is a real image; calculate a second difference between the second evaluation score and the first preset score, and calculate a third difference between the first evaluation score and the second preset score; wherein the second preset score is a score representing a fake image; and update the parameters of the image discriminator based on the second difference and the third difference.

[0238] In the above-described scheme of the present invention, since the image restoration method is pre-set for each object label, the image regions corresponding to each object label in the image to be restored are restored respectively. This means that in this scheme, the image regions where different objects are located in the image to be restored are restored using the image restoration method corresponding to each object. This allows each image restoration method to restore the local image of the image to be restored in a more targeted manner, thereby improving the restoration quality of the image restoration.

[0239] This invention also provides an electronic device, such as... Figure 17 As shown, it includes a processor 1701, a communication interface 1702, a memory 1703, and a communication bus 1704. The processor 1701, the communication interface 1702, and the memory 1703 communicate with each other through the communication bus 1704.

[0240] Memory 1703 is used to store computer programs;

[0241] When processor 1701 executes a program stored in memory 1703, it performs the following steps:

[0242] Determine the location and label of different objects in the image to be repaired;

[0243] Using a pre-set image restoration method for each object label, the image regions corresponding to each object label in the image to be restored are restored to obtain the restored image;

[0244] Among them, the region location corresponding to each object label is the region location of the object to which the object label belongs in the image to be repaired.

[0245] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0246] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0247] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0248] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0249] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image restoration methods.

[0250] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image restoration methods described above.

[0251] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0252] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0253] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0254] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An image inpainting method characterized by, The method comprises: determining the region positions and object labels of different objects in the image to be repaired; performing feature extraction on the image to be repaired to obtain shallow image features of the image to be repaired; from the shallow image features, determining the region features located at the region positions corresponding to each object label as the region features corresponding to each object label; for each object label, performing feature extraction on the image region at the region position corresponding to the object label in the image to be repaired to obtain the region image features corresponding to the object label; and determining the deep feature extraction parameters corresponding to the object label based on the region image features corresponding to the object label; for each object label, performing deep feature extraction on the region features corresponding to the object label based on the deep feature extraction parameters corresponding to the object label to obtain the deep image features corresponding to the object label; performing feature reconstruction on the deep image features corresponding to each object label to obtain a repaired image; wherein the region position corresponding to each object label is the region position of the object to which the object label belongs in the image to be repaired.

2. The method of claim 1, wherein, The feature extraction on the image region at the region position corresponding to the object label in the image to be repaired to obtain the region image features corresponding to the object label comprises: performing feature extraction on the image region at the region position corresponding to the object label in the image to be repaired by using a feature recognition network to obtain the region image features corresponding to the object label; wherein the network structure of the feature recognition network is the network structure of the network for extracting features in a first object recognition model; and the first object recognition model is a neural network model for object recognition.

3. The method of claim 1, wherein, The determination of the deep feature extraction parameters corresponding to the object label based on the region image features corresponding to the object label comprises: inputting the region image features corresponding to the object label into a preset fully connected layer corresponding to the object label to obtain the deep feature extraction parameters corresponding to the object label.

4. The method of claim 1, wherein, The deep feature extraction parameters are network parameter values of specified network parameters in a deep feature extraction network. The deep feature extraction of the region features corresponding to each object label based on the deep feature extraction parameters corresponding to the object label to obtain the deep image features corresponding to the object label comprises: for each object label, loading the network parameter values corresponding to the object label into the specified network parameters in the deep feature extraction network; and performing feature extraction on the region features corresponding to the object label by using the deep feature extraction network with the changed network parameter values to obtain the deep image features corresponding to the object label.

5. The method of claim 4, wherein, The specified network parameters are all network parameters included in the deep feature extraction network, or network parameters of a specified convolutional layer in the deep feature extraction network.

6. The method of claim 1, wherein, The determination of the region positions and object labels of different objects in the image to be repaired comprises: performing semantic segmentation on the image to be repaired to obtain the region positions and object labels of different objects in the image to be repaired; and / or, performing image segmentation on the image to be inpainted to determine region positions of different objects in the image to be inpainted; performing object recognition on image regions at the region positions in the image to be inpainted respectively to obtain corresponding object labels at the region positions; and / or, performing key point recognition on the image to be inpainted to obtain key points in the image to be inpainted; and determining region positions of different objects and object labels in the image to be inpainted based on the key points.

7. The method according to any one of claims 1 to 6, characterized in that, The method is applied to an image inpainting model. The training manner of the image inpainting model comprises: obtaining a sample image and a ground truth image corresponding to the sample image; inputting the sample image into a to-be-trained image inpainting model to enable the to-be-trained image inpainting model to determine region positions of different objects and object labels in the sample image; and performing inpainting on image regions at region positions corresponding to object labels in the sample image respectively by using to-be-trained image inpainting manners set for the object labels to obtain an inpainted image. determining a loss value of the to-be-trained image inpainting model based on the inpainted image and the ground truth image. adjusting model parameters in the to-be-trained image inpainting model based on the obtained loss value.

8. The method of claim 7, wherein, The determining of the loss value of the to-be-trained image inpainting model based on the inpainted image and the ground truth image comprises: calculating a difference in pixel values between the inpainted image and the ground truth image as a first loss value; and / or, inputting the inpainted image and the ground truth image into a second object recognition model respectively; the second object recognition model is a pre-trained neural network model comprising a plurality of sequentially connected feature extraction layers; image features of the inpainted image and image features of the ground truth image output by at least one feature extraction layer of the plurality of sequentially connected feature extraction layers are obtained respectively; a feature difference between the image features of the inpainted image and the image features of the ground truth image is calculated as a second loss value; and / or, inputting the inpainted image into a preset image discriminator to obtain a first evaluation score of the inpainted image output by the image discriminator as a real image; a first difference between the first evaluation score and a first preset score value is calculated as a third loss value; the first preset score value is a score value representing a real image. The loss value of the to-be-trained image inpainting model is determined based on the first loss value, the second loss value and / or the third loss value.

9. The method of claim 8, wherein, Before the image features of the inpainted image and the image features of the ground truth image output by at least one feature extraction layer of the plurality of sequentially connected feature extraction layers are obtained respectively, the method further comprises: inputting region positions of different objects and object labels in the sample image into the second object recognition model. The obtaining of the image features of the inpainted image and the image features of the ground truth image output by at least one feature extraction layer of the plurality of sequentially connected feature extraction layers comprises: For each input object label, the region feature of the repair image corresponding to the object label and the region feature of the ground truth image corresponding to the object label are obtained respectively; wherein, the region feature of any image corresponding to each object label is the region feature output by at least one feature extraction layer in the plurality of sequentially connected feature extraction layers for the image region at the region position corresponding to the object label in the image; The calculation of the feature difference size between the image feature of the repair image and the image feature of the ground truth image comprises: For each input object label, the feature difference size between the region feature of the repair image corresponding to the object label and the region feature of the ground truth image corresponding to the object label is calculated as the feature difference size corresponding to the object label; Based on the feature difference sizes corresponding to the input object labels, the feature difference size between the image feature of the repair image and the image feature of the ground truth image is determined.

10. The method of claim 9, wherein, The input of the repair image into the preset image discriminator to obtain the first evaluation score of the repair image as a real image output by the image discriminator comprises: The input of the repair image and the region position and object label of different objects in the sample image into the preset image discriminator to obtain the region evaluation score of the image region at the region position corresponding to each input object label in the repair image output by the image discriminator; Based on the obtained region evaluation scores, the first evaluation score of the repair image as a real image is determined.

11. The method of claim 8, wherein, After the input of the repair image into the preset image discriminator to obtain the first evaluation score of the repair image as a real image output by the image discriminator, the method further comprises: The input of the ground truth image into the image discriminator to obtain the second evaluation score of the ground truth image as a real image predicted by the image discriminator; The calculation of the second difference value between the second evaluation score and the first preset score value and the calculation of the third difference value between the first evaluation score and the second preset score value; wherein, the second preset score value is a score value representing a false image; Based on the second difference value and the third difference value, the parameters of the image discriminator are updated.

12. An image inpainting apparatus characterized by comprising: The device comprises: An information determination module for determining the region position and object label of different objects in the to-be-repaired image; An image repair module, the image repair module comprising: A first feature extraction submodule for performing feature extraction on the to-be-repaired image to obtain the shallow image feature of the to-be-repaired image; A feature determination submodule for determining the region feature at the region position corresponding to each object label from the shallow image feature as the region feature corresponding to each object label; The second feature extraction submodule comprises a parameter determination unit and a feature extraction unit; the parameter determination unit comprises a determination subunit, which is configured to, for each object label, perform feature extraction on an image region at a region position corresponding to the object label in the image to be repaired to obtain a region image feature corresponding to the object label; and determine a deep feature extraction parameter corresponding to the object label based on the region image feature corresponding to the object label; and the feature extraction unit is configured to, for each object label, perform deep feature extraction on the region feature corresponding to the object label based on the deep feature extraction parameter corresponding to the object label to obtain a deep image feature corresponding to the object label; The feature reconstruction submodule is configured to perform feature reconstruction on the deep image feature corresponding to each object label to obtain a repaired image; and the region position corresponding to each object label is a region position of an object to which the object label belongs in the image to be repaired.

13. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are in communication with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method steps in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium, and the computer program is executed by the processor to implement the method steps in any one of claims 1-11.

Images