Automatic completion and restoration method for certificate photo and equipment capable of processing photo

Abstract

The invention discloses an automatic completion and restoration method for a certificate photo and equipment capable of processing pictures, belongs to the field of image processing, and preliminarily generates an incomplete region through a pre-trained incomplete region segmentation model. And a high-precision natural image matting model is combined to carry out matting, a transparent background portrait image containing a transparency channel is generated, and the incomplete area is further accurately positioned through comparison of the two images. And repairing is carried out in combination with a stable diffusion biograph model, and a repairing result is identified by adopting a certificate rationality identification model, so that the repairing precision can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an automatic completion and repair method and repair device for a certificate photo. Background Art

[0002] With the application of deep learning based on god-level networks, the effect of image restoration technology is getting better and better.

[0003] ID photos are widely used in practical applications. Since ID photos are generated based on different original images, the image information is incomplete, which affects the use effect and recognition accuracy. For example, the photos taken by students through mobile phone selfies will have missing information after being converted into standard photos. If the distance is too close, the shoulders and other parts may be missing.

[0004] Existing image restoration methods usually rely on manual restoration or processing based on simple algorithms, which have some limitations. Therefore, a large number of graduates' certificates still need to be photographed by specialized photography agencies, which is time-consuming and laborious.

[0005] In recent years, with the development of deep learning technology, semantic segmentation, image restoration and generative models based on deep learning have gradually become important tools for image restoration. Such technologies can effectively analyze image semantics and structural information and achieve more natural image restoration effects.

[0006] However, due to the uniqueness of ID photos, there are few related studies on the specific scenario of ID photo restoration, and the following technical problems are still faced: the precise positioning of the damaged area and the rationality of the restoration results are not ideal. Summary of the invention

[0007] The present invention is proposed to alleviate or solve at least one aspect or at least one point of the above problems.

[0008] The present invention provides a method for automatically completing and repairing a certificate photo, which is characterized by comprising the following steps:

[0009] S1: Input or collect a picture of a defective ID photo, and generate a defective area map of the ID photo based on a pre-trained defective area segmentation model;

[0010] S2: The portrait and background in the ID photo are cut out using a high-precision natural image cutout model to generate a transparent background portrait image with a transparency channel;

[0011] S3: Calculate and generate a final incomplete area mask map based on the incomplete area map and the transparent background portrait map containing a transparency channel, so as to locate the area to be repaired.

[0012] S4: Use the incomplete region mask image to repair the image using the image repair technology of the stable diffusion Vincent graph model to complete the incomplete region and generate a repaired image.

[0013] Preferably, the step S3 includes: step S31: extracting a transparent channel from the transparent background portrait image; converting the ID photo image into a grayscale image to facilitate comparison with the transparency channel.

[0014] Step S32: using the transparent channel in the transparent background portrait image and the grayscale distribution of the defective area map of the defective ID photo to compare and find the inconsistent area;

[0015] Step S33: according to the compared difference value, a threshold is set to generate a binary mask image.

[0016] Preferably, step S33 further includes: using an erosion operation to balance the mask image, eliminate noise points and connect broken areas;

[0017] Performing a connected region analysis on the mask image, removing excessively small noise, and retaining the main defective area; outputting a binary final defective area mask image with the same size as the input image, and locating the defective area.

[0018] Preferably, step S32 includes: normalizing the grayscale value of the incomplete area of ​​the ID photo to the [0,1] interval, a larger difference indicates that the area may be incomplete; extracting the edges in the ID photo, comparing them with the edges of the transparent background portrait image, and finding broken or missing areas.

[0019] Preferably, in step S2: the backbone network of the high-precision natural image cutout model adopts a high-resolution network, which is also used as a shared encoder;

[0020] The output of the last stage of the backbone network is fed into the pyramid pooling module to obtain richer semantic context information;

[0021] A branch design is adopted to clarify the semantic prediction and detail prediction tasks; in the semantic context branch, the feature map obtained by the pyramid pooling module and the downsampled feature map of the shared encoder are used as the input of the semantic context.

[0022] Preferably, step S2 also includes: splicing the semantic context feature map from the semantic context branch and the detail feature map of the high-resolution detail branch, and sending them to a process including convolution, batch normalization, rectified linear unit activation function and convolution, batch normalization, S-shaped activation function to generate a guide map.

[0023] Preferably, step S2 further includes: performing a dot multiplication and addition operation on the guide map and the detail feature map of the high-resolution detail branch to generate a feature map of the next stage of the high-resolution detail.

[0024] Preferably, step S5 also includes: the ID photo rationality identification model adopts a lightweight network architecture to construct the ID photo rationality identification model, and the ID photo rationality identification model is trained according to the sorted reasonable and unreasonable ID photo data and by adjusting the number of classifications, feature extraction modules, learning rate adjustment strategies, and optimizer selection.

[0025] Preferably, step S1 includes: S11: collection of incomplete image data and preprocessing of the data;

[0026] S12: Construction of the incomplete region segmentation model;

[0027] S13: Optimizing the incomplete region segmentation model according to convolutional neural network parameter analysis;

[0028] S14: Inputting the incomplete image into the trained incomplete region segmentation model.

[0029] Preferably, S12 further includes: during training, calculating the loss of the feature maps outputted in the previous stages to improve the ability of the low-level convolution to extract image details. During inference, directly fusing the feature maps of the low-level convolution with the feature maps of the high-level convolution containing more semantic information.

[0030] Preferably, the method further comprises step S5: using a pre-trained ID photo rationality identification model to identify the restored image:

[0031] If the restoration result does not meet the standard, the image is further generated to obtain the final restored image that passes the rationality identification; if the restoration result meets the standard, the final restored image is confirmed.

[0032] Preferably, further generating the image includes: using the image restoration technology of the stable diffusion Vincent graph model to restore the restored image again, completing the incomplete area, generating a restored graphic, and then returning to step S5 until the restoration result meets the standard.

[0033] The present invention also provides a device capable of processing images, wherein a readable storage medium is provided therein and a computer program is stored thereon, wherein the computer program, when executed, implements any one of the methods described in claims 1 to 9.

[0034] According to the characteristics of the ID photo, the present invention preliminarily generates the defective area through the pre-trained defective area segmentation model. It also combines the high-precision natural image cutout model to cut out the image, and generates a transparent background portrait image with a transparency channel. By comparing the two, the defective area is further accurately located. It is also combined with the stable diffusion Wensheng map model for repair, and the ID rationality identification model is used to identify the repair results, which can significantly improve the accuracy of the repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The present invention is a flowchart of automatically completing and repairing a damaged ID photo according to an exemplary embodiment of the present invention.

[0036] Figure 2 A flowchart for constructing a defective region segmentation model according to an exemplary embodiment of the present invention.

[0037] Figure 3 A diagram of a short-term densely connected network structure according to an exemplary embodiment of the present invention.

[0038] Figure 4 A schematic diagram of a simple pyramid pooling module according to an exemplary embodiment of the present invention.

[0039] Figure 5 A schematic diagram of a unified attention fusion module of an exemplary embodiment of the present invention.

[0040] Figure 6 This is a network architecture diagram of a high-precision natural image matting model according to an exemplary embodiment of the present invention.

[0041] Figure 7 A guide flow structure diagram of an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall inventive concept of the present invention, and should not be construed as a limitation of the present invention. In the present invention, the same reference numerals represent the same or similar components.

[0043] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present invention.

[0044] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0045] In order to enable those skilled in the art to use the contents of the present invention, the following exemplary embodiments may be provided in combination with specific application scenarios, specific systems, parameters of devices and components, and specific connection methods. However, for those skilled in the art, these embodiments are only examples, and the general principles defined herein may be applied to other embodiments and application scenarios without departing from the spirit and scope of the present invention.

[0046] According to an exemplary embodiment of the present invention: This example provides a method for automatically completing and repairing a damaged ID photo, such as Figure 1 As shown, the method includes the following steps: Step S1: Input a defective ID photo and use a pre-trained defective region segmentation model to generate a defective region map of the ID photo. The defective region segmentation model is a computer vision model used to identify defective regions in an input image. It learns the features of defective regions by training with annotated images so as to accurately segment defective regions in new images. Figure 2 A flowchart illustrating the present invention training a model based on the incomplete region segmentation to segment the incomplete region of the incomplete ID photo. It may include step S11: first, the incomplete image data is collected and the data is preprocessed. Use an image annotation tool to mark the incomplete region in the incomplete image as white, and other regions as black. This step ensures that the model can learn the characteristics of the incomplete region and save the annotation results. Process the annotated image, convert the non-black pixel values ​​in the annotated image to 1, traverse the annotated image pixels, set all non-black pixel values ​​to 1, and finally generate a mask image that will be used to train the model. The incomplete image and the corresponding annotated image are organized into a training data set. In order to evaluate the performance of the model, the data set is divided into a training set and a validation set to ensure that the training set contains enough samples to improve the generalization ability of the model.

[0047] Step S12: Building a defective region segmentation model. According to the annotation accuracy and consistency of the training data set, the defective region segmentation model of the present invention is trained using real-time semantic segmentation technology to accurately extract the defective region. Through the real-time semantic segmentation model, real-time processing can be achieved while ensuring accuracy. During the training process, the model learns based on a large amount of annotated data, so that it can accurately identify and segment the boundaries of the defective region and adapt to different images and various defective regions. This design not only improves the efficiency of segmentation, but also ensures the fluency and reliability of the model in practical applications. Specifically as follows: The real-time semantic segmentation technology is a computer vision technology that aims to classify images pixel by pixel while maintaining sufficient processing speed to achieve real-time application. Semantic segmentation refers to dividing each pixel in the image into a certain class of objects and generating a corresponding segmented image. Real-time means that the segmentation processing speed of the model is fast enough to respond at the millisecond level.

[0048] The real-time semantic segmentation model of the present invention adopts a lightweight semantic segmentation model based on a convolutional neural network, aiming to achieve real-time segmentation through a streamlined network structure and an innovative module combination. The structure of the lightweight semantic segmentation model includes an encoder, a decoder and a segmentation head. The lightweight semantic segmentation model adopts a simple pyramid pooling module to aggregate global context information at a lower computational cost, thereby improving the model's perception of the environmental background and enhancing the segmentation performance. At the same time, a flexible lightweight decoder is used to reduce the computational burden of traditional decoders and improve reasoning efficiency. In addition, the lightweight semantic segmentation model also integrates a unified attention fusion module as a key component for enhancing feature representation. The module generates weights using spatial and channel attention mechanisms, and fuses input features with these weights to help the model capture key semantic information more effectively. Specifically as follows: The encoder uses the backbone network as the basic backbone network to extract high-level semantic features from the input image. Through a series of convolutional layers, pooling layers, and activation functions, the spatial dimension [H, W] of the image is gradually reduced, and the semantic information in the image is enhanced, that is, the number of feature map channels is increased. The task of the encoder is to convert the original image into a low-resolution feature map with rich semantic information, including information such as the object, texture, and structure of the image. The backbone network of the present invention is a short-term densely connected network, which is a lightweight convolutional neural network that aims to improve the model inference speed while ensuring the effectiveness of feature extraction. Combined with Figure 3 As shown in Figure 1, the backbone network of the short-term densely connected network is divided into 6 stages, of which the first 5 stages are mainly used as the backbone for feature extraction and segmentation. The output features of each stage gradually reduce in size, while the number of channels gradually increases. The size of the output feature map of the i-th stage is H×W of the original input The number of channels of the feature map increases with the number of stages, and the number of channels in the i-th stage is 16×2 i. The number of channels is gradually increased in each stage to enhance the network's ability to represent features. When the number of channels of the output feature map of the second stage is greater than 64, a "final convolution layer" is added to the output of the fifth stage. The function of this layer is to ensure that the number of channels of the feature map output by the fifth stage is not less than 1024, so as to better perform subsequent processing. A short dense connection module is used inside each stage. The core design of this module is to achieve more effective information transfer through dense connections. The receptive field of the low-level convolution in the module is small, so more channels are required to extract detailed information. The receptive field of the high-level convolution is larger, and only a small number of channels are required to extract sufficient semantic information. A detailed guidance training branch is added. During training, the loss is calculated for the feature maps output by the previous stages to improve the ability of the low-level convolution to extract image details. During inference, the feature map of the low-level convolution is directly fused with the high-level convolution feature map containing more semantic information. The calculation process is represented by formula (1):

[0049]

[0050] L detail represents the detail loss function; p d Represents the detailed information of each pixel or region in the predicted distribution or feature map; g d represents the true label or target value; L dice represents the Dice loss function; L bce represents the binary cross entropy loss; Represents the predicted value of the i-th pixel position; The true label value of the i-th pixel position; d represents the detail layer; H×W represents the resolution of the feature map; ε represents the smoothing term to prevent the denominator from being zero; the decoder is the part corresponding to the encoder, which is responsible for restoring the low-resolution feature map generated by the encoder to the resolution of the original image and performing pixel-level classification. The decoder usually includes upsampling layers and convolutional layers, which are used to gradually restore the details of the feature map and generate a feature map of the same size as the input image. The decoder maps the semantic features extracted by the encoder to the pixel level in order to classify each pixel. The segmentation head is a fully connected layer or a convolutional layer, which is used to convert the feature map output by the decoder into the final segmentation structure. The simple pyramid pooling module is as follows Figure 4 As shown in Figure 1, the pyramid pooling module is used to fuse the input features. The pyramid pooling module contains three global average pooling operations, and the corresponding pooling kernel sizes are 1x1, 2x2, and 4x4. The output features are subjected to 1x1 convolution and upsampling operations. These upsampled features are added together and a 3x3 convolution operation is applied to generate fine features.

[0051] The flexible lightweight decoder of the present invention is a module for generating high-quality output feature maps, which aims to accelerate the reasoning of the model by reducing the amount of calculation and the amount of parameters, and can adapt to a variety of input conditions and task requirements. The main function of the flexible lightweight decoder is to gradually reduce the number of channels of the feature while gradually increasing the spatial size [H, W] of the feature map during the operation of the decoder, balancing the computational complexity of the encoder and the decoder, making the overall model more efficient.

[0052] The unified attention fusion module of the present invention is as follows Figure 5 As shown in Figure 1, it is a structure for fusing multiple feature maps. It uses the spatial and channel attention mechanism to generate a weight matrix α, and multiplies the input features by the weight matrix to obtain the fused features. This enables the model to better capture the key semantic information in the image and improve the segmentation accuracy. Formula (2) represents its calculation process:

[0053]

[0054] F high represents the output of the deep module; F low Indicates the corresponding part of the encoder; F up It is represented by the upsampled feature; α represents the attention weight coefficient; Upsample(F high ) represents the high-resolution feature map F high The upsampling operation; (1-α) reverse attention weight represents the low-resolution feature map F during the fusion process low The spatial attention module generates a weight using the spatial relationship between pixels, which represents the importance of each pixel in the input feature.

[0055] The channel attention module uses the relationship between channels to generate weights, which indicate the importance of each channel in the input features. The channel attention module uses average pooling and maximum pooling operations to compress the spatial dimensions of the input features.

[0056] Step S13: The present invention optimizes the incomplete region segmentation model according to the convolutional neural network parameter analysis, and evaluates the size and computational efficiency of the overall model by calculating the parameter amount and floating-point operation amount of each layer. In addition, the sorted incomplete images and annotated pictures are used as training data, combined with optimized hyperparameters such as learning rate and batch size to accelerate model convergence, and a dynamic learning rate adjustment strategy is adopted to adapt to the training stage. Combined with data enhancement techniques such as random cropping, rotation and flipping, the diversity of training data is increased and the generalization ability of the model is improved. Through the above convolutional network parameter analysis method, the present invention can effectively optimize the performance of the incomplete region segmentation model in the incomplete region segmentation task, reduce computational complexity, and improve segmentation accuracy, thereby achieving real-time and efficient incomplete region segmentation.

[0057] Step S14: Input the incomplete image into the trained incomplete region segmentation model. The model will use its convolutional neural network structure to perform forward reasoning on the image, and use multiple convolutional layers to extract multi-scale features of the image, highlighting the edges, i.e., the details of the incomplete region. The incomplete region segmentation model reduces the resolution of the feature map by downsampling to obtain a wider range of contextual information, and then restores it to the resolution of the original image by upsampling to accurately locate the incomplete region. After a series of convolution, downsampling, and upsampling operations, the incomplete region segmentation model outputs a segmentation map of the same size as the input image, and the value of each pixel represents the probability that the position belongs to the incomplete region. The probability map is converted into a binary segmentation map through a threshold, and finally a segmentation map of the incomplete region is generated.

[0058] Step S2: Use a high-precision natural image cutout model to accurately cut out the portrait and the background, and generate a transparent background portrait image with a transparency channel. The high-precision natural image cutout model is mainly used to accurately separate the foreground from the background and generate a high-quality transparent background portrait image.

[0059] like Figure 6 As shown in the figure, the backbone network of the high-precision natural image matting model adopts a high-resolution network, which is also used as a shared encoder. The output of the last stage of the backbone network is sent to the pyramid pooling module to obtain richer semantic context information. A branch design is adopted to clarify the semantic prediction and detail prediction tasks. The semantic context branch, the feature map obtained by the pyramid pooling module and the feature map of the shared encoder after 1 / 32 times downsampling are used as the input of the semantic context.

[0060] The semantic context branch consists of five blocks, each of which includes convolution, batch normalization, activation function and a bilinear upsampling module. In addition, the output of the semantic context branch is also used for the supervision of the semantic segmentation task, which includes three categories: foreground, background and transition area.

[0061] The high-resolution detail branch, the intermediate features of the shared encoder are upsampled and concatenated as the initial input of the high-resolution detail branch. At the same time, the guidance flow is introduced from the 1 / 16 and 1 / 4 resolution intermediate features of the semantic context branch to inject semantic information to help the high-resolution details obtain more detailed image details.

[0062] like Figure 7 As shown. The semantic context feature map from the semantic context branch and the detail feature map of the high-resolution detail branch are concatenated and sent to a process including convolution, batch normalization, rectified linear unit activation function and convolution, batch normalization, sigmoid activation function to generate a guide map. Then, the guide map is dot-multiplied and added with the detail feature map of the high-resolution detail branch to generate a feature map of the next stage of the high-resolution detail.

[0063] Formulas (3)-(5) represent its loss function:

[0064]

[0065] Where L s represents the total cross entropy loss; c represents the category index; i represents the index of the pixel; Ω represents the location set of all pixels in the image; Represents the true label value of the i-th pixel in category c; Represents the predicted probability of the i-th pixel in category c; express The natural logarithm of Ω f Indicates that it belongs to the foreground category; Ω b Indicates that it belongs to the background category; Ω t represents the set of pixel positions belonging to the third category;

[0066]

[0067] L d represents the total loss function; Ω t Represents a set of pixels in the target area; i is a pixel index; Represents the transparency loss of the i-th pixel; Represents the gradient loss of the i-th pixel; represents the predicted transparency value of the i-th pixel, generated by the model; The true transparency value of the i-th pixel; ε represents a small positive value to avoid the square difference being zero; represents the gradient of the predicted transparency value of the i-th pixel; Represents the gradient of the true transparency value of the i-th pixel;

[0068]

[0069] L f represents the overall loss function; i represents the pixel index; Ω represents the image pixel set; Represents the transparency loss of the i-th pixel; Represents the gradient loss of the i-th pixel; represents the composite loss of the i-th pixel; represents the predicted composite image value of the i-th pixel; represents the true composite image value of the i-th pixel; ε represents a small positive value to avoid the square difference being zero; represents the predicted transparency value of the i-th pixel; represents the true foreground pixel value of the i-th pixel; Represents the true background pixel value of the i-th pixel; Indicates the probability that a pixel belongs to the background;

[0070] The high-resolution network structure consists of multiple stages, each of which is processed by network branches of different resolutions. At each stage, the network gradually optimizes and fuses features of different scales through cross-branch information interaction, thereby enhancing the expressive power of features. As the depth of the network increases, the detailed information and global semantics in the image can be better captured. The high-resolution network maintains high-resolution features at each stage, while also generating and fusing low-resolution feature maps to obtain richer semantic information. An effective multi-scale feature fusion strategy is adopted to interactively fuse feature maps of different resolutions. Information is exchanged between low-resolution feature maps and high-resolution feature maps, so that each branch can not only learn lower-level detail information, but also integrate global semantic information from other scales. The shared encoder refers to the design in which multiple tasks or multiple modules share the same part of the encoding network structure in the high-precision natural image matting model. When upstream and downstream tasks or multiple branches share the same encoder, the network layer of the encoder is used by multiple tasks, thereby reducing the use of computing resources and parameters, and helping to share information between tasks, improving learning efficiency and generalization ability.

[0071] The pyramid pooling block module pools the feature maps at different scales through the pyramid pooling operation, and then splices or fuses the pooled features to obtain multi-scale semantic information. It helps the model to better understand and segment objects of different scales in the image. The pyramid pooling operation can effectively expand the receptive field of the neural network, enabling the model to better capture the global and local information in the image. The pyramid pooling block module can be applied on the output feature map of the decoder to provide more contextual information for the segmentation task. This helps the model to more accurately classify pixels into different semantic categories, thereby improving the accuracy and generalization ability of segmentation.

[0072] Step S21: Input the incomplete ID photo into the high-precision natural image cutout model. Using the high-precision natural image cutout model, the portrait can be accurately separated from the background to generate an image file containing a transparency channel, in which the background is completely transparent and only the clear and complete portrait part is retained. This process is that the high-precision natural image cutout model performs semantic segmentation and edge refinement on the foreground and background in the image to ensure that the output image also has a highly natural filtering effect in details such as hair and clothing edges. Finally, the transparent background portrait image is generated and saved in a format that supports transparency channels.

[0073] Step S3: Calculate and generate a final mask image of the defective area of ​​the ID photo and the transparent background portrait image to locate the area that needs to be repaired;

[0074] Step S31: Extract a transparency channel from the transparent background portrait image to represent the transparency information of the foreground area, and the range is usually [0,1]. Transparency = 1 represents a completely opaque area, and transparency = 0 represents a completely transparent area. By extracting the transparency channel, we can accurately know which areas in the image are foregrounds and which areas are backgrounds. The closer the transparency value is to 1, the higher the possibility that the area belongs to the foreground, and the closer it is to 0, the more likely the area is to be the background. In this way, the transparency channel provides key information for separating the foreground and background of the image. Convert the ID photo image into a grayscale image to facilitate comparison with the transparency channel.

[0075] Step S32: Use the transparent channel in the transparent background portrait image to compare the grayscale distribution of the incomplete area of ​​the incomplete ID photo image to find inconsistent areas. The grayscale value of the incomplete area of ​​the ID photo needs to be normalized to the [0,1] interval. A larger difference indicates that the area may be incomplete. Extract the edges in the ID photo and compare them with the edges of the transparent background portrait image to find broken or missing areas.

[0076] Step S33: According to the difference value, a threshold is set to generate a binary mask map. 1 in the mask map represents a defective area, and 0 represents a normal area. The mask map is balanced using an erosion operation to eliminate noise points and connect broken areas. The mask map is analyzed for connected areas to remove excessively small noise and retain the main defective areas. A binary mask map with the same size as the input image is output to locate the defective area.

[0077] Step S4: Using the incomplete region mask map, the image is repaired using the image repair technology of the stable diffusion Wensheng graph model to complete the incomplete region and generate a preliminary repaired ID photo; the stable diffusion Wensheng graph model is capable of performing Wensheng graph and image generation tasks, and consists of three parts: a variational autoencoder, a U-type network and a text encoder. Instead of learning to denoise image data, the variational autoencoder is trained to convert the image into a low-dimensional latent space. The process of adding and removing Gaussian noise is applied to this latent representation, and then the final denoised output is decoded into the pixel space. During the forward diffusion process, Gaussian noise is iteratively applied to the compressed latent representation. Each denoising step is completed by a U-type network architecture containing a residual network backbone, and the latent representation is obtained by denoising from the forward diffusion in the reverse direction. The variational autoencoder decoder generates the output image by converting the representation back to the pixel space. The denoising step can be conditioned on a text string, an image, or some other data. The encoding of the conditioned data is exposed to the denoising U-type network architecture through a cross-attention mechanism. The U-shaped network structure is an encoder-decoder structure. Such a structure is simple and efficient. The encoder module is responsible for feature extraction and learning, which can increase the robustness to disturbance noise, reduce the risk of overfitting, reduce the amount of calculation and increase the size of the receptive field. The decoder is responsible for restoring the feature map to the original resolution, and the key move of jump connection integrates the shallow position information and deep semantic information. Jump connection is used as a bridge between the encoder and decoder modules for feature fusion, and the shallow position information and deep semantic information are spliced. The U-shaped network is a fully convolutional neural network. The last layer of the network uses 1x1 convolution, which completely replaces the fully connected layer, so that the input size of the model is no longer limited. Each time after the convolution operation, the feature map will be downsampled to a certain extent. Combining convolution and maximum pooling can extract features from the feature map while calmly downsampling.

[0078] The cross-attention mechanism is used to establish associations between two different modalities or different data sources, extract relevant information and fuse features. Information interaction and alignment are mainly achieved through the relationship between queries, keys and values.

[0079] Step S41: Using the image restoration technology of the stable diffusion Vincent graph model, for the mask map of the incomplete area of ​​the input image, the stable diffusion Vincent graph model will combine the pixel information of the surrounding undamaged parts and the global context semantics to generate high-quality restoration content. In this process, the stable diffusion Vincent graph model takes into account the texture details, lighting conditions and consistency of the surrounding environment of the restoration area to ensure that the generated content is seamlessly integrated with the original image. In addition to referring to the content of the ID photo, the model is further guided to generate restoration results that meet the requirements in combination with the provided text description. During the restoration process, the stable diffusion Vincent graph model not only generates pixel information of the missing area, but also performs detailed processing on the boundary area to ensure that the transition between the mask edge and the original image is natural and there are no obvious traces of splicing. For complex incomplete situations, the stable diffusion Vincent graph model may iterate multiple times to generate and optimize, and gradually improve the restoration quality. Finally, the preliminary restored ID photo is output.

[0080] Step S5: Use the pre-trained ID photo rationality identification model to identify the ID photo image automatic repair image. If the repair result does not meet the standard, further generate the image. Obtain the final repair image that passes the rationality identification. The ID photo rationality identification model uses a lightweight network architecture to construct the ID photo rationality identification model. The ID photo rationality identification model is trained according to the sorted reasonable and unreasonable ID photo data and by adjusting the number of classifications, feature extraction modules, learning rate adjustment strategies, and optimizer selection. The lightweight network architecture is a lightweight convolutional neural network architecture that aims to improve the model reasoning speed and computational efficiency. Specifically, it is a variant of the shuffle network V2 architecture, indicating that the size of the network is reduced to 0.25 times the original. The lightweight network contains multiple stages, each of which contains multiple basic units. The basic unit usually includes a 1x1 convolution, a 3x3 depth convolution, and another 1x1 convolution. A channel shuffle operation is added after the 1x1 convolution, and no activation function is used after the 3x3 depth convolution. This version uses fewer parameters and computing resources while maintaining a certain classification accuracy. It uses point-by-point grouped convolution, which groups the feature maps of the input layer and then uses different convolution kernels for each group, thereby reducing the amount of calculation. Channel shuffling is introduced, which allows information exchange between feature maps of different groups, thereby enhancing the network's feature extraction capabilities.

[0081] Step S51: Use the ID photo rationality identification model to automatically detect and evaluate the ID photo image that has been initially repaired. The model comprehensively analyzes the image to check whether the image meets the standard requirements for ID photos, including but not limited to facial position, light balance, background consistency, and image clarity. If the repair result does not meet the rationality standard, the ID photo rationality identification model will feedback the specific problems that do not meet the standards, and further generate and optimize the image based on these feedback information. In the process of further generation, the stable diffusion Wensheng graph model will make targeted adjustments based on the feedback of the ID photo rationality identification model. After multiple rounds of generation and repair, a final repair image that meets the ID photo standards and passes the ID photo rationality identification is finally generated. This process ensures that the final ID photo meets both technical standards and the visual and specification requirements in actual use, achieving high-quality repair results.

[0082] The automated, high-quality restoration method for the incomplete ID photo problem proposed in the present invention combines deep learning semantic segmentation, transparent background cutout, diffusion model restoration and rationality identification technology. It can not only automatically identify and repair the incomplete areas, but also ensure the rationality and naturalness of the restoration results. It has high practical value and application prospects.

[0083] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes may be made to the embodiments and combinations of elements may be made without departing from the principles and spirit of the invention, the scope of the invention being defined by the appended claims and their equivalents.

Claims

1. A method for automatically completing and repairing a certificate photo, characterized in that: The method comprises the following steps: S1: Input or collect a picture of a defective ID photo, and generate a defective area map of the ID photo based on a pre-trained defective area segmentation model; S2: The portrait and background in the ID photo are cut out using a high-precision natural image cutout model to generate a transparent background portrait image with a transparency channel; S3: calculating and generating a final defective area mask map according to the defective area map and the transparent background portrait map containing a transparency channel, so as to locate the area to be repaired; S4: Use the incomplete region mask image to repair the image using the image repair technology of the stable diffusion Vincent graph model to complete the incomplete region and generate a repaired image.

2. The repair method according to claim 1, characterized in that: The step S3 includes: step S31: extracting a transparent channel from the transparent background portrait image; converting the incomplete ID photo image into a grayscale image to facilitate comparison with the transparency channel; Step S32: using the transparent channel in the transparent background portrait image and the grayscale distribution of the defective area map of the defective ID photo to compare and find the inconsistent area; Step S33: according to the compared difference value, a threshold is set to generate a binary mask image.

3. The repair method according to claim 2, characterized in that: Step S33 also includes: using an erosion operation to balance the mask image, eliminate noise points and connect broken areas; Performing connected region analysis on the mask image to remove excessively small noise and retain the main defective area; Output a binary final defect area mask image with the same size as the input image to locate the defect area.

4. The repair method according to claim 2, characterized in that: Step S32 includes: normalizing the grayscale value of the incomplete area of ​​the ID photo to the [0,1] interval, and a larger difference indicates that the area may be incomplete; extracting the edges in the ID photo, comparing them with the edges of the transparent background portrait image, and finding broken or missing areas.

5. The repair method according to claim 2, characterized in that: In step S2: the backbone network of the high-precision natural image cutout model adopts a high-resolution network, which is also used as a shared encoder; The output of the last stage of the backbone network is fed into the pyramid pooling module to obtain richer semantic context information; Adopt branch design to clarify semantic prediction and detail prediction tasks; In the semantic context branch, the feature map obtained by the pyramid pooling module and the downsampled feature map of the shared encoder are used as the input of the semantic context.

6. The repair method according to claim 2, characterized in that: Step S1 includes: S11: collection of incomplete image data and preprocessing of data; S12: Construction of incomplete region segmentation model; S13: Optimizing the incomplete region segmentation model according to convolutional neural network parameter analysis; S14: Inputting the incomplete image into the trained incomplete region segmentation model.

7. The repair method according to claim 2, characterized in that: The step S5 is also included: using a pre-trained ID photo rationality identification model to identify the restored image: If the restoration result does not meet the standard, the image is further generated to obtain the final restored image that passes the rationality identification; If the repair result meets the requirements, the final repaired graphics are confirmed.

8. The repair method according to claim 7, characterized in that: in, Further generating the image includes: using the image restoration technology of the stable diffusion Vincent graph model to restore the restored image again, completing the incomplete area, generating a restored image, and then returning to step S5 until the restoration result meets the standard.

9. The repair method according to claim 8, characterized in that: In step S5: the ID photo rationality identification model adopts a lightweight network architecture to construct the ID photo rationality identification model, and the ID photo rationality identification model is trained according to the sorted reasonable and unreasonable ID photo data and by adjusting the number of classifications, feature extraction modules, learning rate adjustment strategies, and optimizer selection.

10. A device capable of processing pictures, wherein a readable storage medium is provided therein and a computer program is stored thereon, characterized in that: When the computer program is executed, the method according to any one of claims 1 to 9 is implemented.

Images