Image quality improvement method, device, equipment and medium
By performing degradation and interpolation processing on wafer images, high-frequency components are calculated and adversarial network models are trained to generate, the problem of unsatisfactory image quality improvement effect in the prior art is solved, and higher image quality improvement accuracy and clear details are achieved Spend.
Patent Information
- Application Number
- CN202411075556.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-07
AI Technical Summary
In the prior art, when processing real wafer image data, the image quality improvement effect is not ideal, resulting in insufficient improvement accuracy.
By acquiring the low-quality images and high-quality images of the target wafer, degradation processing and interpolation processing are performed respectively, the first synthetic low-quality images and the second synthetic low-quality images are generated, the high-frequency components between the two are calculated, and the high-quality components are adjusted. The frequency component input is iteratively trained based on the image quality improvement model of the generated adversarial network until the trained model meets the expected performance.
Improve the accuracy of image quality improvement, generate clearer, more detailed and more beautiful images, and meet the quality improvement needs of target low-quality images.
Smart Images

Figure CN118608438B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of wafer image processing technology, and in particular to an image quality improvement method, device, equipment and medium. Background Art
[0002] Wafer is the most commonly used semiconductor device, and its main diameter includes but is not limited to smaller sizes such as 150mm, 200mm and 300mm. During the wafer production process, the wafer image needs to be recognized in order to convey information to the user. However, as the resolution of digital terminals continues to increase, the demand for higher resolution images is also increasing. This is because low-resolution images, if placed on a high-resolution screen, will make the visual effect very poor.
[0003] Existing technologies usually use convolutional neural networks to improve image quality. When constructing training sets, they often use methods that simulate image quality degradation, such as adding Gaussian noise to high-quality images to generate noisy wafer images, or using bilinear interpolation methods to generate low-resolution wafer images. However, since the causes and processes of image quality degradation in the real world are far more complicated than these simulation methods, the improvement effect is not ideal when processing real wafer image data, which affects the accuracy of image quality improvement. Summary of the invention
[0004] Based on this, it is necessary to provide an image quality improvement method, device, equipment and medium to address the above technical problems, so as to solve the problem that the existing technology cannot improve the improvement accuracy of image quality, resulting in unsatisfactory improvement effect when processing real wafer image data.
[0005] A first aspect of an embodiment of the present application provides a method for improving image quality, the method comprising:
[0006] Acquire low-quality and high-quality images of the target wafer;
[0007] Performing degradation processing and interpolation processing on the high-quality image respectively to generate a first synthesized low-quality image and a second synthesized low-quality image, wherein the first synthesized low-quality image is obtained based on the high-quality image processed by a trained downsampling network;
[0008] calculating a high frequency component between the first synthesized low-quality image and the second synthesized low-quality image;
[0009] The high-frequency component is input into an image quality improvement model based on a generative adversarial network, and iterative training is performed until the trained image quality improvement model meets the expected performance, thereby obtaining a target image quality improvement model, wherein the target image quality improvement model is used to improve the image quality of a target low-quality image.
[0010] A second aspect of an embodiment of the present application provides an image quality improvement device, the image quality improvement device comprising:
[0011] An acquisition module, used for acquiring a low-quality image and a high-quality image of a target wafer;
[0012] a processing module, configured to perform degradation processing and interpolation processing on the high-quality image to generate a first synthesized low-quality image and a second synthesized low-quality image, wherein the first synthesized low-quality image is obtained based on the high-quality image processed by a trained downsampling network;
[0013] A calculation module, configured to calculate a high frequency component between the first synthesized low-quality image and the second synthesized low-quality image;
[0014] A training module is used to input the high-frequency component into an image quality improvement model based on a generative adversarial network, and perform iterative training until the trained image quality improvement model meets the expected performance, thereby obtaining a target image quality improvement model, wherein the target image quality improvement model is used to improve the image quality of a target low-quality image.
[0015] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image quality improvement method as described in the first aspect when executing the computer program.
[0016] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image quality improvement method as described in the first aspect is implemented.
[0017] In summary, the present invention provides an image quality improvement method, device, equipment and medium, by acquiring a low-quality image and a high-quality image of a target wafer, performing degradation processing and interpolation processing on the high-quality image respectively, generating a first synthetic low-quality image and a second synthetic low-quality image, wherein the first synthetic low-quality image is obtained based on the high-quality image after training through a downsampling network, calculating the high-frequency component between the first synthetic low-quality image and the second synthetic low-quality image, inputting the high-frequency component into an image quality improvement model based on a generative adversarial network, and performing iterative training until the trained image quality improvement model meets the expected performance, and obtaining a target image quality improvement model, so that the target image quality improvement model improves the image quality of the target low-quality image. Compared with the prior art, the present application calculates the generated first synthetic low-quality image and the second synthetic low-quality image to obtain the high-frequency component between the two synthetic low-quality images, and trains the image quality improvement model through the high-frequency component to obtain the target image quality improvement model, thereby satisfying the quality improvement of the target low-quality image and improving the image quality improvement accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0019] Figure 1 is a schematic diagram of an application environment of an image quality improvement method provided by an embodiment of the present invention;
[0020] Figure 2 is a flow chart of a method for improving image quality provided by an embodiment of the present invention;
[0021] Figure 3 is a structural schematic diagram of an image quality improvement device provided by an embodiment of the present invention;
[0022] Figure 4 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.
[0025] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
[0026] As used in the present specification and the appended claims, the term “if” may be interpreted as “when” or “uponce” or “in response to determining”, depending on the context. Similarly, the phrases “if it is determined” or “if matched to [described condition or event]” may be interpreted as meaning “upon determination” or “in response to determination” or “uponce matched to [described condition or event]” or “in response to matching to [described condition or event]”, depending on the context.
[0027] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0028] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0029] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0030] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.
[0031] An image quality improvement method provided by an embodiment of the present invention can be applied in Figure 1In the application environment, it is designed to solve common defects in images, such as noise, blur and low resolution, so as to improve the accuracy of image quality improvement to produce clearer, more detailed and more beautiful images. Among them, the measuring device communicates with the server through the network. The measuring device is used to measure the image quality in the wafer and send the image quality to the server, which can use the image as a training sample image and input the trained target image quality improvement model, thereby improving the image quality improvement accuracy. Among them, the measuring device can be but not limited to various industrial cameras, industrial sensors, laser measuring instruments, scanning measuring instruments, etc. The server can be implemented with an industrial computer, an independent server, or a server cluster consisting of multiple servers.
[0032] See also Figure 2 , is a flow chart of an image quality improvement method provided by an embodiment of the present invention. The image quality improvement method can be applied to Figure 1 Servers in Figure 2 As shown, the image quality improvement method may include the following steps:
[0033] S201: Acquire a low-quality image and a high-quality image of a target wafer.
[0034] In this embodiment, two camera systems with different objective lens magnifications are used to acquire images with different resolutions and quality characteristics. In the same wafer area, low-quality images are obtained by collecting the upper half of the wafer with a low-magnification objective lens, and high-quality images are obtained by collecting the lower half of the wafer with a high-magnification objective lens. That is, low-quality images and high-quality images are collected using cameras with high and low objective lens magnifications, respectively. The low objective lens magnification is mainly responsible for collecting low-resolution images, which are usually referred to as "low-quality images". In this setting, the parameters of image acquisition include lower spatial resolution and higher noise levels. Generally, low-quality images usually contain characteristics such as camera inherent noise, transmission noise, optical blur, and low resolution, among which, camera inherent noise: random noise generated by the inherent characteristics of camera sensors and electronic components; transmission noise: noise introduced during the transmission of image data from the camera to the processing system, which may be caused by factors such as cable quality and electromagnetic interference; optical blur: due to the limitations of the optical system, such as lens distortion or inaccurate focus, the image details are blurred; low resolution: due to the limitations of the low-magnification objective lens, the details and clarity of the image are low, resulting in the loss of detail information in the image. High objective magnification is used to acquire high-resolution images, which are often referred to as "high-quality images." In this configuration, image acquisition parameters include higher spatial resolution and lower noise levels, which means that the images provide more detail and higher clarity, suitable for detailed analysis and measurement.
[0035] S202: Perform degradation processing and interpolation processing on the high-quality image respectively to generate a first synthesized low-quality image and a second synthesized low-quality image, wherein the first synthesized low-quality image is obtained based on the high-quality image after being processed by a trained downsampling network.
[0036] As an optional embodiment, performing degradation processing on the high-quality image to generate a first synthesized low-quality image includes:
[0037] Processing the high-quality image to generate an initial synthesized low-quality image;
[0038] Training a pre-established downsampling network using the low-quality image and the initial synthesized low-quality image to obtain a trained downsampling network;
[0039] The high-quality image is input into the trained downsampling network to obtain a first synthesized low-quality image.
[0040] Specifically, after obtaining a high-quality image, an initial synthesized low-quality image is generated by performing degradation processing on the high-quality image, wherein the initial synthesized low-quality image can be obtained by the following equation:
[0041] ;
[0042] in, Represented as the initial synthesized low-quality image after degradation, Represented as the input high-quality image, Represents the kernel used in the blur operation on the input high-quality image, Represents high-quality images times downsampling, It is expressed as adding Gaussian noise with a standard deviation of σ to the high-quality image. It is expressed as JPEG lossy compression of high-quality images. Through the above method, the initial synthesized low-quality image corresponding to the high-quality image can be obtained. Among them, the downsampling network uses 23 residual blocks, each residual block contains two convolutional layers and a ReLU activation function, and the convolution kernel size of each convolutional layer is 3 3 two-dimensional convolution, the number of channels is 64. ReLU (Rectified Linear Unit) activation function is a commonly used nonlinear activation function in neural networks. It can help neural networks learn complex features and prevent the problem of gradient disappearance, thereby improving training efficiency and model performance. In order to reduce the spatial resolution of features, the downsampling network uses a bilinear upsampling operation (Bilinear Upsampling) to map low-resolution features back to the image domain through two convolutional layers.
[0043] In an embodiment of the present application, a pre-established downsampling network is trained using low-quality images and initial synthetic low-quality images, that is, the initial synthetic low-quality images are used as input and the real low-quality images are used as target outputs. The model is entered for training, and a suitable loss function (such as MSE, L1 loss, etc.) is used to minimize the difference between the network output and the real image. The network parameters are iteratively optimized until the model achieves satisfactory performance on the validation set, thereby obtaining a trained downsampling network. The initial synthetic low-quality image is then input into the trained downsampling network, and the initial synthetic low-quality image is converted into a 64-channel feature map through downsampling. Next, feature extraction and feature mapping are performed through residual blocks. The larger the number of residual blocks, the deeper the network depth, and the stronger the feature extraction and mapping capabilities, but it also consumes more computing resources. The feature map output by the residual block passes through a 3 The 64-channel feature map before entering the residual network is added after the convolution of 3, and the output feature map is input into the channel attention module. The attention mechanism is applied to assign different weights to different channels of the feature map, and then a 4-fold upsampling operation is performed to restore the feature map to the size of the input image. Finally, a 3 The convolution of 3 transforms the 64-channel feature map into a 3-channel real image for final output, thereby obtaining the first synthesized low-quality image. Through the above method, the downsampling network is trained by the low-quality image and the initial synthesized low-quality image, so that a better quality improved image can be obtained later.
[0044] As an optional embodiment, the downsampling network is trained in the following manner:
[0045] Obtain low-quality training samples and high-quality training samples;
[0046] Inputting the high-quality training samples into a generation network to obtain initial synthesized low-quality training samples, and calculating content losses of the initial synthesized low-quality training samples and the high-quality training samples;
[0047] Inputting the low-quality training samples and the initial synthesized low-quality training samples into a convolutional neural network to obtain the perceptual losses of the low-quality training samples and the initial synthesized low-quality training samples;
[0048] Inputting the low-quality training sample and the initial synthesized low-quality training sample into a discriminant network to obtain adversarial losses of the low-quality training sample and the initial synthesized low-quality training sample;
[0049] Based on the content loss, the perceptual loss and the adversarial loss, the downsampling network is trained until the sum of the content loss, the perceptual loss and the adversarial loss output by the trained downsampling network is less than a preset loss threshold.
[0050] Specifically, a camera for shooting low-resolution images and a camera for shooting high-resolution images respectively shoot the same subject as low-quality training samples and high-quality training samples for training the downsampling network. The high-quality training samples are input into the generative network to obtain initial synthetic low-quality training samples, and the content loss of the initial synthetic low-quality training samples and the high-quality training samples is calculated. Among them, the generative network is a part of the generative adversarial network. The generative adversarial network includes a generative network and a discriminative network. The generative adversarial network uses the generative network and the discriminative network to conduct mutual game learning on the training images input therein to produce a fairly good conversion image. In other words, the purpose of the generative network is to generate initial synthetic low-quality training samples with high clarity in visual perception as much as possible, and the purpose of the discriminative network is to distinguish the initial synthetic low-quality training samples generated by the generative network from the real low-quality training samples. Since there are differences between the high-quality training samples input into the downsampling network and the initial synthesized low-quality training samples output by the downsampling network, in order to improve the clarity of the images generated by the downsampling network, the initial synthesized low-quality training samples are made close to the real high-quality training samples, and then the content loss of the high-quality training samples and the initial synthesized low-quality training samples is calculated to ensure that the synthesized initial synthesized low-quality training samples are consistent with the input high-quality training samples in terms of texture details. Then the low-quality training samples and the initial synthesized low-quality training samples are input into the convolutional neural network, and the perceptual loss of the low-quality training samples and the initial synthesized low-quality training samples is calculated, so as to know the feature differences between the low-quality training samples and the initial synthesized low-quality training samples, and timely adjust the training of the downsampling network so that the training of the downsampling network meets the requirements.
[0051] Since the purpose of the discriminant network is to distinguish the initial synthetic low-quality training samples generated by the generative network from the real low-quality training samples, the initial synthetic low-quality training samples input therein and the real low-quality training samples can be identified according to the discriminant network to calculate the adversarial loss of the output initial synthetic low-quality training samples and the real low-quality training samples. The loss function is a function that maps the values of a random event or its related random variables to non-negative real numbers to represent the risk or loss of the event. In this embodiment, the loss function is used to represent the degree of difference between the real low-quality training samples and the initial synthetic low-quality training samples obtained by training, that is, the loss, so as to return the loss so that the training of the downsampling network is calculated in the direction of the loss and the loss is reduced. The downsampling network is trained according to the content loss of the initial synthetic low-quality training samples and the high-quality training samples, the perceptual loss of the low-quality training samples and the initial synthetic low-quality training samples, and the adversarial loss of the low-quality training samples and the initial synthetic low-quality training samples, until the sum of the error loss and the adversarial loss output by the trained downsampling network is less than the preset loss threshold. It should be noted that the preset loss threshold can be set according to the experience of technicians, and this embodiment does not limit the value of the preset loss threshold.
[0052] In the implementation of this application, by using the initial synthetic low-quality training samples corresponding to the low-quality training samples and the high-quality training samples to train the generative network, convolutional neural network and discriminative network in an adversarial relationship, a downsampling network that meets the requirements can be obtained without additional feature attributes, thereby improving the clarity of the images generated by the downsampling network in visual perception.
[0053] As an optional embodiment, performing interpolation processing on the high-quality image to generate a second synthesized low-quality image includes:
[0054] The high-quality image is processed based on a bicubic interpolation algorithm to generate a second synthesized low-quality image.
[0055] Specifically, the corresponding position of each target pixel in the second synthetic low-quality image in the high-quality image is determined in advance, and then the 16 pixels closest to the pixel are found. The weights of these 16 pixels are calculated by using the bicubic basis function, and the values of these pixels are multiplied by their weights, and then the sum is added to obtain the value of the target pixel. First, it is necessary to determine where each pixel in the second synthetic low-quality image should be placed, which is like determining the coordinates of each pixel on the canvas of the second synthetic low-quality image. Then, for each pixel in the second synthetic low-quality image, instead of giving it a color value directly, its "neighbors" are found in the high-quality image. These neighbors are the 16 pixels closest to this pixel, which can be imagined as a 4x4 grid around this pixel. Then, each neighbor pixel has an impact on the color value of the pixel in the second synthetic low-quality image, but the impact is different. We use the bicubic basis function to calculate the "weight" of each neighbor pixel, which is like the contribution of each neighbor to the final color, and then according to the weight of each neighbor pixel, their color values are weighted summed, which is like mixing the colors of the neighbors according to their contribution. Finally, the result of this weighted summation is the color value of the corresponding pixel in the second synthetic low-quality image.
[0056] In the implementation of this application, the interpolation process of generating a second synthetic low-quality image from a high-quality image can not only enhance the learning process of the model, but also improve its resistance to image noise, blur and other distortion phenomena, thereby performing better in real applications.
[0057] S203: Calculate a high-frequency component between the first synthesized low-quality image and the second synthesized low-quality image.
[0058] As an optional embodiment, calculating the high-frequency component between the first synthesized low-quality image and the second synthesized low-quality image includes:
[0059] Performing convolution calculation on the first synthesized low-quality image and the second synthesized low-quality image based on the Haar wavelet transform method to obtain a first texture feature corresponding to the first synthesized low-quality image and a second texture feature corresponding to the second synthesized low-quality image respectively;
[0060] The difference between the first texture feature and the second texture feature is calculated to obtain a high-frequency component.
[0061] Specifically, by selecting a specific Haar basis function, the first synthetic low-quality image and the second synthetic low-quality image are multiplied by the Haar wavelet matrix respectively to obtain wavelet coefficients, a one-dimensional transform is applied in the horizontal direction, and then a one-dimensional transform is applied again in the vertical direction. This can be decomposed into four parts: one part is the low-frequency component, and the other three parts are high-frequency components in different directions. The low-frequency coefficients represent the main characteristics of the signal, while the high-frequency coefficients represent the details and changes of the signal, thereby obtaining the high-frequency components between the first synthetic low-quality image and the second synthetic low-quality image. That is, the first synthetic low-quality image and the second synthetic low-quality image are transformed by multiplying with the Haar wavelet matrix respectively. This process is similar to filtering, in which the Haar wavelet matrix acts as a filter to extract specific frequency components in the image. First, the image is decomposed in the horizontal direction, which means that each row of the image is processed independently. By multiplying with the Haar wavelet matrix, each row is decomposed into the average (low-frequency part) and the difference (high-frequency part). Then, this process is repeated in the vertical direction to process each column. Finally, the image is decomposed into four parts:
[0062] Top left: Low-frequency components, representing the main features of the image, such as brightness and overall shape.
[0063] Upper right corner: high-frequency components in the horizontal direction, reflecting the changes in the image in the horizontal direction.
[0064] Lower left corner: High-frequency components in the vertical direction, reflecting the changes in the image in the vertical direction.
[0065] Lower right corner: The high-frequency component in the diagonal direction reflects the changes in the image in the diagonal direction.
[0066] After the Haar wavelet transform, the low-frequency component (upper left corner) contains the overall information of the image, such as the background and main shapes. The high-frequency components (upper right corner, lower left corner and lower right corner) contain the details and fast-changing parts of the image, such as edges, textures and noise. This information is more visually obvious and has a direct impact on the details and clarity of the image.
[0067] In the implementation of the present application, by calculating the high-frequency components between the first synthesized low-quality image and the second synthesized low-quality image, the image generation and processing process can be more deeply understood, thereby improving the quality of the final image and the effect of the application.
[0068] S204: Input the high-frequency component into an image quality improvement model based on a generative adversarial network, and perform iterative training until the trained image quality improvement model meets the expected performance, thereby obtaining a target image quality improvement model, wherein the target image quality improvement model is used to improve the image quality of a target low-quality image.
[0069] In an embodiment of the present application, after obtaining the high-frequency component between the first synthesized low-quality image and the second synthesized low-quality image, the high-frequency component is input into the image quality improvement model based on the generative adversarial network by adopting a deep learning method, and an image quality improvement model is trained to learn to convert the low-quality image into a high-quality super-resolution image. This model can achieve the improvement of the wafer image by inputting a low-quality image and outputting a corresponding high-quality image. For example: First, a large number of high-quality images and corresponding low-quality images need to be prepared. The low-quality image can be generated from the high-quality image by the previously described degradation model of blurring and adding noise. A deep learning image quality improvement model is established, taking the low-quality image as input and the high-quality image as the target output. Using the prepared data set, by inputting the low-quality image into the model, the network parameters are continuously adjusted so that it can accurately improve the low-quality image to the high-quality image. The difference between the model output and the real high-quality image can be evaluated using a loss function (such as MSE, SSIM, etc.), and then the network parameters are updated by the back propagation algorithm. After the training is completed, another test data set can be used to evaluate the performance of the target image quality improvement model to see if it can accurately improve the low-quality image to a high-quality super-resolution image.
[0070] In an embodiment of the present application, by inputting high-frequency components into an image quality improvement model based on a generative adversarial network and performing an iterative training process, a target image quality improvement model can be obtained that can improve low-quality images to high-quality images of any size, thereby improving image quality and enhancing image details and clarity.
[0071] As an optional embodiment, after obtaining the target image quality improvement model, the method includes:
[0072] The target image quality improvement model is optimized based on a network pruning module.
[0073] In this implementation, by selecting a suitable network pruning algorithm, such as pruning according to weight importance, channel pruning, structured pruning, etc. These algorithms can be selected according to the model structure and requirements. According to the selected pruning algorithm, the model is pruned to remove unimportant neurons, connections or layers, thereby reducing the complexity and computational burden of the model. After pruning, the model is fine-tuned to recover the performance loss caused by the pruning operation. The fine-tuning process can be achieved by continuing to train on the pruned model. The pruned and fine-tuned model is evaluated again to check whether its performance on the test set has improved, including indicators such as image quality and running time. That is, using cosine similarity as the similarity indicator, the similarity between the output of each module in the network and the output of the last residual module is calculated. The output of the last residual module is regarded as the standard output of the network. Multiple images are input into the network to obtain the output of each module, and then their cosine similarity with the output of the last module is calculated, and the average is taken to obtain the average similarity of each module. Among them, the similarity reflects the contribution of the module to the improvement of network performance. The greater the increase in similarity, the higher the contribution, which indicates that the network layer is more important to the overall network structure. When the similarity is lower than the preset similarity threshold, the convolution modules below the preset similarity threshold are removed, thereby reducing the number of network parameters and the amount of calculation. In order to ensure that the performance of the pruned network is consistent with the previous one, it is necessary to fine-tune the pruned network. Specifically, the Adam learning rate optimizer is used, the initial learning rate is set to 0.0001, the learning rate decays by half every 100 iterations, and the total training cycle is 300 epochs.
[0074] In the implementation of this application, by selecting a structured pruning method, pruning is performed according to the size of the weight, and the part with smaller weight is deleted. After the pruning operation, we fine-tune the model and continue to train the pruned model on the training set so that it can relearn and adapt to the new structure. Finally, re-evaluate the performance of the model on the test set to observe whether the image quality has improved and check whether the model calculation amount has been reduced. Through the optimization of network pruning, we can improve the efficiency and performance of the wafer image enhancement model, making it better when processing wafer images.
[0075] In summary, the present invention provides an image quality improvement method, device, equipment and medium, by acquiring a low-quality image and a high-quality image of a target wafer, performing degradation processing and interpolation processing on the high-quality image respectively, generating a first synthetic low-quality image and a second synthetic low-quality image, wherein the first synthetic low-quality image is obtained based on the high-quality image after training through a downsampling network, calculating the high-frequency component between the first synthetic low-quality image and the second synthetic low-quality image, inputting the high-frequency component into an image quality improvement model based on a generative adversarial network, and performing iterative training until the trained image quality improvement model meets the expected performance, and obtaining a target image quality improvement model, so that the target image quality improvement model improves the image quality of the target low-quality image. Compared with the prior art, the present application calculates the generated first synthetic low-quality image and the second synthetic low-quality image to obtain the high-frequency component between the two synthetic low-quality images, and trains the image quality improvement model through the high-frequency component to obtain the target image quality improvement model, thereby satisfying the quality improvement of the target low-quality image and improving the image quality improvement accuracy.
[0076] See also Figure 3 , Figure 3 Schematic diagram of the structure of the image quality improvement device provided by the embodiment of the present invention. In this embodiment, the terminal includes various units for executing Figure 3 For details, please refer to the steps in the corresponding embodiment. Figure 2 as well as Figure 2 For the convenience of explanation, only the parts related to this embodiment are shown. Figure 3 The image quality improvement device 30 includes: an acquisition module 31, a processing module 32, a calculation module 33, and a training module 34.
[0077] An acquisition module 31, used for acquiring a low-quality image and a high-quality image of a target wafer;
[0078] A processing module 32 is used to perform degradation processing and interpolation processing on the high-quality image to generate a first synthesized low-quality image and a second synthesized low-quality image, wherein the first synthesized low-quality image is obtained based on the high-quality image processed by the trained downsampling network;
[0079] A calculation module 33, configured to calculate a high frequency component between the first synthesized low-quality image and the second synthesized low-quality image;
[0080] The training module 34 is used to input the high-frequency component into the image quality improvement model based on the generative adversarial network, and perform iterative training until the trained image quality improvement model meets the expected performance, thereby obtaining a target image quality improvement model, wherein the target image quality improvement model is used to improve the image quality of the target low-quality image.
[0081] Optionally, the processing module 32 is specifically used for:
[0082] Processing the high-quality image to generate an initial synthesized low-quality image;
[0083] Training a pre-established downsampling network using the low-quality image and the initial synthesized low-quality image to obtain a trained downsampling network;
[0084] The high-quality image is input into the trained downsampling network to obtain a first synthesized low-quality image.
[0085] Optionally, the processing module 32 is further used for:
[0086] Obtain low-quality training samples and high-quality training samples;
[0087] Inputting the high-quality training samples into a generation network to obtain initial synthesized low-quality training samples, and calculating content losses of the initial synthesized low-quality training samples and the high-quality training samples;
[0088] Inputting the low-quality training samples and the initial synthesized low-quality training samples into a convolutional neural network to obtain the perceptual losses of the low-quality training samples and the initial synthesized low-quality training samples;
[0089] Inputting the low-quality training sample and the initial synthesized low-quality training sample into a discriminant network to obtain adversarial losses of the low-quality training sample and the initial synthesized low-quality training sample;
[0090] Based on the content loss, the perceptual loss and the adversarial loss, the downsampling network is trained until the sum of the content loss, the perceptual loss and the adversarial loss output by the trained downsampling network is less than a preset loss threshold.
[0091] Optionally, the processing module 32 is further used for:
[0092] The high-quality image is processed based on a bicubic interpolation algorithm to generate a second synthesized low-quality image.
[0093] Optionally, the calculation module 33 is specifically used for:
[0094] Performing convolution calculation on the first synthesized low-quality image and the second synthesized low-quality image based on the Haar wavelet transform method to obtain a first texture feature corresponding to the first synthesized low-quality image and a second texture feature corresponding to the second synthesized low-quality image respectively;
[0095] The difference between the first texture feature and the second texture feature is calculated to obtain a high-frequency component.
[0096] Optionally, the training module 34 is then specifically used for:
[0097] The target image quality improvement model is optimized based on a network pruning module.
[0098] It should be noted that the information interaction, execution process and other contents between the above-mentioned units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0099] Figure 4 Schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown in the figure), a memory, and a computer program stored in the memory and executable on at least one processor, and when the processor executes the computer program, the steps in the above-mentioned image quality improvement method embodiment are implemented.
[0100] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 4 These are merely examples of computer devices and do not constitute limitations on the computer devices. The computer devices may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, they may also include a network interface, a display screen, and an input device.
[0101] In one embodiment, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor in a computer device, the computer device can perform the steps of any embodiment of an image quality improvement disclosed in the present invention, which will not be repeated here. The computer-readable storage medium can be non-volatile or volatile.
[0102] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0103] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating device and the computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store operating devices, application programs, boot loaders (BootLoader), data, and other programs, such as program codes of computer programs, etc. The memory may also be used to temporarily store data that has been output or is to be output.
[0104] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0105] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0106] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.
Claims
1. A method for improving image quality, characterized in that: include: Acquire low-quality and high-quality images of the target wafer; Performing degradation processing and interpolation processing on the high-quality image respectively to generate a first synthesized low-quality image and a second synthesized low-quality image, wherein the first synthesized low-quality image is obtained based on the high-quality image processed by a trained downsampling network; calculating a high frequency component between the first synthesized low-quality image and the second synthesized low-quality image; Inputting the high-frequency component into an image quality improvement model based on a generative adversarial network, performing iterative training until the trained image quality improvement model meets expected performance, and obtaining a target image quality improvement model, wherein the target image quality improvement model is used to improve the image quality of a target low-quality image; The step of performing degradation processing on the high-quality image to generate a first synthesized low-quality image includes: Processing the high-quality image to generate an initial synthesized low-quality image; Training a pre-established downsampling network using the low-quality image and the initial synthesized low-quality image to obtain a trained downsampling network; The high-quality image is input into the trained downsampling network to obtain a first synthesized low-quality image.
2. The image quality improvement method according to claim 1, characterized in that: The downsampling network is trained as follows: Obtain low-quality training samples and high-quality training samples; Inputting the high-quality training samples into a generation network to obtain initial synthesized low-quality training samples, and calculating content losses of the initial synthesized low-quality training samples and the high-quality training samples; Inputting the low-quality training samples and the initial synthesized low-quality training samples into a convolutional neural network to obtain perceptual losses of the low-quality training samples and the initial synthesized low-quality training samples; Inputting the low-quality training sample and the initial synthesized low-quality training sample into a discriminant network to obtain adversarial losses of the low-quality training sample and the initial synthesized low-quality training sample; Based on the content loss, the perceptual loss and the adversarial loss, the downsampling network is trained until the sum of the content loss, the perceptual loss and the adversarial loss output by the trained downsampling network is less than a preset loss threshold.
3. The image quality improvement method according to claim 1, characterized in that: The interpolation process is performed on the high-quality image to generate a second synthesized low-quality image, comprising: The high-quality image is processed based on a bicubic interpolation algorithm to generate a second synthesized low-quality image.
4. The image quality improvement method according to claim 1, characterized in that: The calculating of the high frequency component between the first synthesized low-quality image and the second synthesized low-quality image comprises: Performing convolution calculation on the first synthesized low-quality image and the second synthesized low-quality image based on the Haar wavelet transform method to obtain a first texture feature corresponding to the first synthesized low-quality image and a second texture feature corresponding to the second synthesized low-quality image respectively; The difference between the first texture feature and the second texture feature is calculated to obtain a high-frequency component.
5. The image quality improvement method according to claim 1, characterized in that: After obtaining the target image quality improvement model, the method includes: The target image quality improvement model is optimized based on a network pruning module.
6. An image quality improvement device, characterized in that: include: An acquisition module, used for acquiring a low-quality image and a high-quality image of a target wafer; a processing module, configured to perform degradation processing and interpolation processing on the high-quality image to generate a first synthesized low-quality image and a second synthesized low-quality image, wherein the first synthesized low-quality image is obtained based on the high-quality image processed by a trained downsampling network; A calculation module, configured to calculate a high frequency component between the first synthesized low-quality image and the second synthesized low-quality image; A training module, used for inputting the high-frequency component into an image quality improvement model based on a generative adversarial network, performing iterative training until the trained image quality improvement model meets the expected performance, and obtaining a target image quality improvement model, wherein the target image quality improvement model is used to improve the image quality of a target low-quality image; Wherein, the processing module is also used for: Processing the high-quality image to generate an initial synthesized low-quality image; Training a pre-established downsampling network using the low-quality image and the initial synthesized low-quality image to obtain a trained downsampling network; The high-quality image is input into the trained downsampling network to obtain a first synthesized low-quality image.
7. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, and the processor implements the image quality improvement method according to any one of claims 1 to 5 when executing the computer program.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image quality improvement method according to any one of claims 1 to 5 is implemented.
Citation Information
Patent Citations
Super-resolution method based on unsupervised deep learning and corresponding equipment
CN117593188A
