Method for establishing and detecting image enhancement model of crystal image

Abstract

The invention relates to the field of crystal image detection, in particular to an image enhancement model establishment and detection method of a crystal image, an image enhancement model establishment method and an image enhancement model establishment method. Comprising a first convolutional layer for performing preliminary feature extraction on an input image, a first optimization module for optimizing a feature map, a second convolutional layer for performing feature extraction on the optimized feature map, and a second optimization module for optimizing the feature map output by the second convolutional layer, which are connected in sequence, the pyramid pooling module is used for carrying out pooling operation on the feature map output by the second optimization module; the third convolution layer is used for carrying out dimension reduction operation on the feature map output by the pyramid pooling module; the fourth convolutional layer is used for carrying out channel recovery on the feature map after the dimension reduction operation; the first activation function is used for carrying out activation operation on the feature map after the channel recovery operation; the image enhancement model obtains a higher-quality image enhancement effect in a low-light environment, and the processing efficiency is improved.

Description

Technical Field

[0001] The invention relates to the field of crystal image detection, and in particular to a method for establishing a detection method for a crystal image using an image enhancement model. Background Art

[0002] The Czochralski method is a common method for producing crystals. In the conventional Czochralski method crystal production process, the entire crystal growth process is usually in a single crystal furnace with high temperature, high pressure, high brightness and narrow space. During the production process, workers often need to observe the situation in the furnace with their naked eyes, but this method can easily cause workers' eyes to be damaged by the thermal radiation light generated by high temperature, and due to the limited accuracy and reaction speed of the human eye, it is difficult to achieve accurate and real-time monitoring of the crystal growth status.

[0003] With the continuous development of technology, industrial cameras have gradually replaced manual observation and become the main monitoring tool. They can capture image information of the crystal growth process in the furnace in real time, avoiding the safety hazards and inefficiency of manual observation. However, there are still some problems with the existing industrial camera image acquisition technology. For example, in a high temperature, high pressure and high brightness environment, strong light will cause the image to be overexposed. A neutral density filter needs to be added to the front end of the camera to capture a clear image of the inside of the single crystal furnace and protect the camera. This will result in insufficient image brightness. Low light, noise interference and other problems may occur during image acquisition, resulting in poor image quality, affecting the judgment of crystal quality and the accuracy of real-time monitoring. Summary of the invention

[0004] The object of the present invention is to provide a method for establishing an image enhancement model of a crystal image to improve the quality of the crystal image.

[0005] In order to achieve the above object, the present invention adopts such technical solution: A method for establishing an image enhancement model, establishing an image enhancement model, the image enhancement model is an improved Zero-DCE network model, including a first convolutional layer for performing preliminary feature extraction on an input image, a first optimization module for optimizing a feature map, a second convolutional layer for performing feature extraction on the optimized feature map, a second optimization module for optimizing the feature map output by the second convolutional layer, a pyramid pooling module for performing pooling operations of different scales on the feature map output by the second optimization module to extract multi-scale context information and fuse the information, a third convolutional layer for performing a dimensionality reduction operation on the feature map output by the pyramid pooling module, a fourth convolutional layer for performing channel recovery on the feature map after the dimensionality reduction operation, and a first activation function for performing an activation operation on the feature map after the channel recovery; The first optimization module and the second optimization module both include a channel attention module and a spatial attention module connected in sequence, which are used to apply the CBAM mechanism to the feature map output by the convolutional layer to further enhance the expression of important features.

[0006] Preferably, the channel attention module divides the input feature map into two paths for processing, one path performs a maximum pooling operation, and the other path performs a global average pooling operation. The two outputs are convolved through the fifth convolutional layer and then output through the second activation function.

[0007] Preferably, the spatial attention module performs maximum pooling and average pooling operations on the input feature map in sequence, convolves the feature map after the average pooling operation through the sixth convolutional layer, and outputs it through the third activation function.

[0008] Preferably, a first gradient mask module is added before the spatial attention module, and the first gradient mask module performs brightness adjustment on the feature map.

[0009] The method for detecting a crystal image comprises the following steps performed in sequence: S1: Acquire crystal image; S2: inputting the crystal image into the image enhancement model established by any of the above-mentioned image enhancement model establishment methods to perform image enhancement operations to obtain a preliminary enhanced image; S3: inputting the preliminary enhanced image into a feature enhancement module to perform a feature enhancement operation to obtain a feature enhanced image; S4: multiplying the preliminary enhanced image by the feature enhanced image to obtain a final enhanced image, and outputting the final enhanced image; S5: Calculate the total error function and use the total error function to back propagate and adjust the network.

[0010] Preferably, the feature enhancement operation of the feature enhancement module in step S3 includes the following steps: S3-1: using convolution kernels of different scales to perform feature extraction operations on the preliminary enhanced image in turn, extracting detail information and local features in the image while retaining the spatial structure of the image, and obtaining a preliminary feature map with regions of different sizes and multi-level structural information; S3-2: Using a channel attention mechanism to optimize the channel dimension information of the preliminary feature map, dynamically adjust the importance of each channel to enhance the performance of key features, and obtain a channel feature map; S3-3: using a gradient mask module to adjust the brightness of the channel feature map to obtain a brightness adjustment map; S3-4: Using a spatial attention mechanism to optimize the spatial dimension of the brightness adjustment map, identify and focus on important areas in the image to enhance details and contrast, and obtain a spatial feature map; S3-5: Use the Sigmoid activation function to activate the spatial feature map and output the feature enhanced image.

[0011] A crystal image detection system comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned crystal image detection methods when executing the computer program.

[0012] A computer product comprises a computer program, wherein when the computer program is executed by a processor, the method for detecting a crystal image as described in any one of the above items is implemented.

[0013] By adopting the above-mentioned design scheme, the beneficial effects of the present invention are as follows: the present invention improves the traditional Zero-DCE network, and constructs an image enhancement model by combining the optimization module and the pyramid pooling module. Different from the traditional Zero-DCE network, the image enhancement model introduces a channel attention module and a spatial attention module, so that the image enhancement model can automatically focus on important detail areas in the image, avoiding excessive enhancement of irrelevant areas, thereby improving the clarity and detail recovery of the image. At the same time, the pyramid pooling module effectively extracts multi-scale contextual information and enhances the expression of crystal surfaces and large-scale structures; the image enhancement model obtains higher quality image enhancement effects in low-light environments, improves processing efficiency, not only improves the usability of the image, but also greatly reduces the dependence on manual intervention, and improves the safety and efficiency of the production process. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a structural diagram of the image enhancement model; Figure 2 Schematic diagram of the structure of the channel attention module; Figure 3 Schematic diagram of the structure of the spatial attention module; Figure 4 It is a comparison chart of the original image and the detected image; Figure 5 The figure is an algorithm flow chart of the method for detecting crystal images; Figure 6 Schematic diagram of the structure of the feature enhancement module. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] The terms "first", "second", "third", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0017] Image enhancement model establishment method, establish an image enhancement model, such as Figure 1 As shown, the image enhancement model is an improved Zero-DCE ((Zero-Reference Deep Curve Estimation)) network model, including a first convolution layer for performing preliminary feature extraction on an input image, a first optimization module for optimizing a feature map, a second convolution layer for performing feature extraction on the optimized feature map, a second optimization module for optimizing the feature map output by the second convolution layer, a pyramid pooling module for performing a pooling operation on the feature map output by the second optimization module, a third convolution layer for performing a dimensionality reduction operation on the feature map output by the pyramid pooling module, a fourth convolution layer for performing channel recovery on the feature map after the dimensionality reduction operation, and a first activation function for performing an activation operation on the feature map after the channel recovery; The first optimization module (CBAM) and the second optimization module (CBAM) both include a channel attention module and a spatial attention module connected in sequence.

[0018] In this embodiment, the first convolution layer is a convolution layer with a convolution kernel of 3*3. The first convolution layer performs preliminary feature extraction on the input image, so that the number of channels of the image increases from 3 (RGB image) to 64; then the Relu activation function is used for nonlinear transformation to enhance feature expression; the crystal surface contains complex textures and details. The first convolution layer can retain the detail information in the image by extracting edges, textures and local patterns, thereby enhancing the recognizability of the crystal surface and structure.

[0019] The second convolution layer is a convolution layer with a convolution kernel of 5*5 and 64 channels. The second convolution layer further extracts features from the optimized feature map, increases the receptive field, and processes it through the Relu activation function. The Czochralski crystal image contains large structures or textures. The use of a 5*5 convolution layer can capture a wider range of image information, which helps to improve the overall structural characteristics of the crystal surface.

[0020] like Figure 2As shown in the figure, the channel attention module divides the input feature map into two paths for processing, one for maximum pooling operation and the other for global average pooling operation. After the two outputs pass through the fifth convolution layer for convolution operation, they are output through the second activation function. The importance of each channel is calculated by global average pooling and maximum pooling, and the channel attention map is generated using the Sigmoid activation function as the second activation function to weight the input features. In crystal images, some channels (such as color channels or texture channels) contain more important information, such as the edges of crystals. The channel attention module helps the model focus on these key features and suppress irrelevant information by giving higher weights to important channels.

[0021] like Figure 3 As shown in the figure, the spatial attention module performs maximum pooling and average pooling operations on the input feature map in sequence, and the feature map after the average pooling operation is convolved through the sixth convolutional layer and output through the third activation function. By calculating the average pooling and maximum pooling results of the feature map, a spatial attention map is generated and the input features are weighted, and the sigmoid activation function is used as the third activation function to generate spatial weights. Crystal images contain different local areas, and the spatial attention module helps to focus on important spatial locations, such as the edges of crystals or areas with defects, and improve the visibility of these areas. Unimportant areas such as the background and black edges of the crystal image can be paid less attention. In addition, by focusing on important areas, spatial attention helps to suppress background noise and improve the clarity of the overall image.

[0022] The second optimization module applies the CBAM mechanism to the feature map output by the second convolutional layer to further enhance the expression of important features.

[0023] The pyramid pooling module performs pooling operations of different scales (1*1, 2*2, 3*3, 6*6) on the feature maps output by the second optimization module to extract multi-scale contextual information, and then fuses these multi-scale information through convolution. The details and large-scale structures in the crystal image, such as the shape, surface features, cracks, etc. of the crystal, can be better represented through multi-scale pooling. The pyramid pooling module helps the model focus on both details and overall structure by extracting features at different scales. By fusing information at different scales, the image enhancement model can better understand the global structure and local details of the crystal image, which helps to improve the naturalness of the enhancement effect.

[0024] The third convolution layer is a convolution layer with a convolution kernel of 1*1, which performs channel dimension reduction on the feature map output by the pyramid pooling module, so that the number of channels of the feature map is restored to 64. Dimension reduction through 1x1 convolution reduces the number of channels of the feature map, thereby reducing the amount of computation for subsequent processing.

[0025] The fourth convolution layer is a convolution layer with a convolution kernel of 3*3, which restores the number of channels of the feature map to the number of channels of the original input image.

[0026] The first activation function is the Sigmoid activation function, which limits the output to the range of [0, 1] to ensure that the pixel values ​​of the enhanced image are legal and output a preliminary enhanced image.

[0027] As a preferred method of this embodiment: a first gradient mask module is added before the spatial attention module, and the first gradient mask module adjusts the brightness of the feature map. In the image of the crystal produced by the direct pulling method, the crystal is usually located in the center of the image, and there is an obvious black edge area around it. These black edge areas not only do not contribute to the characteristic analysis and quality assessment of the crystal, but may interfere with the model's feature extraction of the core area, that is, the crystal part. Therefore, in order to improve the efficiency of the spatial attention module, a first gradient mask module is added before the spatial attention module to pre-screen the input image by a preset brightness threshold. The first gradient mask module calculates the average brightness or grayscale value of the image and compares it with the preset brightness threshold to identify significantly darker areas in the image (such as black edges). These dark areas are usually meaningless to the task, so they can be efficiently eliminated in the preprocessing stage.

[0028] The first gradient mask module uses a two-dimensional Gaussian distribution centered on the image center. Pixels in the central area are given a higher weight, and the weight value gradually decreases from the central area to the surrounding area, forming a smooth transition area. This embodiment uses the following formula as the weight of the gradient mask:

[0029] in is The weight value of and are the pixel coordinates in the image, and are the coordinates of the center point of the Gaussian distribution, is the standard deviation of the Gaussian distribution.

[0030] Smooth transition: Gradient mask can avoid the mutation artifact problem caused by hard boundaries, and is especially suitable for feature extraction in the edge area of ​​​​the crystal.

[0031] More natural feature enhancement: For the blurred areas at the edge of the crystal, the gradient mask can be processed more naturally by gradually reducing the weight, thus avoiding the loss of edge information.

[0032] The spatial attention module generates a spatial attention map based on the input feature map. The spatial attention map has the same spatial dimension as the input feature map, and the value of each position indicates the importance of the position on the feature map. The generated mask is element-wise multiplied with the spatial attention map. For each spatial position on the feature map, its final attention weight will be the product of its original spatial attention weight and the corresponding mask value. This combination can more effectively guide attention, allowing the model to pay more attention to the core area of ​​the image, thereby improving the performance and robustness of the model; the above operation is expressed in the following formula:

[0033] in, is the original spatial attention map, is the mask image, is the final attention weight map.

[0034] This embodiment also provides a method for detecting a crystal image using the above-mentioned image enhancement model.

[0035] Crystal image detection methods, such as Figure 5 As shown, the steps include the following steps performed in sequence: S1: Acquire a crystal image. In this embodiment, an industrial camera is used to collect data. A conventional neutral density filter is added to the industrial camera lens for shooting. A mechanical structure is used to control the industrial camera to collect images or video data of different stages of the crystal growth process. The video data can be converted into image data frame by frame.

[0036] S2: inputting the crystal image into the image enhancement model established by any of the above-mentioned image enhancement model establishment methods to perform image enhancement operations to obtain a preliminary enhanced image; S3: inputting the preliminary enhanced image into the feature enhancement module to perform feature enhancement operation to obtain a feature enhanced image; like Figure 6 As shown, the feature enhancement operation of the feature enhancement module in step S3 includes the following steps: S3-1: Use convolution kernels of different scales to perform feature extraction operations on the preliminary enhanced image in sequence to obtain a preliminary feature map; in this embodiment, convolution kernels of different scales such as 3*3, 5*5, 7*7, etc. can be used to perform feature extraction operations on the preliminary enhanced image in sequence to extract detail information and local features in the image while retaining the spatial structure of the image; multi-scale convolution can extract features at different spatial scales, so that the network can capture regions of different sizes and multi-level structural information while capturing image details.

[0037] S3-2: Use the channel attention mechanism to optimize the channel dimension information of the preliminary feature map and obtain the channel feature map; dynamically adjust the importance of each channel through the channel attention mechanism to enhance the performance of key features.

[0038] S3-3: using a gradient mask module to adjust the brightness of the channel feature map to obtain a brightness adjustment map; S3-4: A spatial attention mechanism is used to optimize the spatial dimension of the brightness adjustment map to obtain a spatial feature map; the spatial attention mechanism is used to identify and focus on important areas in the image to enhance details and contrast.

[0039] S3-5: Use the Sigmoid activation function as the activation function to activate the spatial feature map and output the feature-enhanced image. This ensures that there will not be excessive value fluctuations during the image enhancement process, thereby achieving a more accurate and richer image enhancement effect.

[0040] S4: Multiply the preliminary enhanced image and the feature enhanced image to obtain a final enhanced image, and output the final enhanced image. Figure 4 shown.

[0041] S5: Calculate the total error function and use the total error function to back propagate and adjust the network.

[0042] In this embodiment, the following formula is used as the total error function of the above-mentioned crystal image detection method: ; in, is the total error, is the spatial consistency error, is the weight of the spatial consistency error, is the exposure control error, is the weight of exposure control error, is the color constant error, is the weight of the color constant error, is the illumination smoothing error, is the weight of the lighting smoothing error.

[0043] ; The image is decomposed into several local regions of 4×4 pixels, and the difference between the center i and the adjacent j pixels in the local region is calculated. By minimizing this difference, the spatial consistency of the image is maintained.

[0044] Where K is the number of local regions, There are four adjacent regions centered on region I. The four adjacent regions here refer to the upper, lower, left and right regions. If region I is located at the edge of the image, there are only three adjacent regions. Y is the average brightness value of the local region of the image, and I is the average brightness value of the local region of the input version.

[0045] ; The image is decomposed into several local regions of 16×16 pixels, and the average brightness value in the local region is calculated.

[0046] Where M is the number of local areas, Y is the average brightness value of a local area in the image, and E is the good exposure level (E=0.6).

[0047] ; The image is divided into three channels of RGB, and the mean of the pixel intensity of each channel is calculated to minimize the color difference between different channels and further optimize the color smoothness of the image.

[0048] in, is the average brightness of the P channel in the enhanced image, and (p,q) is a pair of color channels.

[0049] ; By calculating horizontal and vertical gradients and minimizing the total change in gradients, unnatural sudden changes in lighting in the image are avoided.

[0050] Where N is the number of iterations, is the intensity value of channel c of the enhanced image at the nth iteration, is the horizontal gradient, Vertical gradient.

[0051] In this embodiment = 1, = 1, = 0.5, = 20.

[0052] The error function is used to adjust the network in reverse. The input image is passed through the network, the enhanced image is generated, and the total loss is calculated. , each loss item contributes by adjusting the weight. According to the total loss The value of , calculates the gradient of the loss, and uses the Adam optimizer to update the model parameters. Repeat the process of forward propagation, loss calculation, back propagation and parameter update until the model converges, that is, the total loss Reach the preset value. Each loss adjusts its contribution to the total loss through the corresponding weight. In the Czochralski crystal monitoring, the texture and details of the crystal surface are relatively complex. By constraining the consistency of the image gradient, the details and edge features of the crystal surface can be retained while avoiding noise enhancement. In the Czochralski crystal monitoring scene, the monitoring image in a low-light environment may be too dim to identify the morphology and characteristics of the crystal. Through the exposure control loss, the image brightness can be significantly improved while ensuring that the brightness distribution of the enhanced image is uniform. In the crystal monitoring task, color information is crucial to the evaluation of crystal quality. If the color information is distorted, it will interfere with the recognition of crystal texture features. Color constancy loss ensures that the enhanced image can truly reflect the color distribution of the crystal. In the Czochralski crystal monitoring image, the illumination changes are relatively complex. The illumination smoothing loss can avoid drastic fluctuations in the enhancement factor within a small range, thereby improving the smoothness of the image and the stability of the enhancement effect.

[0053] In this embodiment, a system for implementing the above-mentioned crystal image detection method is also provided.

[0054] A crystal image detection system comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above-mentioned crystal image detection methods when executing the computer program.

[0055] In this embodiment, a computer product capable of implementing the above-mentioned crystal image detection method is also provided.

[0056] A computer product comprises a computer program, wherein when the computer program is executed by a processor, the method for detecting a crystal image as described in any one of the above items is implemented.

[0057] In summary, the present invention improves the traditional Zero-DCE network, and constructs an image enhancement model by combining the optimization module and the pyramid pooling module. Different from the traditional Zero-DCE network, the image enhancement model introduces a channel attention module and a spatial attention module, so that the image enhancement model can automatically focus on important detail areas in the image, avoiding excessive enhancement of irrelevant areas, thereby improving the clarity and detail recovery of the image. At the same time, the pyramid pooling module effectively extracts multi-scale contextual information and enhances the expression of crystal surfaces and large-scale structures; the image enhancement model obtains higher quality image enhancement effects in low-light environments, improves processing efficiency, not only improves the usability of the image, but also greatly reduces the dependence on manual intervention, and improves the safety and efficiency of the production process.

[0058] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for establishing an image enhancement model, characterized in that: Establish an image enhancement model, which is an improved Zero-DCE network model, including a first convolutional layer that performs preliminary feature extraction on an input image, a first optimization module that optimizes a feature map, a second convolutional layer that performs feature extraction on the optimized feature map, a second optimization module that optimizes the feature map output by the second convolutional layer, a pyramid pooling module that performs pooling operations of different scales on the feature map output by the second optimization module to extract multi-scale context information and fuses it, a third convolutional layer that performs a dimensionality reduction operation on the feature map output by the pyramid pooling module, a fourth convolutional layer that performs channel recovery on the feature map after the dimensionality reduction operation, and a first activation function that performs an activation operation on the feature map after the channel recovery; The first optimization module and the second optimization module both include a channel attention module and a spatial attention module connected in sequence, which are used to apply the CBAM mechanism to the feature map output by the convolutional layer to further enhance the expression of important features.

2. The method for establishing an image enhancement model according to claim 1, wherein: The channel attention module divides the input feature map into two paths for processing, one path performs a maximum pooling operation, and the other path performs a global average pooling operation. The two outputs are convolved through the fifth convolutional layer and then output through the second activation function.

3. The image enhancement model building method according to claim 2, characterized in that: The spatial attention module performs maximum pooling and average pooling operations on the input feature map in sequence, and convolves the feature map after the average pooling operation through the sixth convolutional layer, and then outputs it through the third activation function.

4. The method for establishing an image enhancement model according to claim 3, wherein: A first gradient mask module is added before the spatial attention module, and the first gradient mask module performs brightness adjustment on the feature map.

5. A method for detecting a crystal image, characterized in that: The steps include the following steps: S1: Acquire crystal image; S2: inputting the crystal image into the image enhancement model established by the image enhancement model establishment method according to any one of claims 1 to 4 to perform image enhancement operation to obtain a preliminary enhanced image; S3: inputting the preliminary enhanced image into a feature enhancement module to perform a feature enhancement operation to obtain a feature enhanced image; S4: multiplying the preliminary enhanced image by the feature enhanced image to obtain a final enhanced image, and outputting the final enhanced image; S5: Calculate the total error function and use the total error function to back propagate and adjust the network.

6. The method for detecting a crystal image according to claim 5, wherein: The feature enhancement operation of the feature enhancement module in step S3 includes the following steps: S3-1: using convolution kernels of different scales to perform feature extraction operations on the preliminary enhanced image in turn, extracting detail information and local features in the image while retaining the spatial structure of the image, and obtaining a preliminary feature map with regions of different sizes and multi-level structural information; S3-2: Using a channel attention mechanism to optimize the channel dimension information of the preliminary feature map, dynamically adjust the importance of each channel to enhance the performance of key features, and obtain a channel feature map; S3-3: using a gradient mask module to adjust the brightness of the channel feature map to obtain a brightness adjustment map; S3-4: Using a spatial attention mechanism to optimize the spatial dimension of the brightness adjustment map, identify and focus on important areas in the image to enhance details and contrast, and obtain a spatial feature map; S3-5: Use the Sigmoid activation function to activate the spatial feature map and output the feature enhanced image.

7. A crystal image detection system, comprising a memory and a processor, wherein a computer program is stored in the memory, characterized in that: When the processor executes the computer program, the crystal image detection method according to any one of claims 5 to 6 is implemented.

8. A computer product, comprising a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting a crystal image according to any one of claims 5 to 6 is implemented.

Images