Image enhancement model training method and image enhancement method

Abstract

The application provides an image enhancement model training method and an image enhancement method, and relates to the technical field of image processing. Through an encoder-decoder architecture and supplemented by adaptive fusion skip connection, effective extraction and fusion of multi-scale features are realized. The encoder is responsible for multi-scale feature coding of the input image, gradually extracts the abstract features of the image and reduces the spatial resolution. The decoder is responsible for gradually restoring the image resolution and using the multi-scale features provided by the encoder for fine reconstruction, and finally outputs the enhanced image. While highlighting the local details of the ink droplets, the global contrast of the image and the statistical characteristics consistent with the physical reality are effectively maintained, and the distinguishability of the ink droplets and the background is significantly improved, providing a high-precision and high-quality image basis for subsequent quantitative analysis of the ink droplets.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to an image enhancement model training method and an image enhancement method. Background Technology

[0002] In precision manufacturing fields such as industrial inkjet printing and bioprinting, high-precision observation and analysis of ink droplet trajectories and final landing points are crucial. The quality of the ink droplet images acquired by the observation system directly determines the accuracy of subsequent landing point localization, size measurement, and satellite point (micro-splash point) identification. However, in actual industrial environments, due to factors such as limited light sources, uneven lighting, and low contrast between ink droplets and the background, the acquired images often suffer from insufficient overall brightness, localized underexposure or overexposure, and blurred ink droplet edges, making it difficult to effectively distinguish ink droplets from the background and severely limiting the accuracy of observation.

[0003] Existing technologies struggle to enhance images and highlight local details of ink droplets while effectively maintaining global contrast and statistical characteristics that conform to physical reality. They also cannot dynamically adjust processing strategies based on image content, resulting in insufficient generalization ability and large errors in subsequent quantitative analysis under complex and variable industrial imaging conditions.

[0004] There is currently no effective technical solution to the above problems. Summary of the Invention

[0005] The purpose of this application is to provide an image enhancement model training method and an image enhancement method, which aims to solve the problem of limited observation accuracy caused by poor ink droplet image quality.

[0006] In a first aspect, this application provides an image enhancement method that uses an image enhancement model to enhance an image. The image enhancement model includes an encoder and a decoder, and the encoder and decoder are adaptively fused through skip connections.

[0007] The encoder consists of multiple coding layers connected in series, used to encode multi-scale features of the input image; the output feature map size of each coding layer is half the size of the input image of that coding layer;

[0008] Each coding layer includes a first CNN layer, an adaptive block module, an image block unit, a Transformer layer, an image fusion unit, and a second CNN layer connected in sequence;

[0009] The first CNN layer is used to extract and downsample local features from the feature map input to the encoding layer, and output a first intermediate feature map containing spatial location information. The adaptive block segmentation module is used to receive the first intermediate feature map and dynamically generate a block segmentation strategy based on the content of the first intermediate feature map. The image block segmentation unit is used to divide the first intermediate feature map into multiple feature blocks according to the block segmentation strategy, and convert each feature block into a feature vector to form an input sequence. The Transformer layer is used to perform self-attention calculation on the input sequence to model global context dependencies and output the corresponding output sequence. The image integration unit is used to integrate the feature vectors in the output sequence into a second intermediate feature map according to the block segmentation strategy. The second CNN layer is used to perform feature fusion and upsampling on the second intermediate feature map and output the final feature map of the encoding layer.

[0010] The above scheme effectively captures and fuses local details and global context of images, thus solving the problems of poor image enhancement effect and insufficient generalization ability of traditional methods in complex industrial environments, and significantly improving the enhancement quality of ink droplet images and the accuracy of subsequent analysis.

[0011] Optionally, the decoder consists of multiple first convolutional layers connected in series. The decoder and encoder have the same number of layers. The input of the first first convolutional layer of the decoder is the output of the last layer of the encoder. The input of the i-th first convolutional layer of the decoder is the fusion of the output of the (i-1)-th first convolutional layer of the decoder and the output of the (k-i+1)-th coding layer of the encoder, where i=2 to k. The decoder is used to gradually restore the image resolution and output the enhanced final image, where k is the total number of all first convolutional layers of the decoder.

[0012] This technical solution uses the same number of layers as the encoder and employs a layer-by-layer fusion approach. This allows the decoder to fully utilize the feature information extracted by the encoder at different scales, thereby more effectively restoring image resolution and outputting high-quality enhanced images, further improving the precision and accuracy of image enhancement.

[0013] Optionally, the fusion expression for the skip connection is:

[0014] ;

[0015] In the formula, For decoder number Layer input, For encoder number The output feature map of the layer, For decoder number The output of the layer, For the decoder in the Adaptive fusion weight coefficients during layer feature fusion.

[0016] This technical solution introduces adaptive fusion weight coefficients, enabling the model to dynamically adjust the fusion ratio according to the importance of features at different levels. This achieves more flexible and effective feature fusion, further optimizing the image enhancement effect and avoiding information loss or redundancy that may result from fixed-weight fusion.

[0017] Secondly, this application provides an image enhancement model training method for training an image enhancement model, wherein the image enhancement model is applied to any of the image enhancement methods described above, and the image enhancement model training method includes the following steps:

[0018] Obtain a training sample set containing images of ink droplet landing points;

[0019] Based on the training sample set, the image enhancement model is trained using a composite loss function to obtain the trained image enhancement model. The composite loss function includes pixel mean absolute error loss, structural similarity loss, feature perception loss, and moment feature loss.

[0020] This technical solution uses a training sample set containing images of ink droplet landing points and a composite loss function for training. This allows the model to comprehensively consider pixel-level, structural-level, perceptual-level, and moment feature-level information of the image during training, thereby effectively improving the model's enhancement effect on ink droplet images. This allows the model to better adapt to the characteristics of ink droplet images while maintaining image details, structure, and visual quality.

[0021] Alternatively, the expression for the composite loss function is:

[0022] ;

[0023] In the formula, For the total loss, For pixel mean absolute error loss, For structural similarity loss, For feature-perceived loss, For moment feature loss, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient. It is the fourth weighting coefficient.

[0024] Optionally, the expression for the pixel mean absolute error loss is:

[0025] ;

[0026] In the formula, For the enhanced image, The target image.

[0027] Optionally, the expression for structural similarity loss is:

[0028] ;

[0029] ;

[0030] In the formula, It is a structural similarity index. The average pixel value of the enhanced image. The average pixel value of the target image. The standard deviation of the enhanced image pixels, The standard deviation of the target image pixels. To determine the covariance between the enhanced image and the target image, This is the first preset constant. This is the second preset constant. This is the third preset constant.

[0031] Optionally, the expression for the feature-aware loss is:

[0032] ;

[0033] In the formula, The number of selected feature maps refers to the number of specific layers extracted from the pre-trained network for calculating the loss. The number of selected feature maps layer, To enhance the image in the pre-trained network Feature maps output by the layer For the target image in the pre-trained network Feature maps output by the layer.

[0034] Alternatively, the expression for the moment feature loss is:

[0035] ;

[0036] In the formula, The highest order of the moment eigenvalues. For the first Moments of order one, For the first Weighting coefficients of the order moment feature. For the enhanced image, the first Each order moment feature, For the target image Each order moment feature.

[0037] Thirdly, this application also provides an image enhancement method, including the steps of:

[0038] Obtain the image of the ink droplet landing point to be enhanced;

[0039] The image of the ink droplet landing point to be enhanced is input into the image enhancement model trained according to any of the image enhancement model training methods described above, and the image enhancement model outputs the final enhanced image.

[0040] This application provides a complete image enhancement process through this technical solution, which can apply the trained image enhancement model to actual ink droplet landing images, thereby efficiently and accurately outputting the enhanced final image. This effectively solves the problem of poor ink droplet image quality in fields such as industrial inkjet printing, and provides a high-quality image foundation for subsequent high-precision observation and analysis.

[0041] As can be seen from the above, the image enhancement model training method and image enhancement method provided in this application, through the introduction of skip connections between the encoder and decoder for adaptive fusion, and the innovative combination of CNN and Transformer within the encoding layer, can effectively solve the problems of insufficient overall brightness, local overexposure or underexposure, and blurred ink droplet edges in existing ink droplet images. The first CNN layer extracts and downsamples local features and provides positional encoding for the Transformer, making up for the lack of positional awareness in the standard Transformer. The adaptive segmentation module dynamically generates segmentation strategies based on the image content, enabling the Transformer layer to selectively process features in different regions, avoiding the limitations of traditional fixed segmentation strategies. The Transformer layer models global contextual dependencies through a self-attention mechanism, effectively capturing long-range information of the image. The image integration unit integrates the feature vectors output by the Transformer according to the segmentation strategy, ensuring the closure and consistency of the information flow. The second CNN layer further performs feature fusion and upsampling, restoring the image spatial size and achieving secondary fusion of local-global-local features. This multi-scale, adaptive, and combined local and global enhancement mechanism enables the model to dynamically adjust the enhancement strategy according to the complex and variable characteristics of ink droplet images. While highlighting the local details of ink droplets, it effectively maintains the global contrast and statistical properties of the image that conform to physical reality, significantly improving the distinction between ink droplets and the background. It overcomes the shortcomings of traditional methods in terms of insufficient generalization ability, thus providing a high-precision and high-quality image foundation for subsequent quantitative analysis of ink droplets, and greatly improving the performance and reliability of the observation system.

[0042] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the structure of the image enhancement model provided in the embodiments of this application.

[0044] Figure 2 This is a schematic diagram of the encoder provided in an embodiment of this application.

[0045] Figure 3 This is a schematic diagram of the structure of the first CNN layer provided in an embodiment of this application.

[0046] Figure 4 This is a schematic diagram of the Transformer layer provided in an embodiment of this application.

[0047] Figure 5 This is a schematic diagram of the structure of the adaptive block segmentation module provided in an embodiment of this application.

[0048] Figure 6 This is a schematic diagram of the structure of the second CNN layer provided in an embodiment of this application.

[0049] Labeling Explanation: 101, Encoder; 102, Encoding Layer; 103, Decoder; 104, First Convolutional Layer; 105, First CNN Layer; 106, Transformer Layer; 107, Second CNN Layer; 108, Adaptive Blocking Module; 109, Image Blocking Unit; 110, Image Integration Unit; 201, Linear Transformation Layer; 202, Multi-head Self-Attention; 203, First Normalization Layer; 204, Feedforward Neural Network; 205, Second Normalization Layer. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0051] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0052] Please refer to Figure 1-2 This application provides a method for training an image enhancement model and an image enhancement method. It aims to address the problem of limited observation accuracy caused by poor ink droplet image quality.

[0053] In a first aspect, this application provides an image enhancement method. The image enhancement method uses an image enhancement model to enhance the image. The image enhancement model includes an encoder 101 and a decoder 103. The encoder 101 and the decoder 103 are adaptively fused through skip connections.

[0054] The encoder 101 consists of multiple coding layers 102 connected in series, used to perform multi-scale feature encoding on the input image; the output feature map size of each coding layer 102 is half the size of the input image of that coding layer 102;

[0055] Each coding layer 102 includes a first CNN layer 105, an adaptive block module 108, an image block unit 109, a Transformer layer 106, an image integration unit 110, and a second CNN layer 107 connected in sequence.

[0056] The first CNN layer 105 is used to extract and downsample local features from the feature map input to the coding layer 102, and output a first intermediate feature map containing spatial location information; the adaptive block segmentation module 108 is used to receive the first intermediate feature map and dynamically generate a block segmentation strategy according to the content of the first intermediate feature map; the image block segmentation unit 109 is used to divide the first intermediate feature map into multiple feature blocks according to the block segmentation strategy, and convert each feature block into a feature vector to form an input sequence; the Transformer layer 106 is used to perform self-attention calculation on the input sequence to model global context dependencies, and output the corresponding output sequence; the image integration unit 110 is used to integrate the feature vectors in the output sequence into a second intermediate feature map according to the block segmentation strategy; the second CNN layer 107 is used to perform feature fusion and upsampling on the second intermediate feature map, and output the final feature map of the coding layer 102.

[0057] The image enhancement model proposed in this application aims to enhance ink droplet impact images with high quality using deep learning technology, overcoming the challenges encountered by traditional methods in complex industrial environments. The core of this model lies in its encoder 101-decoder 103 architecture, supplemented by adaptive fusion skip connections to achieve effective extraction and fusion of multi-scale features. Specifically, encoder 101 is responsible for encoding multi-scale features of the input image, progressively extracting abstract features and reducing its spatial resolution. Decoder 103 is responsible for progressively restoring the image resolution and using the multi-scale features provided by encoder 101 for refined reconstruction, ultimately outputting the enhanced image.

[0058] Specifically, such as Figure 1 and Figure 2 As shown, encoder 101 consists of multiple coding layers 102 connected in series. Each coding layer 102 halves the size of the input feature map, thereby achieving multi-scale feature extraction. Each coding layer 102 integrates multiple functional modules to collaboratively complete local feature extraction, global modeling, and integration. The first CNN layer 105 is mainly responsible for local feature extraction and downsampling of the input feature map, while providing necessary spatial location information for the subsequent Transformer layer 106. The adaptive segmentation module 108 dynamically generates a segmentation strategy based on the content of the first intermediate feature map, enabling the image segmentation unit 109 to divide the feature map into multiple feature blocks according to the strategy and convert them into the input sequence required by the Transformer layer 106. The Transformer layer 106 performs self-attention calculation on this basis to capture global contextual dependencies. Subsequently, the image integration unit 110 integrates the output sequence of the Transformer layer 106 back into the feature map form according to the same segmentation strategy. Finally, the second CNN layer 107 performs feature fusion and upsampling on the integrated feature map, outputting the final feature map of coding layer 102.

[0059] In one implementation, the first CNN layer 105 may consist of a series of convolutional layers, activation function layers, and pooling layers. The first CNN layer 105 employs three downsampling networks, each containing a downsampling operation and a convolutional operation, thereby progressively downsampling the feature map size to 1 / 8 of the original input image size, such as... Figure 3 As shown, where H is the height of the input image, W is the width of the input image, C is the number of channels of the input image, C1 is the number of channels of the input image of the first downsampling network in the first CNN layer 105, C2 is the number of channels of the output image of the first downsampling network in the first CNN layer 105, C3 is the number of channels of the output image of the second downsampling network in the first CNN layer 105, C4 is the number of channels of the output image of the third downsampling network in the first CNN layer 105, and F... in down The input image is the first downsampled network in the first CNN layer 105, and the F1 score is... down This is the output of the first downsampled network in the first 105 layers of the first CNN, F2. down F is the output of the second downsampling network in the first CNN layer 105. out downThis is the output of the second CNN layer 107. This design not only effectively reduces the computational complexity of the subsequent Transformer layer 106, but also provides important positional encoding information for the Transformer layer 106 through convolution operations, making up for the inherent deficiency of the standard Transformer in lacking position awareness when processing images.

[0060] The adaptive block segmentation module 108 can be composed of multiple convolutional layers and corresponding activation functions, such as... Figure 4 As shown, H, W, and C represent the height, width, and number of channels of the output feature map of the first CNN layer 105, respectively, and N is the number of mask images output by the adaptive block segmentation module 108. Multiple convolutional layers are used to extract features from the input feature map, and activation layers are used to convert the output of the convolutional layers into multiple mask image information (…). ), Each pixel value in the mask image information is between 0 and 1 (representing the probability or weight of the corresponding pixel belonging to the corresponding semantic block).

[0061] The main function of the image segmentation unit 109 is to transform the strategy (mask) generated by the adaptive segmentation module 108 into a data format (sequence) that can be processed by the Transformer layer 106. Specifically, the image segmentation unit 109 transforms the feature map output by the first CNN layer 105 into a data format (sequence). Multiple mask images obtained by the adaptive block segmentation module 108 The image is divided into patches to represent different parts of the image. These patches are then unfolded into one-dimensional vectors. It serves as the input to Transformer layer 106. Its mathematical expression can be represented as: ;

[0062] in, For one-dimensional unfolding operations, The first output of the adaptive block module 108 Mask information;

[0063] Transformer layer 106 receives the input sequence guided by the adaptive block segmentation module 108. Its core mechanism is self-attention, which focuses on achieving long-range dependency modeling and acquiring global information about the image. (See reference...) Figure 5 The Transformer layer 106 includes a linear transformation layer 201, a multi-head self-attention layer 202, a first normalization layer 203, a feedforward neural network 204, and a second normalization layer 205. The linear transformation layer 201 takes each feature vector input from the image block unit 109 and passes it through three different weight matrices. , , The query vector is multiplied by all the input embedding vector sequences, dividing them into a query vector matrix (Q), a key vector matrix (K), and a value vector matrix (V). Next, the multi-head self-attention 202 performs a dot product operation between the query vector and the key vector to obtain the relevance between all embedding vectors in the input sequence. The relevance score is then processed by the SoftMax operation in this multi-head self-attention 202 to obtain an attention score map. The SoftMax operation ensures that for a given query, the sum of the attention weights for all keys is 1. Finally, the attention score is weighted and summed with the value vectors to obtain the output of the multi-head self-attention. The mathematical expression for the operation between any vector in the query vector matrix Q, the key vector matrix K, and the value vector matrix V is:

[0064] ;

[0065] ;

[0066] ;

[0067] in, To query the i-th vector in the vector matrix Q, Let J be the j-th vector in the key vector matrix K. For the i-th query vector With the j-th key vector The original correlation score is calculated. For the i-th query vector With the j-th key vector Attention score The total number of input embedding vectors, For the i-th query vector With the nth key vector The original correlation score is calculated. Let j be the j-th vector in the value vector matrix V. for The corresponding output vector.

[0068] By querying each vector in the vector matrix and performing the above calculations to obtain the corresponding output vector, we can obtain the output vector sequence of the multi-head self-attention 202 for all input vector sequences. Extending vector operations to matrix operations, the complete calculation formula for the self-attention mechanism is:

[0069] ;

[0070] in, The output vector sequence; This involves matrix multiplication to obtain a relevance score between any two vectors in the query vector matrix and the key vector matrix. is the dimension of any vector in the key vector matrix K, used to scale the result of matrix multiplication to prevent gradient vanishing.

[0071] The feedforward neural network 204 adds nonlinear transformations to independently and deeply refine and map the information after self-attention fusion, enhancing the model's expressive power. The first normalization layer 203 and the second normalization layer 205 can control the scale of data distribution, which is beneficial for stable model training and faster convergence.

[0072] The image integration unit 110 performs the inverse transformation from the sequence output of Transformer layer 106 to the feature map representation, which is crucial for information flow closure. It utilizes the same mask for integration, ensuring the consistency and reversibility of the segmentation and integration processes. Specifically, the image integration unit 110 integrates multiple one-dimensional vectors output from Transformer layer 106... The image is transformed into a two-dimensional feature map of the first CNN layer 105, thereby obtaining multiple image patches representing information from different locations in the image. Then, the mask information obtained by the adaptive segmentation module 108 is used. Different image patches are masked, and finally, the masked image patches are superimposed through a union operation to obtain the integrated feature map F_fuse. Its mathematical expression can be represented as:

[0073] ;

[0074] in, The first output of Transformer A one-dimensional vector, The integrated feature map is represented by Λ(·), which is a two-dimensional deformation operation.

[0075] The second CNN layer 107 is used to perform feature fusion and upsampling on the second intermediate feature map, outputting the final feature map of the coding layer 102. (Refer to...) Figure 6 This layer typically employs two upsampling networks, each containing a convolutional operation and an upsampling operation. Their function is to perform feature enhancement and upsampling to restore the spatial dimensions. Specifically, the first upsampling network takes the output F of the image integration unit 110... in up The output F of the third downsampling network in the first CNN layer 105 out down The fused feature map is used as input, and the second upsampling network takes the output F1 score of the first upsampling network as input. up The output F2 of the second downsampling network in the first CNN layer 105down The fused feature map is used as input, F out down This is the output of the second CNN layer 107. H and W are the height and width of the input image for each coding layer 102, respectively. C4 is the number of channels in the input feature map of the first upsampled network, C3 is the number of channels in the output feature map of the first upsampled network, and C2 is the number of channels in the output feature map of the second upsampled network. Figure 6 C2, C3, C4 and Figure 3 The corresponding C2, C3, and C4 have the same number of channels. Specifically, it upsamples the feature map size back to half the size of the original input coding layer 102, realizing the scale transformation of encoder 101. At the same time, it fuses with the features of the first CNN layer 105 through skip connections, realizing the secondary fusion of local-global-local features within coding layer 102, which greatly enriches the hierarchy of output features.

[0076] The image enhancement model of this application effectively addresses several challenges in ink droplet image enhancement through its unique encoder 101-decoder 103 architecture and adaptive fusion mechanism. First, the first CNN layer 105 in encoder 101 provides feature maps with spatial location information for the subsequent Transformer layer 106 through local feature extraction and downsampling, while reducing computational complexity. The adaptive segmentation module 108 dynamically generates segmentation strategies based on image content, enabling Transformer layer 106 to perform more refined global context modeling for different regions of the image, avoiding detail loss or information redundancy that may occur with traditional fixed segmentation strategies. Transformer layer 106 utilizes a self-attention mechanism to capture long-range dependencies, effectively integrating global information and compensating for the shortcomings of CNNs in processing global information. The image integration unit 110 ensures the consistency of information during segmentation and integration, enabling accurate reconstruction of feature maps. Finally, the second CNN layer 107 restores the spatial resolution of the image through feature fusion and upsampling, further enhancing the expressive power of the features. The skip connections between encoder 101 and decoder 103 and the adaptive fusion mechanism enable features at different scales to be effectively combined, thereby restoring image details while maintaining global consistency.

[0077] Traditional image enhancement methods often employ fixed processing strategies, making it difficult to adapt to the complex and ever-changing industrial imaging environment. This results in a struggle to balance enhancing local details with maintaining global contrast when enhancing ink droplet images. For example, some methods may over-enhance local areas, leading to artifacts or noise in the image, while others may fail to effectively improve the contrast between the ink droplet and the background. The core innovation of this application lies in the introduction of an adaptive block segmentation module 108, which dynamically adjusts the feature map segmentation strategy based on the semantic content of the feature map. Compared to traditional fixed block segmentation or global processing methods, the model in this application can more intelligently identify key regions (such as ink droplets) in the image and process them accordingly. This effectively maintains the global contrast and physically accurate statistical properties of the image while enhancing the local details of the ink droplet. This adaptive processing capability significantly improves the model's generalization ability and robustness under complex industrial imaging conditions, providing a higher-quality image foundation for subsequent ink droplet landing point localization, size measurement, and satellite point recognition, thereby effectively improving the observation accuracy and reliability in the field of precision manufacturing.

[0078] In some implementations, the decoder 103 is composed of multiple first convolutional layers 104 connected in series. The decoder 103 and the encoder 101 have the same number of layers. The input of the first first convolutional layer 104 of the decoder 103 is the output of the last layer of the encoder 101. The input of the i-th first convolutional layer 104 of the decoder 103 is the fusion of the output of the (i-1)-th first convolutional layer 104 of the decoder 103 and the output of the (k-i+1)-th coding layer 102 of the encoder 101, where i=2 to k. The decoder 103 is used to gradually restore the image resolution and output the enhanced final image, where k is the total number of all first convolutional layers 104 of the decoder 103.

[0079] Specifically, the decoder 103 is designed to consist of multiple first convolutional layers 104 connected in series, with the same number of layers as the encoder 101. This symmetrical layer design helps to establish a balance between feature extraction and resolution restoration. The input to the first first convolutional layer 104 of the decoder 103 is set to the output of the last layer of the encoder 101, meaning that the decoder 103 starts image reconstruction from the deepest and most abstract features extracted by the encoder 101. Furthermore, for the i-th first convolutional layer 104 of the decoder 103, its input is not simply from the previous decoding layer, but rather a fusion of the output of the (i-1)-th first convolutional layer 104 of the decoder 103 and the output of the (k-i+1)-th encoding layer 102 of the encoder 101. Here, k represents the total number of all first convolutional layers 104 in the decoder 103. This fusion mechanism, i.e., skip connections, ensures that during decoding, not only are the upsampled features generated by the current decoding layer utilized, but also the richer and higher-resolution features from the corresponding layer of the encoder 101 are introduced. For example, when i=2, the input to the second first convolutional layer 104 of the decoder 103 will be the fusion of the output of the first first convolutional layer 104 of the decoder 103 and the output of the (k-2+1)th coding layer 102 of the encoder 101. This layer-by-layer fusion method enables the decoder 103 to gradually and finely restore the image resolution and finally output the enhanced image.

[0080] The solution proposed in this application effectively solves the aforementioned problems by designing the decoder 103 as a series of convolutional layers with the same number of layers as the encoder 101 and employing a specific skip-connection fusion strategy. Specifically, the first convolutional layer 104 of the decoder 103 directly receives the output of the last layer of the encoder 101, ensuring that the image enhancement model can reconstruct from the most abstract semantic features. More importantly, for subsequent decoding layers, their input is set to be the fusion of the output of the previous decoding layer and the output of the corresponding layer of the encoder 101. This layer-by-layer, symmetrical skip-connection fusion mechanism allows the decoder 103 to fully utilize the rich feature information extracted by the encoder 101 at different scales during the gradual restoration of image resolution. The deep features of the encoder 101 provide high-level semantic information, while the shallow features retain more spatial details. By adaptively fusing these features from different levels, the decoder 103 can effectively compensate for the detail information lost due to downsampling during the encoding process, thereby achieving more accurate and detailed image resolution restoration.

[0081] In some preferred embodiments, such as Figure 1As shown, assume that both encoder 101 and decoder 103 contain k=5 layers. Encoder 101 consists of a first coding layer 102, a second coding layer 102, a third coding layer 102, a fourth coding layer 102, and a fifth coding layer 102 connected in series. Decoder 103 consists of a first convolutional layer 104, a second convolutional layer 104, a third convolutional layer 104, a fourth convolutional layer 104, and a fifth convolutional layer 104 connected in series. Specifically, the input to the first convolutional layer 104 of decoder 103 is the output of the fifth coding layer 102. For the second convolutional layer 104 of decoder 103, its input is the fusion of the output of the first convolutional layer 104 and the output of the fourth coding layer 102. For the third convolutional layer 104 of decoder 103, its input is the fusion of the output of the second convolutional layer 104 and the output of the third coding layer 102. For the fourth first convolutional layer 104 of decoder 103, its input is the fusion of the output of the third first convolutional layer 104 of decoder 103 and the output of the second coding layer 102. For the fifth first convolutional layer 104 of decoder 103, its input is the fusion of the output of the fourth first convolutional layer 104 of decoder 103 and the output of the first coding layer 102. In this way, decoder 103 can receive feature information from the corresponding level of encoder 101 in each first convolutional layer 104, thereby continuously supplementing and refining image details while progressively upsampling to restore image resolution, and finally outputting a high-quality enhanced image.

[0082] In some implementations, the fusion expression for skip connections is:

[0083] ;

[0084] In the formula, For decoder 103 Layer input, For encoder 101 The output feature map of the layer, For decoder 103 The output of the layer, For decoder 103 in the Adaptive fusion weight coefficients during layer feature fusion.

[0085] Specifically, This refers to decoder 103 in the... The input feature map received by the layer. This can be understood as encoder 101 in the... The feature map output by the layer contains the encoded information of the input image at a specific scale. This refers to decoder 103 in the... The feature map output after layer processing has undergone preliminary restoration and refinement. Among them, This can be understood as an adaptive fusion weight coefficient, whose value typically ranges from 0 to 1. This coefficient is used to dynamically adjust the output features of the previous layer of decoder 103. Output features corresponding to encoder 101 The relative contributions in the current fusion process. At that time, the input to the first layer of decoder 103 directly came from the output of the last layer of encoder 101, so fusion was not required. At that time, the input to the layer of decoder 103 was obtained by weighted summation of and . The purpose of this weighted summation is to achieve adaptive feature fusion at different levels to better balance the multi-scale contextual information from encoder 101 and the local detail information from decoder 103. In practical applications, the adaptive fusion weight coefficients can be dynamically predicted and generated by a lightweight network module (e.g., composed of several second convolutional layers and activation functions) based on the current feature map content. The aim is to enable the model to intelligently determine the fusion ratio of features at different levels based on the local semantic information of the image and the importance of the features, thereby optimizing feature utilization efficiency.

[0086] The proposed solution introduces adaptive fusion weight coefficients, enabling the feature fusion process of skip connections to move beyond fixed proportions and dynamically adjust based on the actual content of the feature maps. Specifically, during the gradual image resolution recovery process of the decoder 103, the input of each layer no longer simply depends on the output of the previous layer or the corresponding output of the encoder 101. Instead, a learnable adaptive fusion weight coefficient determines the balance between the two. This mechanism allows the model to flexibly prioritize either the rich contextual information provided by the encoder 101 or the refined local details by the decoder 103 when processing image features of different regions or scales. For example, in smooth areas of the image, the model may prefer to utilize the features recovered by the decoder 103; while in areas rich in detail or with ink droplets, the model may prefer to introduce multi-scale features provided by the encoder 101 to better capture and recover details. Thus, this adaptive fusion mechanism ensures that feature information from different levels is more effectively integrated and utilized during image enhancement, overcoming the problem of insufficient feature utilization that may result from fixed fusion strategies. Especially in ink droplet image enhancement tasks, it can more accurately recover image details, suppress noise, and generate enhanced images with higher visual quality.

[0087] Secondly, this application provides an image enhancement model training method for training an image enhancement model, wherein the image enhancement model is applied to any of the image enhancement methods described above, and the image enhancement model training method includes the following steps:

[0088] Obtain a training sample set containing images of ink droplet landing points;

[0089] Based on the training sample set, the image enhancement model is trained using a composite loss function to obtain the trained image enhancement model. The composite loss function includes pixel mean absolute error loss, structural similarity loss, feature perception loss, and moment feature loss.

[0090] The training method of this application introduces a composite loss function containing multiple loss terms, which can optimize the model from multiple dimensions such as pixel, structure, perception and statistical moments. This enables the trained image enhancement model to process ink droplet images more effectively, enhance image quality and highlight local details of ink droplets, while maintaining the global contrast of the image and the statistical characteristics that conform to physical reality, thus significantly improving the accuracy and reliability of the ink droplet observation system.

[0091] The structure and working principle of the image enhancement model have been described in the above embodiments and will not be repeated here. It should be emphasized that, in order to enable the above image enhancement model to effectively cope with the complex challenges of ink droplet images in real industrial environments, this application further proposes a specialized training method.

[0092] In the training method of this application, the composite loss function is designed to include pixel mean absolute error loss, structural similarity loss, feature-aware loss, and moment feature loss. Pixel mean absolute error loss measures the difference between the enhanced image and the target image at the pixel level, aiming to make the pixel values ​​of the enhanced image as close as possible to the target image. Structural similarity loss focuses on the structural information of the image, aiming to ensure that the brightness, contrast, and structure of the enhanced image are consistent with the target image, thereby better preserving the visual quality of the image. Feature-aware loss measures the perceptual similarity between the enhanced image and the target image by comparing the high-level features extracted from the pre-trained network, which helps the model learn to generate images that are more consistent with human visual perception. Moment feature loss constrains the statistical properties of the enhanced image, such as the mean, variance, and skewness, to ensure consistency with the statistical properties of the target image, which is crucial for maintaining the physical realism of the ink droplet image. By comprehensively utilizing these four loss functions, the model can be optimized from multiple dimensions during training, resulting in a trained image enhancement model that performs excellently in terms of pixel, structural, perceptual, and statistical properties.

[0093] The image enhancement model training method proposed in this application aims to address the limitations of traditional training methods when processing ink droplet images. Existing traditional training methods often rely on single pixel-level losses or simple structural losses. These loss functions struggle to comprehensively consider the local details, global contrast, and physically realistic statistical properties of ink droplet images when optimizing the model. For example, a model trained solely using pixel mean absolute error loss may tend to generate blurred images to minimize pixel differences, resulting in the loss of ink droplet edge details. Conversely, models focusing only on structural similarity may fail to effectively improve the contrast between the ink droplet and the background in certain situations, or may not guarantee that the statistical properties of the enhanced image are consistent with the real ink droplet image.

[0094] The core innovation of this application lies in the introduction of a composite loss function comprising pixel mean absolute error loss, structural similarity loss, feature perception loss, and moment feature loss. This composite loss function, through multi-dimensional and multi-level constraints, enables the training process to more comprehensively guide model learning. Pixel mean absolute error loss ensures the overall accuracy of the image, structural similarity loss maintains the visual structure of the image, feature perception loss improves the perceptual quality of the image, and moment feature loss guarantees that the statistical properties of the enhanced image match those of the real ink droplet image. Therefore, this training method can significantly improve the accuracy and reliability of the ink droplet observation system, providing a high-quality image foundation for the precision manufacturing field.

[0095] In some implementations, the composite loss function is expressed as follows:

[0096] ;

[0097] In the formula, For the total loss, For pixel mean absolute error loss, For structural similarity loss, For feature-perceived loss, For moment feature loss, As the first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, These are the fourth weighting coefficients, which are used to balance the contribution of different loss terms to the total loss. They can be adjusted according to the specific training task and image characteristics to optimize the training effect of the model and the final image enhancement quality.

[0098] Through the aforementioned technical solutions, the composite loss function L can more comprehensively and precisely guide the training of image enhancement models. Compared to using only a single loss function, this weighted combination of multi-dimensional losses enables the model to simultaneously optimize multiple aspects such as pixel accuracy, structural similarity, perceptual quality, and texture detail, effectively avoiding local optima or performance deficiencies in certain areas that may result from a single loss function. Consequently, the trained image enhancement model can generate visually more natural, more detailed, and higher-quality enhanced images, especially for images with complex textures and details, such as ink droplet images, significantly improving their enhancement effect and visual acceptability.

[0099] In some implementations, the expression for the pixel mean absolute error loss is:

[0100] ;

[0101] In the formula, For the enhanced image, The target image.

[0102] The solution in this application, by employing the aforementioned expression for pixel mean absolute error loss, enables the image enhancement model to directly and effectively measure the enhanced image during training. With target image The difference lies at the pixel level. It is precisely because this loss function directly calculates the absolute difference in pixel values ​​that the model can be guided to minimize this difference, thereby prompting the model to generate an enhanced image that is closer to the target image in terms of pixel attributes such as brightness and color.

[0103] In some implementations, the expression for structural similarity loss is:

[0104] ;

[0105] ;

[0106] In the formula, It is a structural similarity index. The average pixel value of the enhanced image. The average pixel value of the target image. The standard deviation of the enhanced image pixels, The standard deviation of the target image pixels. To determine the covariance between the enhanced image and the target image, This is the first preset constant. This is the second preset constant. This is the third preset constant.

[0107] The purpose of the first, second, and third preset constants is to avoid the denominator being zero and to stabilize the calculation results.

[0108] The proposed solution incorporates structural similarity loss L_SSIM into a composite loss function, enabling the image enhancement model to not only focus on pixel-level differences during training but also perceive and learn the structural information of the image. Structural similarity loss L_SSIM guides the model to generate enhancement results that visually more closely resemble the target image by comparing the enhanced image Î with the target image I in terms of brightness, contrast, and structure. Especially when processing ink droplet images, their complex textures and edge details are crucial for visual quality. Structural similarity loss L_SSIM effectively encourages the model to preserve these important structural features, avoiding over-smoothing or introducing artifacts, thus significantly improving the clarity and realism of the enhanced image.

[0109] In some implementations, the feature-aware loss is expressed as:

[0110] ;

[0111] In the formula, The number of selected feature maps refers to the number of specific layers extracted from the pre-trained network for calculating the loss. The number of selected feature maps layer, To enhance the image in the pre-trained network Feature maps output by the layer For the target image in the pre-trained network Feature maps output by the layer.

[0112] Pre-trained networks typically refer to deep convolutional neural networks trained on large datasets (such as ImageNet), such as VGG, ResNet, or EfficientNet. The features learned by these networks have good generalization ability and can effectively represent the semantic content and stylistic information of images.

[0113] This application's scheme introduces feature-aware loss, enabling the training process to focus not only on pixel-level differences but also on high-level semantic and stylistic information optimization. Specifically, when the image enhancement model generates an enhanced image... At that time, the image With target image They will be fed into a pre-trained network, and their feature maps at different intermediate layers will be extracted. and Feature-aware loss By calculating the differences between these feature maps, the model's parameters are adjusted. If the enhanced image... Perceptually related to the target image The closer the features are, the more similar the feature maps extracted in the pre-trained network will be, resulting in less feature perception loss. This mechanism enables the model to learn to generate images with higher visual quality and more natural textures, especially when processing ink droplet images, where it can better recover the details and structure of the image, making it appear more realistic and clear to the human eye.

[0114] Through the aforementioned technical solutions, the image enhancement model can effectively capture deep semantic information and perceptual features of images during training, rather than being limited to pixel-level matching. This enables the model to generate enhanced images that are visually more realistic, richer in detail, and more structurally sound. Especially when processing images with complex textures and details, such as ink droplet images, the feature-perceptual loss effectively avoids generating blurry or distorted results, thereby significantly improving the overall quality of image enhancement and the user experience.

[0115] In some implementations, the expression for the moment feature loss is:

[0116] ;

[0117] In the formula, The highest order of the moment eigenvalues. For the first Moments of order one, For the first Weighting coefficients of the order moment feature. For the enhanced image, the first Each order moment feature, For the target image Each order moment feature.

[0118] This application's approach introduces moment feature loss, enabling the image enhancement model to not only focus on pixel-level differences (such as pixel mean absolute error loss), structural similarity (such as structural similarity loss), and high-level semantic features (such as feature-aware loss) during training, but also effectively capture the deep structural and textural information of the image. Moment features can describe the geometric shape, spatial distribution, and texture details of an image, which are crucial for accurately recovering the details of the ink droplet landing image. By minimizing the moment feature loss between the enhanced image and the target image, the model is guided to generate an enhanced image that is statistically closer to the real image, thereby better repairing ink droplet marks and stains in the image while maintaining overall visual quality, ensuring the structural integrity and detail clarity of the image.

[0119] Through the above technical solutions, the image enhancement model can learn the intrinsic characteristics of images more comprehensively during training. Especially when dealing with images of ink droplets with complex textures and irregular shapes, the introduction of moment feature loss can effectively compensate for the shortcomings of other loss functions in capturing these details. As a result, the model can generate visually more natural and clearer enhanced images, significantly improving the quality and robustness of image enhancement, especially demonstrating superiority in removing ink droplet marks and restoring image details.

[0120] Thirdly, this application also provides an image enhancement method, including the steps of:

[0121] Obtain the image of the ink droplet landing point to be enhanced;

[0122] The image of the ink droplet landing point to be enhanced is input into the image enhancement model trained according to any of the image enhancement model training methods described above, and the image enhancement model outputs the final enhanced image.

[0123] The image of the ink droplet landing point to be enhanced can be acquired in real time by a high-speed camera or read from a pre-stored image database. In some implementations, image acquisition can be manually triggered, with the operator inputting a command when observing the ink droplet image that needs enhancement; in other implementations, image acquisition can be set to be automatically triggered, for example, by continuously monitoring and automatically capturing ink droplet images that meet specific conditions through an image sensor.

[0124] Furthermore, the acquired image of the ink droplet landing point to be enhanced is input into the image enhancement model trained according to the image enhancement model training method described above. This image enhancement model is constructed and trained according to the structure described in the above embodiments. The input process can involve directly passing image data as the input tensor to the model. For example, the image data can be converted to a specific format and size required by the model and fed into the model for processing via a programming interface (API) or direct data stream.

[0125] Subsequently, the image enhancement model outputs the enhanced final image. This output image, processed by the model, highlights ink droplet details and optimizes contrast. The enhanced image can be directly displayed on a monitor for operator viewing, or saved to local storage or a cloud server for subsequent analysis and archiving. As a preferred implementation, the enhanced image can also be transmitted in real time to downstream image analysis systems, such as modules for ink droplet location, size measurement, or satellite point recognition, to support automated, high-precision analysis tasks.

[0126] The core innovation of this application lies in its use of a specially trained image enhancement model. This model, through adaptive block segmentation module 108 and composite loss function techniques, dynamically adjusts the processing strategy based on the semantic content of the image, thereby better balancing local details and global contrast when enhancing ink droplet images. By inputting the image of the ink droplet landing point to be enhanced into this trained model, this method can intelligently identify key regions (such as ink droplets) in the image and process them accordingly. This effectively maintains the global contrast and statistical characteristics of the image while enhancing the local details of the ink droplets. This adaptive processing capability based on a deep learning model significantly improves the quality and efficiency of ink droplet image enhancement, providing a higher-quality image foundation for subsequent ink droplet landing point localization, size measurement, and satellite point recognition. This effectively improves the observation accuracy and reliability in the field of precision manufacturing, overcoming the problems of insufficient generalization ability and large quantitative analysis errors in existing technologies.

[0127] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0128] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image enhancement method, characterized in that, Image enhancement methods use image enhancement models to enhance images. These models consist of an encoder and a decoder, which are adaptively fused through skip connections. The encoder consists of multiple coding layers connected in series, used to encode multi-scale features of the input image; the output feature map size of each coding layer is half the size of the input image of that coding layer; Each coding layer includes a first CNN layer, an adaptive block module, an image block unit, a Transformer layer, an image fusion unit, and a second CNN layer connected in sequence; The first CNN layer is used to extract and downsample local features from the feature map input to the encoding layer, and output a first intermediate feature map containing spatial location information. The adaptive block segmentation module is used to receive the first intermediate feature map and dynamically generate a block segmentation strategy based on the content of the first intermediate feature map. The image block segmentation unit is used to divide the first intermediate feature map into multiple feature blocks according to the block segmentation strategy, and convert each feature block into a feature vector to form an input sequence. The Transformer layer is used to perform self-attention calculation on the input sequence to model global context dependencies and output the corresponding output sequence. The image integration unit is used to integrate the feature vectors in the output sequence into a second intermediate feature map according to the block segmentation strategy. The second CNN layer is used to perform feature fusion and upsampling on the second intermediate feature map and output the final feature map of the encoding layer.

2. The image enhancement method according to claim 1, characterized in that, The decoder consists of multiple first convolutional layers connected in series. The decoder and encoder have the same number of layers. The input of the first first convolutional layer of the decoder is the output of the last layer of the encoder. The input of the i-th first convolutional layer of the decoder is the fusion of the output of the (i-1)-th first convolutional layer of the decoder and the output of the (k-i+1)-th coding layer of the encoder, where i = 2 to k. The decoder is used to gradually restore the image resolution and output the enhanced final image, where k is the total number of all first convolutional layers of the decoder.

3. The image enhancement method according to claim 2, characterized in that, The fusion expression for skip connections is: ； In the formula, For decoder number Layer input, For encoder number The output feature map of the layer, For the decoder The output of the layer, For the decoder in the Adaptive fusion weight coefficients during layer feature fusion.

4. A method for training an image enhancement model, characterized in that, For training an image enhancement model, said image enhancement model is applied to the image enhancement method as described in any one of claims 1-3, the image enhancement model training method comprising the steps of: Obtain a training sample set containing images of ink droplet landing points; Based on the training sample set, the image enhancement model is trained using a composite loss function to obtain the trained image enhancement model. The composite loss function includes pixel mean absolute error loss, structural similarity loss, feature perception loss, and moment feature loss.

5. The image enhancement model training method according to claim 4, characterized in that, The expression for the composite loss function is: ； In the formula, For the total loss, For pixel mean absolute error loss, For structural similarity loss, For feature-perceived loss, For moment feature loss, As the first weighting coefficient, This is the second weighting coefficient. The third weighting coefficient, It is the fourth weighting coefficient.

6. The image enhancement model training method according to claim 5, characterized in that, The expression for the pixel mean absolute error loss is: ； In the formula, For the enhanced image, The target image.

7. The image enhancement model training method according to claim 5, characterized in that, The expression for structural similarity loss is: ； ； In the formula, It is a structural similarity index. The average pixel value of the enhanced image. The average pixel value of the target image. The standard deviation of the enhanced image pixels, The standard deviation of the target image pixels. To determine the covariance between the enhanced image and the target image, This is the first preset constant. This is the second preset constant. This is the third preset constant.

8. The image enhancement model training method according to claim 5, characterized in that, The expression for feature-aware loss is: ； In the formula, The number of selected feature maps refers to the number of specific layers extracted from the pre-trained network for calculating the loss. The number of selected feature maps layer, To enhance the image in the pre-trained network Feature maps output by the layer For the target image in the pre-trained network Feature maps output by the layer.

9. The image enhancement model training method according to claim 5, characterized in that, The expression for the moment feature loss is: ； In the formula, The highest order of the moment eigenvalues. For the first Moments of order one, For the first Weighting coefficients of the order moment feature. For the enhanced image, the first Each order moment feature, For the target image Each order moment feature.

10. An image enhancement method, characterized in that, Including the following steps: Obtain the image of the ink droplet landing point to be enhanced; The image of the ink droplet landing point to be enhanced is input into the image enhancement model trained by the image enhancement model training method according to any one of claims 4-9, and the image enhancement model outputs the enhanced final image.

Images