Image denoising method and device, electronic equipment and storage medium

Abstract

The invention discloses an image denoising method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a glue spraying point noise image and attribute information thereof, and inputting the glue spraying point noise image and the attribute information into a pre-trained image denoising model (comprising a global association module, a feature enhancement module, a noise prediction module and an image denoising module); performing kernel mapping attention processing on the glue spraying point noise image and the attribute information by using a global association module to obtain global multi-modal features; performing feature enhancement processing on the global multi-modal features by using a feature enhancement module to obtain enhanced multi-modal features; decoding the enhanced multi-modal features by using a noise prediction module to obtain noise features of the glue spraying point noise image; and processing the noise features and the glue spraying point noise image by using an image denoising module to obtain a denoised image of the glue spraying point noise image. According to the technical scheme of the embodiment of the invention, the accuracy of image denoising is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of image denoising, and particularly relate to an image denoising method and device, an electronic device, and a storage medium. BACKGROUND

[0002] As a core carrier of tobacco leaf packaging, the glue spraying quality of the cigarette box is directly related to the sealing performance of the packaging (such as preventing the tobacco from getting wet and scattering) and the appearance quality of the product (such as avoiding packaging defects caused by glue overflow and incomplete glue points). Therefore, high-precision detection of the glue spraying point image of the cigarette box through machine vision technology is a key link to ensure the quality of tobacco production. However, the noise (such as blur caused by equipment vibration, speckle caused by glue splashing, and bright-dark noise caused by light source fluctuation) in the glue spraying point image can seriously interfere with the recognition of key features such as glue point edge, size, and continuity by the detection algorithm. Therefore, efficient denoising of the glue spraying point image of the cigarette box is particularly important to improve detection accuracy.

[0003] Currently, the denoising method for glue spraying point noise images mainly relies on single-level feature information extracted from the image (such as only extracting visual features of the image through a convolutional network) to carry out denoising work. However, relying solely on single-level feature data cannot fully capture the distribution and essential characteristics of noise, which makes it difficult to accurately distinguish between "noise" and "effective details of the glue spraying point" during the denoising process, resulting in problems such as "excessive denoising and loss of key details" or "incomplete denoising and residual noise", and ultimately causing insufficient accuracy of denoising.

[0004] Therefore, there is an urgent need to propose a new method to solve the above problems. SUMMARY

[0005] The present application provides an image denoising method, device, electronic device and storage medium to improve the accuracy of image denoising.

[0006] In a first aspect, an embodiment of the present application provides an image denoising method, which comprises:

[0007] obtaining a glue spraying point noise image and attribute information of the glue spraying point noise image, inputting the glue spraying point noise image and the attribute information into a pre-trained image denoising model, the image denoising model comprising a global correlation module, a feature enhancement module, a noise prediction module, and an image denoising module;

[0008] performing kernel mapping attention processing on the glue spraying point noise image and the attribute information using the global correlation module to obtain global multi-modal features of the glue spraying point noise image;

[0009] The global multi-modal feature is processed by the feature enhancement module to obtain an enhanced multi-modal feature of the glue spraying point noise image;

[0010] The enhanced multi-modal feature is decoded by the noise prediction module to obtain a noise feature of the glue spraying point noise image;

[0011] The noise feature and the glue spraying point noise image are processed by the image denoising module to obtain a denoised image of the glue spraying point noise image.

[0012] The technical scheme of the embodiment of the application first acquires a glue spraying point noise image and attribute information of the glue spraying point noise image, inputs the glue spraying point noise image and the attribute information into a pre-trained image denoising model, and lays a data foundation for subsequently obtaining a denoised image of the glue spraying point noise image. Then, the global correlation module is used to perform kernel mapping attention processing on the glue spraying point noise image and the attribute information to obtain a global multi-modal feature of the glue spraying point noise image, realize deep fusion of cross-modal information, improve the richness of the feature, and effectively solve the limitation of a single mode. On this basis, with the help of kernel mapping attention processing, the region related to key information (such as glue point edges and shapes) of the glue spraying point in the image can be automatically focused on, while noise interference is effectively suppressed, further enhancing the anti-noise capability of feature extraction. Not only does this lay a more accurate and robust feature foundation for subsequent image denoising processing, but it also helps to comprehensively capture the distribution law and essential characteristics of noise. Then, the feature enhancement module is used to perform feature enhancement processing on the global multi-modal feature to obtain an enhanced multi-modal feature of the glue spraying point noise image, effectively improving the richness and adaptability of the feature, enhancing the robustness of the feature to noise, and retaining key details of the glue spraying point, thereby improving the accuracy of subsequent denoising processing. The noise prediction module is then used to decode the enhanced multi-modal feature to obtain a noise feature of the glue spraying point noise image, which not only helps to accurately locate noise information, but also enhances the explainability of the obtained noise feature, providing a targeted basis for subsequent denoising tasks, thereby effectively improving denoising precision and the quality of the denoised image. Finally, the image denoising module is used to process the noise feature and the glue spraying point noise image to obtain a denoised image of the glue spraying point noise image, realizing effective suppression of the glue spraying point noise and improving the quality of the denoised image, thereby providing a more reliable image input for subsequent glue spraying point detection and improving detection accuracy. Therefore, the technical scheme of the application solves the problem of insufficient denoising accuracy caused by relying only on single-level feature data for denoising in the prior art.

[0013] In a second aspect, the embodiment of the application further provides an image denoising device, which comprises:

[0014] The acquisition module is configured to acquire a glue spraying point noise image and attribute information of the glue spraying point noise image, input the glue spraying point noise image and the attribute information into a pre-trained image denoising model, and the image denoising model comprises a global correlation module, a feature enhancement module, a noise prediction module and an image denoising module.

[0015] The feature extraction module is configured to perform kernel mapping attention processing on the glue spraying point noise image and the attribute information by using the global correlation module to obtain global multi-modal features of the glue spraying point noise image.

[0016] The feature enhancement module is configured to perform feature enhancement processing on the global multi-modal features by using the feature enhancement module to obtain enhanced multi-modal features of the glue spraying point noise image.

[0017] The decoding module is configured to perform decoding on the enhanced multi-modal features by using the noise prediction module to obtain noise features of the glue spraying point noise image.

[0018] The denoising module is configured to perform processing on the noise features and the glue spraying point noise image by using the image denoising module to obtain a denoised image of the glue spraying point noise image.

[0019] In a third aspect, an electronic device is also provided, and the electronic device comprises:

[0020] at least one processor; and a memory connected with the at least one processor in communication;

[0021] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image denoising method described in any of the embodiments of the present application.

[0022] In a fourth aspect, a storage medium containing computer executable instructions is also provided, and the computer executable instructions, when executed by a computer processor, implement the image denoising method described in any of the embodiments of the present application.

[0023] It should be noted that the above computer instructions can be stored on a computer readable storage medium in whole or in part. The computer readable storage medium can be packaged together with the processor of the image denoising device, or can be packaged separately from the processor of the image denoising device, and the present application does not limit this.

[0024] The second aspect, the third aspect and the fourth aspect in the present application can refer to the detailed description of the first aspect, and the beneficial effects of the description of the second aspect, the third aspect and the fourth aspect can refer to the beneficial effect analysis of the first aspect, which will not be described here.

[0025] In the present application, the names of the above-mentioned image denoising device do not constitute a limitation on the devices or functional modules themselves, and in actual implementation, these devices or functional modules can appear with other names. As long as the functions of each device or functional module are similar to those in the present application, they belong to the scope of the claims of the present application and equivalent technologies.

[0026] These aspects or other aspects of the present application will be more apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 A flow chart of an image denoising method provided by an embodiment of the present application is shown in the following figure:

[0029] Figure 2a A flow chart of another image denoising method provided by an embodiment of the present application is shown in the following figure:

[0030] Figure 2b A structure diagram of an image denoising model provided by an embodiment of the present application is shown in the following figure:

[0031] Figure 3 A structure diagram of an image denoising device provided by an embodiment of the present application is shown in the following figure:

[0032] Figure 4 A structure diagram of an electronic device provided by an embodiment of the present application is shown in the following figure. DETAILED DESCRIPTION

[0033] The present application will be further described in detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, in order to facilitate the description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0034] The term "and / or" in this paper is only a description of the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B together, and the existence of B alone.

[0035] The terms "first" and "second" and the like in the description of the application and in the claims of the present application are used for distinguishing between similar objects, or to distinguish between different processing steps, and are not necessarily used to describe a particular sequential order, unless explicitly stated to do so.

[0036] Furthermore, the terms "comprise", "comprising", "have", "having", "include", "including", "contain", "containing", "provide", "providing", "offer", "offering", "specify", "specifying", and the like, are used in the description of the application and in the claims of the present application and are intended to be interpreted as "including" rather than "consisting of". For example, a process, method, object, or apparatus that comprises a list of steps or elements is not necessarily limited to the listed steps or elements, but can include additional steps or elements not expressly listed or inherent to such process, method, object, or apparatus.

[0037] Before some example embodiments are discussed in more detail, it should be mentioned that some example embodiments are described as processes or methods depicted as flow diagrams. Although the processes are described in a certain order, many of the operations can be performed concurrently, in parallel, or simultaneously. In addition, the order of the operations can be re-arranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figure. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc. Furthermore, embodiments and features of the present application can be combined with each other as long as there is no conflict.

[0038] It should be noted that, in the embodiments of the present application, the words "exemplary" and "for example" are used to mean "an example of" or "an example, only. Any implementation or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or advantageous than other implementation or design solutions.

[0039] In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0040] Figure 1 A flowchart of an image denoising method provided by an embodiment of the present application, the embodiment can be applicable to the case where the glue spraying point noise image of the cigarette case needs to be denoised, and the method can be executed by an image denoising device. The device can be realized in the form of software and / or hardware. For example, the device can be an electronic device. Referring to Figure 1 The image denoising method of the embodiment specifically includes the following steps:

[0041] In step 110, the glue spraying point noise image and the attribute information of the glue spraying point noise image are obtained, and the glue spraying point noise image and the attribute information are input into the pre-trained image denoising model.

[0042] Specifically, the glue point noise image refers to an image containing cigarette case glue points, and random noise points in the image need to be removed. The attribute information of the glue point noise image refers to structured or unstructured additional data related to the glue point noise image, for example: the attribute information can be the type of the image, semantic information, acquisition time, acquisition device parameters, glue point position information, glue spraying process parameters, etc. The image denoising model refers to a model obtained after training according to historical glue point noise images (noisy input) and corresponding noise-free glue point images (or high-fidelity reference images). The image denoising model includes a global correlation module, a feature enhancement module, a noise prediction module, and an image denoising module.

[0043] In a specific implementation, the glue point image (i.e., the glue point noise image) to be detected can be obtained by an image acquisition device such as an industrial camera installed near the glue spraying area of the cigarette case. Then, the image is input into the trained image denoising model for processing. The model is trained based on historical glue point noise images (such as cigarette packet glue point images) and corresponding noise-free reference images, and can effectively identify and remove noise components in the input image, thereby generating a clear denoised image.

[0044] In addition, to further improve the quality and accuracy of the denoised image, the collected glue point noise image can be preprocessed, and specific means include image cropping, image enhancement, and artificial noise injection operations, so as to optimize the quality of the input image and improve the denoising effect of the model.

[0045] In this embodiment, the above steps lay a data foundation for obtaining a denoised image of the glue point noise image.

[0046] Step 120, using the global correlation module to perform kernel mapping attention processing on the glue point noise image and the attribute information to obtain global multi-modal features of the glue point noise image.

[0047] Specifically, the global correlation module refers to a module that performs kernel mapping attention processing on the glue point noise image and the attribute information. The global multi-modal feature refers to a feature that fuses the glue point noise image and the attribute information.

[0048] In a specific implementation, the glue point noise image can be encoded first to obtain an image vector, and features can be extracted from the image vector to obtain image features. At the same time, the attribute information can be encoded (for example, converted into a text vector through an embedding layer, and then optimized through a multi-layer perception machine) to obtain text features. Then, the text features are used to gate residual modulation of the image features to obtain first multi-modal intermediate features. Subsequently, a suitable attention mechanism (such as self-attention, cross-attention, etc.) can be selected according to actual conditions or requirements, and the selected attention mechanism is applied to kernel mapping attention processing of the first multi-modal intermediate features. Specifically, the first multi-modal intermediate features can be mapped to a high-dimensional feature space through a kernel function to enhance the non-linear expression ability of the features. Then, the attention weights of different feature positions are calculated to capture more fine correlation patterns and generate multi-modal attention features. Next, the text features are used to perform scale transformation on the multi-modal attention features to obtain first multi-modal scale adjustment features. Finally, the first multi-modal scale adjustment features and the image vector are connected in residual to perform element-level addition operation to obtain global multi-modal features.

[0049] In this embodiment, through the above steps, the deep fusion of cross-modal information is realized, effectively solving the limitations of a single mode (such as being susceptible to noise interference by relying only on the image mode), and the richness of the features is improved. On this basis, with the help of kernel mapping attention processing, the area related to the key information of the glue point (such as the edge and shape of the glue point) in the image can be automatically focused on, while effectively suppressing noise interference, further enhancing the anti-noise ability of feature extraction, which not only lays a more accurate and robust feature foundation for subsequent image denoising processing, but also helps to fully capture the distribution law and essential characteristics of noise.

[0050] Step 130, performing feature enhancement processing on the global multi-modal features by using a feature enhancement module to obtain enhanced multi-modal features of the glue point noise image.

[0051] Specifically, the feature enhancement module refers to a module for performing feature enhancement processing on the global multi-modal features. The enhanced multi-modal features refer to the global multi-modal features processed by the feature enhancement module.

[0052] In a specific implementation, the global multi-modal features can be subjected to layer normalization processing first to obtain normalized multi-modal features. Then, a gating signal is generated by using the text features, and the normalized multi-modal features are modulated based on the gating signal to obtain second multi-modal intermediate features. Next, the second multi-modal intermediate features are subjected to non-linear feature enhancement processing by using a feedforward neural network to obtain enhanced intermediate features. Subsequently, the enhanced intermediate features are subjected to scale adjustment by using the text features to obtain second multi-modal scale adjustment features. Finally, the second multi-modal scale adjustment features and the global multi-modal features are connected in residual to obtain the enhanced multi-modal features.

[0053] In this embodiment, through the above steps, the richness and adaptability of the features are effectively improved, the robustness of the features to noise is enhanced, the key details of the glue spraying points are preserved, and the accuracy of subsequent denoising processing is improved.

[0054] Step 140, decoding the enhanced multi-modal feature by using a noise prediction module to obtain noise features of the glue spraying point noise image.

[0055] Specifically, the noise prediction module refers to a module that processes the enhanced multi-modal feature through a decoding operation to extract noise-related information in the glue spraying point noise image. The noise features refer to the distribution characteristics of the noise in the glue spraying point noise image, for example: the noise features can include noise feature maps and covariance feature maps.

[0056] In a specific implementation, a suitable decoder (such as a transposed convolutional neural network, a decoder based on a Transformer architecture, a recurrent neural network and its variants, a multi-layer perception, a fully connected neural network, a linear regression model, etc.) can be selected according to actual conditions or requirements, and then the selected decoder is applied to decode the enhanced multi-modal feature to obtain noise features (such as noise feature maps, noise intensity vectors, covariance feature maps, etc.) of the glue spraying point noise image. For example: if the noise features are noise feature maps and covariance feature maps, the calculation formula can be: n, e = Decode(x n ), where n is the noise feature map, e is the covariance feature map, Decode is the selected decoder, such as a linear layer decoder, x n is the enhanced multi-modal feature.

[0057] In this embodiment, through the above steps, the enhanced multi-modal feature can be converted into "explainable, high-precision, and strongly adaptive" noise features, which not only accurately locate the noise information but also strengthen the explainability of itself, providing a targeted basis for subsequent denoising tasks, thereby effectively improving the denoising precision and the quality of the image after denoising.

[0058] Step 150, processing the noise features and the glue spraying point noise image by using an image denoising module to obtain a denoised image of the glue spraying point noise image.

[0059] Specifically, the image denoising module refers to a module that combines the noise features and the original glue spraying point noise image to perform denoising processing on the glue spraying point noise image. The denoised image refers to the glue spraying point image obtained after denoising processing.

[0060] In a specific implementation, a suitable de-noising method (such as statistical filtering, spatial domain filtering, transformer diffusion model, residual network, etc.) can be selected according to actual conditions or requirements, and then the selected de-noising method is applied to process the noise feature and the glue point noise image to obtain a de-noised image of the glue point noise image.

[0061] In the embodiment, the effective suppression of the glue point noise is achieved through the above steps, the quality of the de-noised image is improved, and more reliable image input is provided for subsequent glue point detection, thereby improving the detection accuracy.

[0062] The image de-noising method provided by the embodiment of the application first acquires a glue point noise image and attribute information of the glue point noise image, inputs the glue point noise image and the attribute information into a pre-trained image de-noising model, and lays a data foundation for subsequently obtaining a de-noised image of the glue point noise image. Then, the global correlation module is used to perform kernel mapping attention processing on the glue point noise image and the attribute information to obtain global multi-modal features of the glue point noise image, realize deep fusion of cross-modal information, improve the richness of the features, and effectively solve the limitations of a single mode. On this basis, with the help of kernel mapping attention processing, the region related to the key information (such as the edge and shape of the glue point) in the image can be automatically focused on, while the noise interference is effectively suppressed, and the anti-noise capability of feature extraction is further enhanced. Not only does this lay a more accurate and robust feature foundation for subsequent image de-noising processing, but also helps to comprehensively capture the distribution law and essential features of the noise. Then, the feature enhancement module is used to perform feature enhancement processing on the global multi-modal features to obtain enhanced multi-modal features of the glue point noise image, effectively improve the richness and adaptability of the features, enhance the robustness of the features to the noise, and at the same time retain the key details of the glue point, thereby improving the accuracy of subsequent de-noising processing. The noise prediction module is then used to decode the enhanced multi-modal features to obtain noise features of the glue point noise image, which not only helps to accurately locate the noise information, but also enhances the explainability of the obtained noise features, provides a targeted basis for subsequent de-noising tasks, and effectively improves the de-noising precision and the quality of the de-noised image. Finally, the image de-noising module is used to process the noise features and the glue point noise image to obtain a de-noised image of the glue point noise image, realize effective suppression of the glue point noise, improve the quality of the de-noised image, and provide more reliable image input for subsequent glue point detection, thereby improving the detection accuracy. Therefore, the technical solution of the application solves the problem of insufficient de-noising accuracy caused by relying only on single-level feature data for de-noising in the prior art.

[0063] Figure 2aA flowchart of another image denoising method provided by an embodiment of the present invention is provided. This embodiment is a specific embodiment based on the above embodiment. In this embodiment, the method may further include:

[0064] Step 210: Obtain the glue spray point noise image and the attribute information of the glue spray point noise image, and input the glue spray point noise image and attribute information into the pre-trained image denoising model.

[0065] Specifically, the image denoising model includes a global correlation module, a feature enhancement module, a noise prediction module, and an image denoising module.

[0066] For example, Figure 2b This is a schematic diagram of the structure of an image denoising model provided in an embodiment of the present invention, as shown below. Figure 2b As shown, the denoising process of this model is as follows: First, the input is the image of the glue spray point noise and its attribute information. The image of the glue spray point noise is segmented into image blocks; then, through linear projection, the features of the image blocks are mapped to a high-dimensional vector space to obtain image vectors. Next, the image vectors are subjected to layer normalization to obtain image features. Simultaneously, the attribute information of the glue spray point noise image is split using a text segmenter to obtain discrete text vectors. Then, a temporal dimension is added to the discrete text sequence, i.e., a time step is added, generating text vectors. Finally, a fully connected layer performs a linear transformation on the text vectors to obtain text features.

[0067] Next, the text features are mapped to the gating parameters (B1 and γ1) of the first gating unit. The first gating unit combines its own gating parameters to perform gated residual modulation on the image features, obtaining the first multimodal intermediate features. The first multimodal intermediate features are input into the attention mechanism unit and processed through kernel mapping attention to obtain multimodal attention features. Then, the text features are mapped to the first scale weight (α1), and the multimodal attention features are scaled to obtain the first multimodal scale-adjusted features. Finally, the first multimodal scale-adjusted features and the image vector are residually concatenated to obtain the global multimodal features. The process of obtaining global multimodal features from image vectors can be represented by the following formula: x1 = x p +α1·PERFORMER(GSU1(LN(x p )));where x1 is the global multimodal feature, x p is the image vector, PERFORMER is the attention mechanism used by the attention mechanism unit, GSU1 is the first gating unit, and LN is the layer normalization network.

[0068] Next, the global multi-modal feature is subjected to layer normalization processing to obtain a normalized multi-modal feature; the text feature is mapped to gate parameters (β2 and γ2) of a second gating unit, and then the second gating unit utilizes its own gate parameters to perform gate residual modulation on the normalized multi-modal feature to obtain a second multi-modal intermediate feature; the second multi-modal intermediate feature is input into a feed-forward neural network (FFN), i.e., a feature enhancement unit, to perform feature enhancement processing on the second multi-modal intermediate feature through nonlinear transformation to obtain an enhanced intermediate feature; the text feature is mapped to a second scale weight (α2) to perform scale transformation on the enhanced intermediate feature to obtain a second multi-modal scale-adjusted feature; the second multi-modal scale-adjusted feature and the global multi-modal feature are subjected to residual connection to obtain an enhanced multi-modal feature. The process of obtaining the enhanced multi-modal feature from the global multi-modal feature can be represented by the following formula: x2 = x1 + α2·FFN(GSU2(LN(x1))); wherein x2 is the enhanced multi-modal feature, FFN is the feed-forward neural network, GSU2 is the second gating unit, and LN is the layer normalization network.

[0069] Finally, the enhanced multi-modal feature is decoded by the noise prediction module to obtain a noise feature of the glue spraying point noise image, and the noise feature and the glue spraying point noise image are processed by the image denoising module to obtain a denoised image of the glue spraying point noise image.

[0070] In addition, to further improve the feature depth, the process in the dashed box in the flowchart can be repeated N times (N≥1).

[0071] Further, the training process of the image denoising model is as follows: first, obtain each historical glue spraying point noise image and the corresponding noise-free glue spraying point image. Clean and preprocess these data to ensure that they meet the input requirements of the model. Then, use the obtained historical glue spraying point noise image and the corresponding noise-free glue spraying point image as training data to train the deep learning model. This process usually includes two steps of forward propagation and back propagation. Forward propagation is to pass the input data through the model to obtain the prediction result, and then calculate the loss between the prediction result and the true target. Back propagation is to update the parameters of the model according to the loss function (such as the discriminative loss function, the mean square error loss function, the diffusion loss function, etc.) to reduce the difference between the prediction result and the true target. The deep learning model can be a diffusion model, a diffusion model combined with a Transformer, a diffusion model combined with a discriminator, etc., and the embodiments of the present application do not limit this. Finally, the model can be optimized based on the back propagation algorithm until the loss function converges, thereby obtaining the image denoising model. For example, gradient descent or Adam can be selected to determine the learning rate and other hyperparameters to optimize the model. For example, the specific formula of the discriminative loss function is as follows:

[0072]

[0073] wherein, L disc is a discriminant loss function value, λ t is a weight coefficient, t is a time step, I noisy is a glue point noise image, I denoised is a denoised image.

[0074] In practical applications, in order to improve the training efficiency of the image denoising model, a mean square error loss function can be used, and the specific calculation formula is as follows:

[0075]

[0076] wherein, L simple (θ) is a mean square error loss function value, is a predicted noise, is a real noise.

[0077] Optionally, the global correlation module includes a first feature encoding unit, a second feature encoding unit, a first gating unit and an attention mechanism unit.

[0078] In step 211, the first feature encoding unit is used to encode the glue point noise image to obtain an image vector, and the second feature encoding unit is used to encode the attribute information to obtain a text vector.

[0079] Specifically, the first feature encoding unit refers to a unit for encoding the glue point noise image. The image vector refers to a vector obtained by processing the glue point noise image through the first feature encoding unit. The second feature encoding unit refers to a unit for encoding the attribute information. The text vector refers to a vector obtained by processing the attribute information through the second feature encoding unit.

[0080] In a specific implementation, a suitable encoding technology (such as a convolutional neural network, image block encoding of a Transformer, etc.) can be selected according to actual conditions or requirements, and then the selected encoding technology is applied to encode the glue point noise image to obtain the image vector. For example, a visual Transformer (VIT) can be used as the encoding technology, and the built-in image block function of the VIT is used to encode the glue point noise image to obtain the image vector, and the specific calculation formula is as follows: wherein, x p is an image vector, I noisy is a glue point noise image, Patchify() is an image block function of the VIT, N p is the total number of image blocks, and D is the vector dimension after embedding of each image block, i.e., the dimension of the image vector.

[0081] Similarly, a suitable encoding technology (such as one-hot encoding expansion, byte pair encoding, text encoding module of Transformer, word embedding model, etc.) can be selected according to actual conditions or needs, and then the selected encoding technology is applied to encode the attribute information (such as the category of the glue spraying point noise image) of the glue spraying point noise image to obtain a text vector. For example, byte pair encoding (BPE) can be used as the encoding technology, and its subword segmentation algorithm is used to encode the attribute information of the glue spraying point noise image to obtain a text vector. The specific calculation formula is: where x t is the text vector, t cond is the attribute information of the glue spraying point noise image, Tokenizer() is the subword segmentation algorithm of BPE, N t is the number of text tokens, and D is the dimension of the text vector.

[0082] In this embodiment, through the above steps, the core features (such as the position, shape, distribution density, and noise interference mode of the glue spraying point) of the glue spraying point noise image can be focused on, irrelevant information can be effectively filtered out, and key visual features related to the quality of the glue spraying can be highlighted, thereby improving the quality of the subsequent image features. At the same time, the discrete attribute information is converted into a continuous text vector, which lays a foundation for realizing cross-modal fusion.

[0083] In step 212, the image vector is subjected to layer normalization processing to obtain an image feature, and the text vector is subjected to linear transformation to obtain a text feature.

[0084] Specifically, the image feature refers to the feature obtained by subjecting the image vector to layer normalization processing. The text feature refers to the feature obtained by subjecting the text vector to linear transformation.

[0085] In a specific implementation, the image vector can be subjected to layer normalization processing to obtain an image feature, for example, by calculating the mean and variance of each image vector, converting each element in the vector into a distribution with a mean of 0 and a variance of 1, thereby obtaining the layer normalized image feature. At the same time, the text vector is subjected to linear transformation by using a fully connected layer to obtain a text feature.

[0086] In this embodiment, the layer normalization processing of the image vector can improve the stability of the obtained image feature and ensure the consistency of the noise-related visual features. On the other hand, the linear transformation of the text vector can obtain a text feature with a dimension matching that of the image feature, ensuring that the features of the two modalities can interact in the same space (such as through parameter regulation of a gating unit), and ensuring that the “prior knowledge” of the text feature can effectively act on the image feature processing, thereby improving the quality of the subsequent first multi-modal intermediate feature.

[0087] Further, the first feature encoding unit is used to encode the glue point noise image to obtain an image vector, including: performing image block processing on the glue point noise image to obtain an image block; performing linear projection on the image block to obtain the image vector; correspondingly, the second feature encoding unit is used to encode the attribute information to obtain a text vector, including: segmenting the attribute information to obtain a discrete text sequence; performing embedding mapping on the discrete text sequence to obtain a discrete text vector; and fusing the discrete text vector and a time step to obtain the text vector.

[0088] Specifically, the image block refers to a smaller and local rectangular area formed by segmenting the glue point noise image according to a specific rule. The discrete text sequence refers to a discrete text unit sequence obtained by segmenting the attribute information. The discrete text vector refers to a vector obtained by embedding mapping on the discrete text sequence. The time step refers to the position or order of each element in the text sequence.

[0089] In a specific implementation, the glue point noise image can be uniformly segmented into non-overlapping sub-regions according to a preset size (such as 16x16 pixels) to obtain the image block. Then, linear projection is performed on each image block to obtain the image vector.

[0090] Correspondingly, the attribute information can be segmented into tokens according to semantic units to obtain the discrete text sequence; then the discrete text sequence is mapped into a vector through a pre-trained word embedding model (such as Word2Vec, GloVe, etc.) or a self-defined embedding matrix to obtain the discrete text vector; then a unique time step index is assigned to each Token in the discrete text vector, and a position encoding vector with the same dimension as the text vector is generated through a sine or cosine function, and is added to the text vector element by element to obtain the text vector.

[0091] In this embodiment, the image block is obtained by performing image block processing on the glue point noise image, which can focus on the local features of the glue points (such as the shape, edge and noise distribution of a single glue point), and also retains the global layout information through the collection of all image blocks; then the image block is linearly projected to obtain the image block embedding vector, which can reduce the data dimension while retaining the key visual information, and reduce the subsequent calculation burden. Correspondingly, the attribute information is segmented to obtain the discrete text sequence; the discrete text sequence is embedded and mapped to obtain the discrete text vector, which provides a semantic basis for subsequent cross-modal interaction; and the discrete text vector and the time step are fused to obtain the text vector, which can ensure the accuracy of the timing information of the text and guarantee the semantic integrity of the obtained text features.

[0092] In step 213, the first gating unit is used for gated residual modulation on the image feature to obtain a first multi-modal intermediate feature.

[0093] Specifically, the first gating unit refers to a unit for gated residual modulation on the image feature. The first multi-modal intermediate feature refers to a feature obtained after the gated residual modulation on the image feature.

[0094] In a specific implementation, the multi-layer perception can be used to map the text feature to a gating adjustment parameter (such as a scale parameter, an offset parameter, etc.), and the specific calculation formula is as follows: β, γ = MLP(x t ), where β is the scale parameter, γ is the offset parameter, MLP is the multi-layer perception, and x t is the text feature. Then, the gated switch unit (GSU) is used to perform element-by-element operation on the image feature and the gating adjustment parameter to obtain the first multi-modal intermediate feature.

[0095] In this embodiment, through the above steps, not only is the text semantic used for the optimization guidance of the image feature, but also is the multi-modal information deeply fused, the accuracy of the feature representation is improved, and the feature expression ability is enhanced.

[0096] In step 214, the attention mechanism unit is used for kernel mapping attention processing on the first multi-modal intermediate feature to obtain a multi-modal attention feature.

[0097] Specifically, the attention mechanism unit refers to a unit for kernel mapping attention processing on the first multi-modal intermediate feature. The multi-modal attention feature refers to a feature obtained after the kernel mapping attention processing on the first multi-modal intermediate feature.

[0098] In a specific implementation, a suitable attention mechanism (such as self-attention, cross-attention, etc.) can be selected according to actual conditions or requirements, and then the selected attention mechanism is applied to the kernel mapping attention processing on the first multi-modal intermediate feature. Specifically, the first multi-modal intermediate feature can be mapped to a high-dimensional feature space through a kernel function to enhance the nonlinear expression ability of the feature; then the attention weights of different feature positions are calculated to capture more fine-grained correlation patterns to generate the multi-modal attention feature.

[0099] In this embodiment, through the above steps, not only is the computational complexity reduced, the computational efficiency improved, and the nonlinear relationship modeling ability enhanced, but also is the multi-modal fine-grained interaction better captured, the key information focused, and the noise interference suppressed, thereby improving the feature discriminability and strengthening the cross-modal consistency, and providing a more reliable feature basis for subsequent processing tasks.

[0100] Further, step 214 can specifically include: performing linear transformation on the first multi-modal intermediate feature by using different transformation parameters to obtain a key vector, a value vector and a query vector; performing kernel space projection on the key vector by using a kernel mapping function to obtain a mapped key vector, and performing kernel space projection on the query vector by using the kernel mapping function to obtain a mapped query vector; and performing kernel normalization attention calculation on the mapped key vector, the value vector and the mapped query vector to obtain a multi-modal attention feature.

[0101] Specifically, the mapped key vector refers to a vector obtained by mapping the key vector to a kernel space by using the kernel mapping function. The mapped query vector refers to a vector obtained by mapping the query vector to the kernel space by using the kernel mapping function.

[0102] In a specific implementation, the first multi-modal intermediate feature can be first linearly transformed by using different transformation parameters (such as three independent matrices) to obtain the key vector, the value vector and the query vector. Then, the kernel space projection is performed on the key vector and the query vector by using the kernel mapping function respectively to obtain the mapped key vector and the mapped query vector, and the formula of the kernel mapping function is as follows:

[0103]

[0104] wherein, is the output of the kernel mapping function, such as the mapped key vector or the mapped query vector; x is the input of the kernel mapping function, such as the key vector or the query vector; is the square of the L2 norm of x; exp() is an exponential decay factor; ω i is the i th cosine transformation weight vector, and there are m in total; b i is the i th bias term of the cosine transformation, and there are m in total.

[0105] Finally, the kernel normalization attention calculation is performed on the mapped key vector, the value vector and the mapped query vector to obtain the multi-modal attention feature, and the specific calculation formula is as follows:

[0106]

[0107] wherein, Performer(Q, K, V) is the multi-modal attention feature, is the mapped query vector, is the mapped key vector, T is a transpose, V is the value vector, is a full 1 vector.

[0108] In this embodiment, by using the approximation calculation, the attention calculation complexity is reduced from O(N 2) is reduced to O(N), not only significantly improving the computing efficiency, but also supporting processing longer sequence data to enhance the global feature capturing ability. At the same time, the low-dimensional features are mapped to high-dimensional Hilbert space by using the kernel mapping function, breaking through the linear bottleneck - making the nonlinear association that already exists in the original feature space linearly separable in the high-dimensional space; and allowing different position features to interact and integrate efficiently in the high-dimensional space, accurately capturing global dependency relationships. In addition, by focusing on key areas through kernel normalization operation, local feature expression is strengthened, so that the finally generated multi-modal attention features can simultaneously fuse global dependency relationships and local detail information.

[0109] Step 215, scale transformation is performed on the multi-modal attention feature by using the text feature to obtain a first multi-modal scale adjusted feature.

[0110] Specifically, the first multi-modal scale adjusted feature refers to a feature obtained after scale transformation is performed on the multi-modal attention feature by using the text feature.

[0111] In a specific implementation, another multi-layer perception machine can be used to convert the text feature into a scale weight aligned with the dimension of the multi-modal attention feature, and then the obtained scale weight is used to perform element-wise weighting on the multi-modal attention feature to obtain the first multi-modal scale adjusted feature.

[0112] In this embodiment, the "fine-grained binding" of the text and the multi-modal feature is realized through the above steps, which can not only avoid cross-modal information misplacement, but also improve the discriminability and robustness of the feature, so that the effective information in the feature is more prominent and the interference information is more weakened.

[0113] Step 216, residual connection is performed on the first multi-modal scale adjusted feature and the image vector to obtain a global multi-modal feature.

[0114] In a specific implementation, after obtaining the first multi-modal scale adjusted feature, an element-wise addition operation can be performed on the first multi-modal scale adjusted feature and the image vector to obtain the global multi-modal feature.

[0115] In this embodiment, the complementary fusion of "basic visual information" and "multi-modal optimized information" is realized through the above steps, which provides more comprehensive feature support for the subsequent image denoising task, and further improves the accuracy of subsequent denoising.

[0116] Step 217, a feature enhancement module is used to perform feature enhancement processing on the global multi-modal feature to obtain an enhanced multi-modal feature of the glue spraying point noise image.

[0117] Optionally, the feature enhancement module includes a normalization unit, a second gating unit, and a feature enhancement unit.

[0118] Further, step 217 can specifically include: performing layer normalization processing on the global multi-modal feature by using a normalization unit to obtain a normalized multi-modal feature; performing gated residual modulation on the normalized multi-modal feature by using a second gating unit to obtain a second multi-modal intermediate feature; performing nonlinear feature enhancement processing on the second multi-modal intermediate feature by using a feature enhancement unit to obtain an enhanced intermediate feature; performing scale transformation on the enhanced intermediate feature by using the text feature to obtain a second multi-modal scale adjusted feature; and performing residual connection on the second multi-modal scale adjusted feature and the global multi-modal feature to obtain an enhanced multi-modal feature.

[0119] Specifically, the normalization unit refers to a unit for performing layer normalization processing on the global multi-modal feature. The normalized multi-modal feature refers to a feature obtained after the normalization unit processing. The second gating unit refers to a unit for performing gated residual modulation on the normalized multi-modal feature. The second multi-modal intermediate feature refers to a feature obtained after the second gating unit processing. The feature enhancement unit refers to a unit for performing nonlinear feature enhancement processing on the second multi-modal intermediate feature. The enhanced intermediate feature refers to a feature obtained after the feature enhancement unit processing. The second multi-modal scale adjusted feature refers to a feature obtained after performing scale transformation on the enhanced intermediate feature by using the text feature.

[0120] In a specific implementation, the global multi-modal feature can be first processed by layer normalization to obtain a normalized multi-modal feature. Then, another multi-layer perception is used to map the text feature to the gating adjustment parameters (such as scale parameters, offset parameters, etc.) required by the second gating unit. After that, another gating switch unit is used to perform element-by-element operation (such as feature x scale parameter + offset parameter) on the normalized multi-modal feature and the aforementioned obtained gating adjustment parameters to obtain a second multi-modal intermediate feature. Next, the point-level feedforward neural network is used to perform nonlinear feature enhancement processing on the second multi-modal intermediate feature to obtain an enhanced intermediate feature. Then, another multi-layer perception is used to convert the text feature into scale weights aligned with the enhanced intermediate feature. After that, the enhanced intermediate feature is scaled by using the weights to obtain a second multi-modal scale adjusted feature. Finally, element-by-element addition operation is performed on the second multi-modal scale adjusted feature and the global multi-modal feature to obtain an enhanced multi-modal feature.

[0121] In this embodiment, the global multi-modal features are first normalized to ensure stable feature distribution, and then two text-guided operations (gate modulation and scale transformation) are used to achieve deeper cross-modal semantic binding. Then, non-linear enhancement is used to capture high-order feature patterns, strengthening the association between local details and global structure: avoiding overfitting of simple feature patterns by the model, improving the adaptability of features to complex scenes. Finally, residual connection is used to avoid loss of global information caused by multiple rounds of enhancement operations, and to integrate the details of the second enhancement, so that the final enhanced multi-modal features have both global vision and fine-grained details, providing more comprehensive and accurate feature support for subsequent image denoising tasks.

[0122] Step 218, using a noise prediction module to decode the enhanced multi-modal features to obtain noise features of the glue point noise image.

[0123] Optionally, the noise prediction module includes a prediction unit and a splitting unit, and the noise features include a noise feature map and a covariance feature map.

[0124] Further, step 218 can specifically include: using the prediction unit to perform linear transformation on the enhanced multi-modal features to obtain intermediate noise features; and using the splitting unit to split the intermediate noise features to obtain the noise feature map and the covariance feature map.

[0125] Specifically, the prediction unit refers to a unit that performs linear transformation on the enhanced multi-modal features. The splitting unit refers to a unit that splits the intermediate noise features. The intermediate noise features refer to features obtained after linear transformation by the prediction unit, which contain preliminary information related to noise. The noise feature map refers to a feature map representing the distribution of noise in the glue point noise image. The covariance feature map refers to a feature map representing the correlation between noise.

[0126] In a specific implementation, the enhanced multi-modal features can be mapped to a new feature space through a linear layer (such as a fully connected or convolutional layer), thereby obtaining intermediate noise features. Then, the intermediate noise features can be evenly divided into two parts along the channel dimension: the first half can be directly used as the noise feature map to estimate the noise intensity at each position in the image, and the second half is mapped to non-negative values through a non-linear transformation to serve as the covariance feature map to represent the uncertainty of noise estimation.

[0127] In this embodiment, through the above steps, on the one hand, the calculation process of the noise features is simplified, the implementation complexity of determining the noise features is reduced, and the calculation efficiency is improved; on the other hand, by outputting the noise feature map and the covariance feature map, respectively from the "noise position marking" and "noise fluctuation quantization" dimensions, support is provided for accurate differentiation between noise and normal areas in industrial scenarios, robust response to complex noise, precise positioning of noise, and characteristic analysis of noise.

[0128] Step 219, processing the noise feature and the glue point noise image by using an image denoising module to obtain a denoised image of the glue point noise image.

[0129] Further, step 219 can specifically include: denoising the glue point noise image by using the noise feature map to obtain an initial denoised image; and correcting the initial denoised image by using the covariance feature map to obtain the denoised image.

[0130] Specifically, the initial denoised image refers to a preliminary denoising result obtained by denoising the glue point noise image by using the noise feature map.

[0131] In a specific implementation, residual calculation can be performed on the glue point noise image and the noise feature map to obtain the initial denoised image. Then, the initial denoised image is corrected by using a preset covariance adjustment strategy (obtained in a model training stage) and the covariance feature map to obtain the denoised image. For example, the covariance adjustment strategy learned in the training stage is used to determine a covariance weight coefficient, and the weight coefficient is multiplied by the covariance feature map to obtain a covariance correction amount. Finally, the initial denoised image is corrected according to the formula "denoised image = initial denoised image - covariance correction amount" to obtain the denoised image.

[0132] In this embodiment, the above steps can make the denoising more targeted, avoid excessive processing of normal areas, reduce the computational overhead, and thus improve the accuracy of the determined denoised image.

[0133] The image denoising method provided by the embodiment of the application first acquires a glue spraying point noise image and attribute information of the glue spraying point noise image, inputs the glue spraying point noise image and the attribute information into a pre-trained image denoising model, and lays a data foundation for obtaining a denoised image of the glue spraying point noise image subsequently. Then, a first feature encoding unit is used to encode the glue spraying point noise image to obtain an image vector, and a second feature encoding unit is used to encode the attribute information to obtain a text vector, which can focus on core features (such as the position, shape, distribution density and noise interference mode of the glue spraying point) of the glue spraying point noise image, effectively filter irrelevant information, highlight key visual features related to glue quality, and thus improve the quality of subsequent image features. At the same time, the discrete attribute information is converted into a continuous text vector, which not only lays a foundation for realizing cross-modal fusion and improves the quality of subsequent text features. The image vector is subjected to layer normalization processing to obtain an image feature, and the text vector is subjected to linear transformation to obtain a text feature, which can improve the stability of the image feature and ensure the consistency of noise-related visual features; at the same time, the text feature with a dimension matching that of the image feature can be obtained, ensuring that the features of the two modalities can interact in the same space (such as being regulated by the parameters of a gating unit), so that the “prior knowledge” contained in the text feature effectively acts on the image feature processing, and thus the quality of the first multi-modal intermediate feature is improved. The first gating unit is used to gate residual modulation of the image feature to obtain the first multi-modal intermediate feature, which not only realizes the optimization and guidance of the text semantics to the image feature, but also realizes deep fusion of multi-modal information, improves the accuracy of feature representation, and enhances the feature expression ability. The attention mechanism unit is used to perform kernel mapping attention processing on the first multi-modal intermediate feature to obtain a multi-modal attention feature, which not only reduces the computational complexity, improves the computational efficiency, enhances the non-linear relationship modeling capability, but also better captures multi-modal fine-grained interaction, focuses on key information and suppresses noise interference, and thus improves the feature discriminability and strengthens the cross-modal consistency, providing a more reliable feature basis for subsequent processing tasks. The text feature is used to perform scale transformation on the multi-modal attention feature to obtain a first multi-modal scale adjustment feature, realizing “fine-grained binding” of the text and the multi-modal feature, which can not only avoid cross-modal information misplacement, but also improve the discriminability and robustness of the feature, making the effective information in the feature more prominent and the interference information weaker. The first multi-modal scale adjustment feature and the image vector are subjected to residual connection to obtain a global multi-modal feature, realizing complementary fusion of “basic visual information” and “multi-modal optimization information”, providing more comprehensive feature support for subsequent image denoising tasks, and thus improving the accuracy of subsequent denoising. Then, the feature enhancement module is used to perform feature enhancement processing on the global multi-modal feature to obtain an enhanced multi-modal feature of the glue spraying point noise image, effectively improving the richness and adaptability of the feature, enhancing the robustness of the feature to noise, while retaining the key details of the glue spraying point, and thus improving the accuracy of subsequent denoising processing.The noise prediction module is used for decoding the enhanced multi-modal feature to obtain the noise feature of the glue spraying point noise image, which not only helps to accurately locate the noise information, but also enhances the explainability of the obtained noise feature, provides a targeted basis for a subsequent denoising task, and effectively improves the denoising precision and the quality of the denoised image. Finally, the image denoising module is used for processing the noise feature and the glue spraying point noise image to obtain a denoised image of the glue spraying point noise image, effectively suppresses the glue spraying point noise, improves the quality of the denoised image, and provides a more reliable image input for a subsequent glue spraying point detection, and improves the detection precision. Therefore, the technical scheme of the present application solves the problem of insufficient denoising accuracy caused by using only single-level feature data for denoising in the prior art.

[0134] Figure 3 The structure of the image denoising device provided by the embodiment of the present application is shown in the figure. The device and the image denoising method of each embodiment described above belong to the same inventive concept. Details not described in detail in the embodiment of the image denoising device can be referred to the embodiment of the image denoising method described above.

[0135] As shown in the figure, the device comprises: Figure 3

[0136] The acquisition module 310 is configured to acquire a glue spraying point noise image and attribute information of the glue spraying point noise image, and input the glue spraying point noise image and the attribute information into a pre-trained image denoising model. The image denoising model comprises a global correlation module, a feature enhancement module, a noise prediction module, and an image denoising module.

[0137] The feature extraction module 320 is configured to perform kernel mapping attention processing on the glue spraying point noise image and the attribute information by using the global correlation module to obtain global multi-modal features of the glue spraying point noise image.

[0138] The feature enhancement module 330 is configured to perform feature enhancement processing on the global multi-modal features by using the feature enhancement module to obtain enhanced multi-modal features of the glue spraying point noise image.

[0139] The decoding module 340 is configured to decode the enhanced multi-modal features by using the noise prediction module to obtain noise features of the glue spraying point noise image.

[0140] The denoising module 350 is configured to process the noise features and the glue spraying point noise image by using the image denoising module to obtain a denoised image of the glue spraying point noise image.

[0141] ​On the basis of the above-mentioned embodiment, the global correlation module comprises a first feature encoding unit, a second feature encoding unit, a first gating unit and an attention mechanism unit, and the feature extraction module 320 is specifically configured to:

[0142] The first feature encoding unit is configured to encode the glue point noise image to obtain an image vector, and the second feature encoding unit is configured to encode the attribute information to obtain a text vector;

[0143] The image vector is subjected to layer normalization processing to obtain the image feature, and the text vector is subjected to linear transformation to obtain the text feature;

[0144] The first gating unit is configured to perform gating residual modulation on the image feature to obtain a first multi-modal intermediate feature;

[0145] The attention mechanism unit is configured to perform kernel mapping attention processing on the first multi-modal intermediate feature to obtain a multi-modal attention feature;

[0146] The text feature is configured to perform scale transformation on the multi-modal attention feature to obtain a first multi-modal scale adjustment feature;

[0147] The first multi-modal scale adjustment feature and the image vector are subjected to residual connection to obtain the global multi-modal feature.

[0148] On the basis of the above-mentioned embodiment, the feature extraction module 320 utilizes the first feature encoding unit to encode the glue point noise image to obtain an image vector, comprising:

[0149] The glue point noise image is subjected to image block processing to obtain an image block;

[0150] The image block is subjected to linear projection to obtain the image vector.

[0151] Correspondingly, the feature extraction module 320 utilizes the second feature encoding unit to encode the attribute information to obtain a text vector, comprising:

[0152] The attribute information is segmented to obtain a discrete text sequence;

[0153] The discrete text sequence is subjected to embedding mapping to obtain a discrete text vector;

[0154] The discrete text vector and the time step are fused to obtain the text vector.

[0155] On the basis of the above-mentioned embodiments, the feature extraction module 320 utilizes the attention mechanism unit to perform kernel mapping attention processing on the first multi-modal intermediate feature to obtain a multi-modal attention feature, including:

[0156] The first multi-modal intermediate feature is linearly transformed using different transformation parameters to obtain a key vector, a value vector, and a query vector;

[0157] The key vector is projected in the kernel space using a kernel mapping function to obtain a mapped key vector, and the query vector is projected in the kernel space using the kernel mapping function to obtain a mapped query vector;

[0158] The mapped key vector, the value vector, and the mapped query vector are subjected to kernel normalization attention calculation to obtain the multi-modal attention feature.

[0159] On the basis of the above-mentioned embodiments, the feature enhancement module includes a normalization unit, a second gating unit, and a feature enhancement unit, the feature enhancement module 330, specifically for:

[0160] The global multi-modal feature is subjected to layer normalization processing using the normalization unit to obtain a normalized multi-modal feature;

[0161] The normalized multi-modal feature is subjected to gating residual modulation using the second gating unit to obtain a second multi-modal intermediate feature;

[0162] The second multi-modal intermediate feature is subjected to nonlinear feature enhancement processing using the feature enhancement unit to obtain an enhanced intermediate feature;

[0163] The enhanced intermediate feature is subjected to scale transformation using the text feature to obtain a second multi-modal scale-adjusted feature;

[0164] The second multi-modal scale-adjusted feature and the global multi-modal feature are subjected to residual connection to obtain the enhanced multi-modal feature.

[0165] On the basis of the above-mentioned embodiments, the noise prediction module includes a prediction unit and a splitting unit, the noise feature includes a noise feature map and a covariance feature map, and the decoding module 340 is specifically for:

[0166] The enhanced multi-modal feature is linearly transformed using the prediction unit to obtain an intermediate noise feature;

[0167] The intermediate noise feature is split using the splitting unit to obtain the noise feature map and the covariance feature map.

[0168] On the basis of the above-mentioned embodiments, the denoising module 350 is specifically for:

[0169] The noise feature map is used for denoising the glue spraying point noise image, and an initial denoised image is obtained.

[0170] The initial denoised image is corrected by using the covariance feature map, and a denoised image is obtained.

[0171] The image denoising device provided in the embodiments of the present application can execute the image denoising method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0172] It should be noted that in the embodiments of the above image denoising device, each unit and module included is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for easy mutual distinction, and does not serve to limit the protection scope of the present application.

[0173] Figure 4 A structural schematic diagram of an electronic device provided in the embodiments of the present application is provided. Figure 4 A block diagram of an exemplary electronic device 4 suitable for implementing embodiments of the present application is shown. Figure 4 The electronic device 4 shown is merely an example and should not impose any limitation on the function and use range of the embodiments of the present application.

[0174] As shown in Figure 4 The electronic device 4 is shown in the form of a general computing electronic device. The components of the electronic device 4 can include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects the various system components including the system memory 28 and the processing unit 16.

[0175] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0176] The electronic device 4 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 4 and includes both volatile and nonvolatile media, removable and non-removable media.

[0177] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 4 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 4 Although not shown, a magnetic disk drive can also be utilized in some embodiments to read from and write to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be utilized in some embodiments to read from and write to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM or other optical media). Figure 4 Although not shown, a magnetic disk drive can also be utilized in some embodiments to read from and write to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be utilized in some embodiments to read from and write to a removable, non-volatile optical disk (e.g., a CD-ROM, DVD-ROM or other optical media).

[0178] Program / utility 40 having a set (at least one) of program modules 42 can be stored in, for example, system memory 28 by way of example, and can include an operating system, one or more application programs, other program modules, and program data, each of or some combination of which can provide functionality for implementing embodiments of the present application.

[0179] Electronic device 4 can also communicate with one or more external devices 14 such as a keyboard or pointing device, a display 24, etc.; other devices such as are necessary to facilitate interaction with a user. By way of example, communication can be achieved by input / output (I / O) interfaces 22. Additionally, electronic device 4 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, by way of network adapter 20. As will be appreciated, other hardware and / or software elements can be utilized in conjunction with electronic device 4 such as, for example, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. Figure 4 As will be appreciated, although not shown, other hardware and / or software elements can be utilized in conjunction with electronic device 4 such as, for example, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. ​ As will be appreciated, although not shown, other hardware and / or software elements can be utilized in conjunction with electronic device 4 such as, for example, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0180] The processing unit 16 executes various functional applications and page displays by running programs stored in the system memory 28, such as implementing the image denoising method provided by the embodiments of the present application, which comprises:

[0181] The spray point noise image and the attribute information of the spray point noise image are obtained, and the spray point noise image and the attribute information are input into a pre-trained image denoising model, wherein the image denoising model comprises a global correlation module, a feature enhancement module, a noise prediction module and an image denoising module.

[0182] The spray point noise image and the attribute information are subjected to kernel mapping attention processing by using the global correlation module, so as to obtain global multi-modal features of the spray point noise image.

[0183] The global multi-modal features are subjected to feature enhancement processing by using the feature enhancement module, so as to obtain enhanced multi-modal features of the spray point noise image.

[0184] The enhanced multi-modal features are decoded by using the noise prediction module, so as to obtain noise features of the spray point noise image.

[0185] The noise features and the spray point noise image are processed by using the image denoising module, so as to obtain a denoised image of the spray point noise image.

[0186] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the image denoising method provided by any of the embodiments of the present application.

[0187] The computer readable storage medium provided by the embodiments of the present application stores a computer program, and the program is executed by the processor to implement, for example, the image denoising method provided by the embodiments of the present application, which comprises:

[0188] The spray point noise image and the attribute information of the spray point noise image are obtained, and the spray point noise image and the attribute information are input into a pre-trained image denoising model, wherein the image denoising model comprises a global correlation module, a feature enhancement module, a noise prediction module and an image denoising module.

[0189] The spray point noise image and the attribute information are subjected to kernel mapping attention processing by using the global correlation module, so as to obtain global multi-modal features of the spray point noise image.

[0190] The global multi-modal features are subjected to feature enhancement processing by using the feature enhancement module, so as to obtain enhanced multi-modal features of the spray point noise image.

[0191] The noise prediction module is used for decoding the enhanced multi-modal feature to obtain a noise feature of the glue spraying point noise image.

[0192] The image denoising module is used for processing the noise feature and the glue spraying point noise image to obtain a denoised image of the glue spraying point noise image.

[0193] The computer storage medium of the embodiment of the application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0194] The computer readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave propagating through a transmission medium, in which the computer readable program code is carried. Such a propagating data signal can take on many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device.

[0195] The program code contained on the computer readable medium can be transmitted in any suitable medium, including but not limited to wireless, wire, cable, optical fiber, RF, etc., or any suitable combination of the above.

[0196] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0197] Those skilled in the art will appreciate that the modules or steps of the present application described above can be implemented using general computing devices, which can be centralized on a single computing device or distributed on a network of multiple computing devices. Alternatively, they can be implemented using computer-executable program code, which can be stored in a storage device and executed by a computing device, or they can be implemented as individual integrated circuit modules, or a plurality of modules or steps can be implemented as a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0198] In addition, the acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of laws and regulations.

[0199] Note that the above are only preferred embodiments of the present application and the technical principles applied. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, re-adjustments and substitutions can be made by those skilled in the art without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. An image denoising method, characterized in that, The method includes: Obtain the glue spray point noise image and the attribute information of the glue spray point noise image, and input the glue spray point noise image and the attribute information into a pre-trained image denoising model. The image denoising model includes a global association module, a feature enhancement module, a noise prediction module and an image denoising module. The global association module is used to perform kernel mapping attention processing on the glue spray point noise image and the attribute information to obtain the global multimodal features of the glue spray point noise image; The feature enhancement module is used to perform feature enhancement processing on the global multimodal features to obtain the enhanced multimodal features of the glue spray point noise image; The noise prediction module is used to decode the enhanced multimodal features to obtain the noise features of the glue spray point noise image; The image denoising module is used to process the noise features and the glue spray point noise image to obtain a denoised image of the glue spray point noise image.

2. The image denoising method according to claim 1, characterized in that, The global association module includes a first feature encoding unit, a second feature encoding unit, a first gating unit, and an attention mechanism unit. The global association module performs kernel mapping attention processing on the glue-spraying point noise image and the attribute information to obtain the global multimodal features of the glue-spraying point noise image, including: The first feature encoding unit is used to encode the noise image of the glue spray point to obtain an image vector, and the second feature encoding unit is used to encode the attribute information to obtain a text vector; The image vector is subjected to layer normalization to obtain the image features, and the text vector is subjected to linear transformation to obtain the text features; The first gating unit is used to perform gated residual modulation on the image features to obtain the first multimodal intermediate features; The attention mechanism unit is used to perform kernel mapping attention processing on the first multimodal intermediate features to obtain multimodal attention features; The multimodal attention features are scaled using the text features to obtain the first multimodal scale-adjusted features; The first multimodal scaling feature and the image vector are residually concatenated to obtain the global multimodal feature.

3. The image denoising method according to claim 2, characterized in that, The first feature encoding unit is used to encode the noise image of the glue spray dots to obtain an image vector, including: The noise image of the glue spray point is processed into image blocks to obtain image blocks; The image vector is obtained by linearly projecting the image patch; Accordingly, the attribute information is encoded using the second feature encoding unit to obtain a text vector, including: The attribute information is segmented to obtain a discrete text sequence; The discrete text sequence is embedded and mapped to obtain discrete text vectors; The discrete text vector and the time step are fused to obtain the text vector.

4. The image denoising method according to claim 2, characterized in that, The attention mechanism unit is used to perform kernel mapping attention processing on the first multimodal intermediate features to obtain multimodal attention features, including: By applying linear transformations to the first multimodal intermediate features using different transformation parameters, key vectors, value vectors, and query vectors are obtained. The key vector is projected into the kernel space using a kernel mapping function to obtain a mapped key vector, and the query vector is projected into the kernel space using the kernel mapping function to obtain a mapped query vector. Kernel-normalized attention is calculated on the mapping key vector, the value vector, and the mapping query vector to obtain the multimodal attention features.

5. The image denoising method according to claim 2, characterized in that, The feature enhancement module includes a normalization unit, a second gating unit, and a feature enhancement unit. The feature enhancement module performs feature enhancement processing on the global multimodal features to obtain enhanced multimodal features of the glue-spraying point noise image, including: The normalization unit is used to perform layer normalization on the global multimodal features to obtain normalized multimodal features; The normalized multimodal features are subjected to gated residual modulation using the second gating unit to obtain the second multimodal intermediate features; The feature enhancement unit is used to perform nonlinear feature enhancement processing on the second multimodal intermediate feature to obtain enhanced intermediate features; The enhanced intermediate features are scaled using the text features to obtain the second multimodal scale-adjusted features; The enhanced multimodal features are obtained by performing a residual connection between the second multimodal scaling feature and the global multimodal feature.

6. The image denoising method according to claim 1, characterized in that, The noise prediction module includes a prediction unit and a splitting unit. The noise features include a noise feature map and a covariance feature map. The noise prediction module decodes the enhanced multimodal features to obtain the noise features of the glue spray point noise image, including: The prediction unit is used to perform a linear transformation on the enhanced multimodal features to obtain intermediate noise features; The intermediate noise features are split using the splitting unit to obtain the noise feature map and the covariance feature map.

7. The image denoising method according to claim 6, characterized in that, The image denoising module processes the noise features and the glue spray point noise image to obtain a denoised image of the glue spray point noise image, including: The noise feature map is used to denoise the noise image of the glue spray point to obtain an initial denoised image; The initial denoised image is corrected using the covariance feature map to obtain a denoised image.

8. An image denoising device, characterized in that, The device includes: The acquisition module is used to acquire the glue spray point noise image and the attribute information of the glue spray point noise image, and input the glue spray point noise image and the attribute information into a pre-trained image denoising model. The image denoising model includes a global association module, a feature enhancement module, a noise prediction module and an image denoising module. The feature extraction module is used to perform kernel mapping attention processing on the glue spray point noise image and the attribute information using the global association module to obtain the global multimodal features of the glue spray point noise image; The feature enhancement module is used to perform feature enhancement processing on the global multimodal features to obtain the enhanced multimodal features of the glue spray point noise image; The decoding module is used to decode the enhanced multimodal features using the noise prediction module to obtain the noise features of the glue spray point noise image; The denoising module is used to process the noise features and the glue spray point noise image using the image denoising module to obtain a denoised image of the glue spray point noise image.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the image denoising method according to any one of claims 1-7.

10. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the image denoising method according to any one of claims 1-7.

Images