Image restoration method, device, equipment and storage medium

Abstract

The application relates to the technical field of industrial intelligent detection, and discloses an image restoration method, device, equipment and storage medium. According to the dynamic gradient empowerment perception module, the channel slices of each neighborhood pixel can be dynamically obtained according to the gradient values of the center pixel and the neighborhood pixels, and the context features of the center pixel can be spliced and generated. In this way, the information aggregation strategy can be adaptively adjusted according to the local feature gradient, so that the detail retention of the clear area and the accurate enhancement of the blurred area can be realized. The dynamic gradient empowerment perception module is combined with a linear bidirectional weighted key value calculation mode with a global receptive field, so that global context modeling can be realized, the calculation complexity can be reduced, the detail features accurately reserved in the clear area can cross the whole image with the efficiency of linear calculation to guide and correct the feature expression of the blurred area, so that a deep feature map with strong inhibition ability for out-of-focus blur can be extracted, and accurate restoration of the out-of-focus blurred image can be realized.

Description

Technical Field

[0001] This application relates to the fields of computer vision and industrial intelligent inspection technology, and in particular to an image restoration method, apparatus, device and storage medium. Background Technology

[0002] In high-speed industrial production lines such as packaging and printing, automated optical inspection based on high-resolution images is a core technology for ensuring quality. However, industrial cameras are constrained by optical physical limitations such as field curvature when imaging over a wide area, often resulting in out-of-focus blurring in the output image, where the center is sharp but the edges are blurred. This non-uniform image degradation can obscure minute defects in the edge areas of the product, such as printing flaws and scratches, significantly reducing the accuracy and reliability of automated inspection algorithms and posing a fundamental technical challenge to product quality control.

[0003] Existing technologies have limitations in addressing this problem: traditional convolutional networks, due to their limited receptive field, struggle to handle large-scale, spatially varying blur. While visual Transformers possess global modeling capabilities, their quadratic computational complexity cannot meet the stringent efficiency requirements of high frame rates and high resolutions in industrial online inspection. In recent years, linear complexity models, such as RWKV (Receptance Weighted Key Value), have shown great potential. However, the general architecture of these models lacks a specialized perception mechanism for local image sharpness gradients, and their robustness and restoration accuracy still need improvement when processing transitional regions from sharp to blurry.

[0004] Therefore, how to accurately restore images with out-of-focus blur is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides an image restoration method, apparatus, device, and storage medium for accurate restoration of images exhibiting defocus blur.

[0006] In a first aspect, this application provides an image restoration method, comprising: Obtain the out-of-focus blurred image to be restored; The out-of-focus blurred image is input into the encoder as a feature map, processed by each encoder layer stacked in sequence, and the feature map processed by the last encoder layer is used as the restored depth feature map. In this process, each encoder layer obtains channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel in the input feature map. The channel slices of each neighboring pixel are concatenated to generate context features. The context features are encoded to obtain an admission gating vector and a key vector. The key vector is used to calculate global interaction features through bidirectional weighted key values. The processed feature map is obtained based on the admission gating vector and global interaction features. The gradient value of a neighboring pixel is proportional to the number of channel slices. The depth feature map is input into the decoder to obtain the restored image of the out-of-focus blurred image.

[0007] Optionally, based on the gradient values ​​of the center pixel and its neighboring pixels in the input feature map, channel slices of each neighboring pixel are obtained, including: Each pixel in the input feature map is used as the center pixel, and the number of channels is assigned to each neighboring pixel based on the gradient value between each center pixel and each neighboring pixel. Based on the number of channels allocated to each neighboring pixel, obtain the channel slice of each neighboring pixel.

[0008] Optionally, the number of channels is allocated to each neighboring pixel based on the gradient value between each center pixel and each neighboring pixel, including: Calculate the gradient value between each center pixel and its corresponding neighboring pixels; The weight value of each neighboring pixel is determined based on the gradient value of each center pixel and the weight value of each neighboring pixel. Based on the weight values ​​of each neighboring pixel and the total number of channels in the feature map, a corresponding number of channels are assigned to each neighboring pixel; the weight value of each neighboring pixel is inversely proportional to the number of channels assigned.

[0009] Optionally, the gradient values ​​of each center pixel and its corresponding neighboring pixels are calculated, including: Calculate the feature value of each center pixel and the feature difference between it and the feature values ​​of its corresponding neighboring pixels; The feature difference between each center pixel and its corresponding neighboring pixels is used as the gradient value between each center pixel and its corresponding neighboring pixels.

[0010] Optionally, the weight value of each neighboring pixel is determined based on the gradient values ​​of each neighboring pixel of each center pixel, including: The weight value of each neighboring pixel is calculated using the gradient values ​​of each center pixel and the inverse proportional calculation function; wherein the inverse proportional calculation function includes a smoothing factor.

[0011] Optionally, based on the number of channels allocated to each neighboring pixel, a channel slice for each neighboring pixel is obtained, including: According to a predetermined order, obtain channel slices corresponding to the number of channels of each neighboring pixel from the neighborhood features of each neighboring pixel; Accordingly, the channel slices of each neighboring pixel are concatenated to generate contextual features, including: The channels of each neighboring pixel of each center pixel are sliced ​​and spliced ​​in the predetermined order to obtain the context features of each center pixel.

[0012] Optionally, the context features are encoded to obtain an admission gating vector and a key-value vector. The key-value vector is then used to calculate the global interaction features through bidirectional weighted key-value calculation. The processed feature map obtained based on the admission gating vector and the global interaction features includes: The contextual features of each pixel are encoded into an admission gating vector, a key vector, and a value vector; Bidirectional weighted key-value calculation is performed on the key vector and value vector of each pixel to obtain the global interaction features of each pixel; Based on the admission gating vector of each pixel and the global interaction features, a processed feature map is generated. The processed feature map is then used as the input feature map for the next encoder layer and processed until the restored depth feature map is obtained.

[0013] Secondly, this application provides an image restoration apparatus, comprising: The acquisition module is used to acquire the out-of-focus blurred image to be restored; The first processing module takes the defocused and blurred image as a feature map and inputs it into the encoder. The encoder processes the image through stacked encoder layers, and the feature map processed by the last encoder layer is used as the restored depth feature map. Each encoder layer obtains channel slices of neighboring pixels based on the gradient values ​​of the center pixel and its neighboring pixels in the input feature map. These channel slices are then concatenated to generate context features. The context features are encoded to obtain an acceptance gate vector and a key vector. The key vector is then used to calculate global interaction features through bidirectional weighted key calculation. The processed feature map is obtained based on the acceptance gate vector and the global interaction features. The gradient value of a neighboring pixel is proportional to the number of channel slices. The second processing module is used to input the depth feature map into the decoder to obtain the restored image of the out-of-focus blurred image.

[0014] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the above-described image restoration method when executing the computer program.

[0015] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described image restoration method.

[0016] Compared with the prior art, the technical solutions provided in this application have the following advantages: This application provides an image restoration method, apparatus, device, and storage medium. This application acquires a defocused blurred image to be restored; inputs the defocused blurred image as a feature map into an encoder, processes it through sequentially stacked encoder layers, and uses the feature map processed by the last encoder layer as the restored depth feature map; wherein, each encoder layer includes a dynamic gradient weighting sensing module, which obtains channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel in the input feature map, the gradient value of the neighboring pixel being proportional to the number of channel slices; concatenates the channel slices of each neighboring pixel to generate context features, encodes the context features to obtain an acceptance gate vector and a key value vector, calculates the global interaction features using bidirectional weighted key values ​​from the key value vector, and obtains the processed feature map based on the acceptance gate vector and the global interaction features; inputs the depth feature map into a decoder to obtain the restored image of the defocused blurred image.

[0017] As can be seen, this application, through a dynamic gradient weighting receptive module, can dynamically obtain channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel, and stitch them together to generate the contextual features of the center pixel. In this way, the information aggregation strategy can be adaptively adjusted according to the local feature gradient, thereby achieving the preservation of details in the clear area and the precise enhancement of the blurred area, improving the restoration accuracy. In the dynamic gradient weighting receptive module, this application combines a linear bidirectional weighted key value calculation method with a global receptive field to achieve global context modeling and reduce computational complexity. This method allows the detailed features that are precisely preserved in the clear area to be used to guide and correct the feature expression of the blurred area across the entire image with linear computation efficiency, thereby extracting a deep feature map with strong suppression ability for defocus blur, laying a solid foundation for the final decoder to reconstruct a globally clear restored image, and achieving accurate restoration of defocus blurred images. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0021] Figure 1 This is a schematic flowchart of an image restoration method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the overall architecture provided for an embodiment of this application; Figure 3 A schematic diagram of the dynamic gradient weighting sensing module structure provided in the embodiments of this application; Figure 4 This is a schematic diagram of an image restoration device provided in an embodiment of this application; Figure 5 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0022] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0023] It should be noted that, in the optional embodiments of this application, the data related to object information, when applied to specific products or technologies, requires the permission or consent of the object. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this application involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0026] This application addresses the problem of insufficient restoration accuracy in existing high-efficiency vision models when handling industrial defocus blur tasks due to a lack of dynamic perception and adaptive processing capabilities for the degree of blur. To resolve this issue, an image restoration method, apparatus, device, and storage medium are proposed. This solution is an image restoration method for high-resolution imaging, applicable to online quality inspection systems for printed patterns on cans, boxes, etc., in the packaging industry. It solves the problem of edge field-of-view defocus blur caused by optical aberrations; that is, the restored image obtained through this solution can be input into the online quality inspection system for quality inspection.

[0027] This application aims to propose a spatial information fusion paradigm that can proactively perceive and utilize local image sharpness differences. A dynamic gradient-weighted sensing module is constructed, enabling it to adaptively adjust its information aggregation strategy by analyzing local feature gradients, thereby achieving detail preservation in sharp areas and precise enhancement in blurry areas. This application also provides a mechanism for guiding blurry area restoration by sharp areas. By combining the aforementioned dynamic sensing module with a linear attention model with a global receptive field, it efficiently uses the sharp texture information of the image center region as prior knowledge, transmitting it to guide the detail reconstruction of blurry edge areas. This application further provides a defocus blur suppression solution that combines high precision and high efficiency. Integrating the above technologies, an end-to-end network is provided specifically designed for industrial quality inspection scenarios such as the packaging industry, to meet the triple requirements of high definition, high efficiency, and high stability for industrial online inspection systems. The following embodiments further illustrate this solution: See Figure 1 This is a schematic diagram of an image restoration method provided in an embodiment of this application, such as... Figure 1 As shown, the method specifically includes the following steps: S101. Obtain the out-of-focus blurred image to be restored; In this application, the defocused blurred image can be an image acquired by an industrial camera in a high-speed industrial production line such as packaging and printing. When the image is wide-area imaging, it is limited by optical physical constraints such as field curvature, resulting in a defocused blurring phenomenon in the output image, where the center is clear and the edges are blurred. Therefore, this application refers to images with this phenomenon as defocused blurred images.

[0028] S102. Input the out-of-focus blurred image as a feature map into the encoder, process it through each encoder layer stacked in sequence, and use the feature map processed by the last encoder layer as the restored depth feature map.

[0029] In this process, each encoder layer obtains channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel in the input feature map. The channel slices of each neighboring pixel are concatenated to generate context features. The context features are encoded to obtain an admission gating vector and a key value vector. The key value vector is used to calculate the global interaction features through bidirectional weighted key value. The processed feature map is obtained based on the admission gating vector and the global interaction features. The gradient value of the neighboring pixel is proportional to the number of channel slices.

[0030] S103. Input the depth feature map into the decoder to obtain the restored image of the out-of-focus blurred image.

[0031] This application provides a defocus blur suppression scheme based on an encoder-decoder architecture. The scheme innovatively introduces a dynamic gradient weighting sensing module into the encoder, which serves as the core spatial information processing unit of the encoder. The decoder consists of conventional convolutional layers and upsampling layers, used to reconstruct the restored depth feature map into a high-resolution and clear restored image.

[0032] See Figure 2 This is a schematic diagram of the overall architecture provided in the embodiments of this application, such as... Figure 2 As shown, the overall architecture of this application mainly includes an encoder and a decoder. The encoder comprises multiple encoder layers stacked sequentially, each encoder layer including a dynamic gradient weighting receptive module, where ×M represents M consecutively connected blocks of the dynamic gradient weighting receptive module. The decoder includes k×k convolutional layers and upsampling layers, where ×N represents N consecutively connected blocks of convolutional layers with a kernel size of k×k. The upsampling layer can be linear interpolation upsampling or deconvolution upsampling. The encoder takes an image with defocus blur as input and outputs a deblurred restored image.

[0033] In this application, the dynamic gradient weighting perception module, within an efficient linear attention model (such as Vision-RWKV (Vision-Receptance Weighted Key-Value)) framework, endows it with the ability to perceive and utilize local gradient information, thereby achieving adaptive processing of defocus blur. It transforms the neighborhood weight from a static, preset parameter into a dynamic variable that depends on real-time input. This enables the module to perceive local image sharpness, thus achieving adaptive processing.

[0034] Specifically, each dynamic gradient weighting sensing module processes the input feature map in the same way. The feature map input to the dynamic gradient weighting sensing module of the first encoder layer is the defocused and blurred image to be restored. Its output feature map is a powerful feature map that contains both fine local perception and global sharpness prior, which is then used as the input to the dynamic gradient weighting sensing module of the second encoder layer for further processing, and so on. After processing by the dynamic gradient weighting sensing module of the last encoder layer (the Mth one), a depth feature map is output. By stacking multiple encoder layers containing dynamic gradient weighting sensing modules, this application can efficiently extract a depth feature map with strong suppression capability for defocus blur, laying a solid foundation for the final decoder to reconstruct a globally sharp restored image.

[0035] When processing the input feature map, the dynamic gradient weighting receptive module of each encoder layer first needs to obtain the channel slices of each neighboring pixel based on the gradient values ​​of each center pixel and its neighboring pixels in the feature map. In this application, the receptive field of each pixel is first defined as its four spatially adjacent pixels, that is, the neighboring pixels of each center pixel are the pixels adjacent to the center pixel in the up, down, left, and right directions, to ensure computational efficiency. This gradient value can reflect the feature differences between adjacent pixels. In sharp areas such as the center of the image, the feature differences between adjacent pixels are usually large, that is, the gradient values ​​are large and the details are clear; while in blurry areas such as the edges, the feature differences between adjacent pixels are small, that is, the gradient values ​​are small and the details are blurred.

[0036] This application dynamically obtains channel slices of each neighboring pixel of the center pixel based on the gradient values ​​of the center pixel and each neighboring pixel. The gradient value of the neighboring pixel is proportional to the number of channel slices. Therefore, in out-of-focus blurred areas, the feature difference between neighboring pixels and the center pixel is small, the gradient value is small, and the number of channel slices is small. This is because these neighboring pixels provide low-information redundant information, and only a few channels are needed to summarize their features, thus avoiding the waste of computational resources. In contrast, in sharp image areas, the feature difference between neighboring pixels and the center pixel is large, the gradient value is large, and the number of channel slices is large. This is because neighboring pixels carry high-information key details, requiring more channels for fine-tuning and representation.

[0037] In another embodiment of this application, obtaining a channel slice of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel in the input feature map includes: taking each pixel in the input feature map as the center pixel, allocating a number of channels to each neighboring pixel based on the gradient values ​​of each center pixel and each neighboring pixel; and obtaining a channel slice of each neighboring pixel based on the number of channels allocated to each neighboring pixel.

[0038] Specifically, when obtaining channel slices, this application determines the number of channels allocated to each neighboring pixel based on the gradient value, and obtains gradient slices based on the allocated number of channels; the larger the gradient value, the more channels are allocated, and the smaller the gradient value, the fewer channels are allocated. This channel slice refers to a sub-feature map obtained from a portion of the channels in the feature map of the neighboring pixels, based on the allocated number of channels. For example, if the number of channels allocated to a neighboring pixel is 10, then 10 channels of features need to be obtained from the feature map of that neighboring pixel to generate a channel slice for that neighboring pixel. After obtaining the channel slices of each neighboring pixel, they need to be concatenated in the acquisition order to generate context features. The context features are encoded to obtain an acceptance gate vector and a key value vector. The key value vector is then used to calculate the global interaction features through bidirectional weighted key value calculation. The processed feature map is obtained based on the acceptance gate vector and the global interaction features.

[0039] Specifically, this application implements bidirectional weighted key value calculation through the Bi-WKV (Bidirectional Receptance Weighted Key-Value) module. The Bi-WKV module is a bidirectional global information mixer designed to adapt to the symmetry of images. It can achieve efficient interaction of complete pixel sequences with linear computational complexity, while overcoming the limitations of traditional unidirectional models. This step is crucial, as it allows detailed features precisely preserved in sharp regions to guide and correct feature representations in blurred regions across the entire image with linear computational efficiency. Finally, the global interaction features output after global interaction are fused with the acceptance gating vector to complete the calculation of the entire dynamic gradient weighting receptive module. A powerful feature map containing both fine local perception and global sharpness priors is then passed to the next encoder layer of the network for further processing until the last encoder layer outputs a depth feature map. This depth feature map is then input into the decoder, which reconstructs a high-resolution, sharp restored image from the restored depth feature map.

[0040] In summary, this application, through a dynamic gradient weighting receptive module, can dynamically obtain channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel, and then stitch them together to generate the contextual features of the center pixel. This method allows for adaptive adjustment of the information aggregation strategy based on local feature gradients, thereby achieving detail preservation in sharp areas and precise enhancement of blurry areas, improving restoration accuracy. Furthermore, by combining the dynamic gradient weighting receptive module with a linear bidirectional weighted key-value calculation method with a global receptive field, global context modeling can be achieved, reducing computational complexity. This method allows the detailed features precisely preserved in sharp areas to be used linearly across the entire image to guide and correct the feature representation of blurry areas, thereby extracting a deep feature map with strong suppression capabilities against defocus blur. This lays a solid foundation for the final decoder to reconstruct a globally sharp restored image, achieving accurate restoration of defocus blurred images.

[0041] In another embodiment of this application, the number of channels is allocated to each neighboring pixel based on the gradient value between each center pixel and each neighboring pixel, including: Calculate the gradient value between each center pixel and its corresponding neighboring pixels; The weight value of each neighboring pixel is determined based on the gradient value of each center pixel and the weight value of each neighboring pixel. Based on the weight values ​​of each neighboring pixel and the total number of channels in the feature map, a corresponding number of channels are assigned to each neighboring pixel; the weight value of each neighboring pixel is inversely proportional to the number of channels assigned.

[0042] In this application, when allocating the number of channels to each neighboring pixel, it is first necessary to perceive the local sharpness gradient. That is, after taking each pixel in the input feature map as the center pixel, the gradient value between each center pixel and its corresponding neighboring pixels is calculated. Specifically, when calculating the gradient value between each center pixel and its corresponding neighboring pixels, the feature value of each center pixel and the feature value of its corresponding neighboring pixels can be calculated. The feature difference between each center pixel and its corresponding neighboring pixels is then used as the gradient value between each center pixel and its corresponding neighboring pixels.

[0043] Specifically, this application can calculate the feature difference between the feature value of the center pixel and the feature values ​​of its four neighboring pixels. This feature difference directly reflects the local sharpness gradient in a physical sense, and therefore can be directly used as the gradient value. In this embodiment, the gradient value of each center pixel includes four gradient values ​​of its four neighboring pixels in four directions: the gradient value between the center pixel and the adjacent pixels in the upward direction, the gradient value between the center pixel and the adjacent pixels in the downward direction, the gradient value between the center pixel and the adjacent pixels in the left direction, and the gradient value between the center pixel and the adjacent pixels in the right direction. Furthermore, the feature map input to the encoder layer is a defocused blurred image, so the feature difference between the feature value of the center pixel and the feature values ​​of its four neighboring pixels is specifically the pixel difference between the pixel value of the center pixel and the pixel values ​​of its four neighboring pixels.

[0044] After determining the gradient values ​​of each neighboring pixel of each center pixel, this application needs to dynamically generate the weight values ​​of each neighboring pixel based on the gradient values. The generation rule is that the weight value is inversely proportional to the gradient value. That is, the greater the feature difference between a neighboring pixel and the center pixel (higher gradient value), the smaller its weight value; conversely, the smaller the feature difference (lower gradient value), the larger its weight value. To ensure computational stability, this application specifically uses the gradient values ​​of each neighboring pixel of each center pixel and an inverse proportionality calculation function to calculate the weight values ​​of each neighboring pixel. The inverse proportionality calculation function includes a smoothing factor. This application adds a very small smoothing factor to the inverse proportionality calculation function during implementation. For example, the inverse proportionality calculation function can be: ; Where w is the weight value and d is the gradient value. The smoothing factor can specifically be... or The reason for introducing a smoothing factor in this application is that in actual industrial quality inspection scenarios, such as the surface of aluminum cans or packaging boxes, there are often large areas of solid-color backgrounds or smooth regions, where the feature difference between adjacent pixels in these regions tends to be close to zero. Without processing, this would lead to infinitely large weights, causing model training to crash or inference to fail. Therefore, introducing a smoothing factor into the inverse proportional calculation function in this application ensures the stability of the calculation.

[0045] Since the gradient value of each center pixel includes the gradient values ​​of the four neighboring pixels in four directions, the weight values ​​calculated in this application also include the weight values ​​of the four neighboring pixels in four directions. That is, the weight values ​​between the center pixel and the adjacent pixels in the upward direction, the weight values ​​between the center pixel and the adjacent pixels in the downward direction, the weight values ​​between the center pixel and the adjacent pixels in the left direction, and the weight values ​​between the center pixel and the adjacent pixels in the right direction. The gradient values ​​and weight values ​​in these four directions remain independent. This method can endow the network with the ability to perceive anisotropy.

[0046] The dynamic weight generation method for neighborhood pixels provided in this application is deeply integrated with the task of defocus blur suppression: in blurred areas, low gradients lead to high weights for the neighborhood, so the module will strongly aggregate the surrounding information, which plays a role in smoothing to eliminate artifacts and provide a stable base for subsequent restoration; in sharp areas, high gradients lead to low weights for the neighborhood, so the module will suppress the over-integration of neighborhood information, thereby protecting the original and precious image details from being destroyed to the greatest extent.

[0047] After determining the weight values ​​of neighborhood pixels through the above process, this application allocates a corresponding number of channels to each neighboring pixel based on its weight value and the total number of channels in the feature map. Specifically, this application does not use a simple feature-weighted average method, but introduces a mechanism for dynamically reorganizing channel resources based on local sharpness. First, based on the dynamic weights generated in the previous step, this application allocates processing channels to the information flow from four neighborhood directions (up, down, left, and right). The core principle of allocation is: the higher the similarity between the neighborhood and the center (i.e., the larger the weight value), the fewer channels are allocated; conversely, the lower the similarity between the neighborhood and the center (i.e., the smaller the weight value), the more channels are allocated. In other words, the weight value of each neighboring pixel is inversely proportional to the number of channels allocated. The reason this application adopts this allocation method is that in out-of-focus blurred areas, the feature difference between the neighboring and center pixels is small, and the weight value is large. In this case, the module allocates fewer channels because these neighborhoods provide low-information redundant information, and only a few channels are needed to summarize their features, thus avoiding waste of computational resources. In areas of sharp image, the feature difference between the neighboring pixels and the center pixel is large (such as at edges or on fine textures), and the weight value is small. In this case, the module actually allocates more channels to them because these neighborhoods carry key details with high information content, requiring a more powerful channel capacity (more channels) for fine processing and representation.

[0048] In another embodiment of this application, obtaining channel slices of each neighboring pixel according to the number of channels allocated to each neighboring pixel includes: obtaining channel slices corresponding to the number of channels of each neighboring pixel from the neighborhood features of each neighboring pixel in a predetermined order; correspondingly, splicing the channel slices of each neighboring pixel to generate context features includes: splicing the channel slices of each neighboring pixel of each center pixel in the predetermined order to obtain the context features of each center pixel.

[0049] Specifically, the shape of the input feature map is typically (B, C, H, W), where B represents the batch size, C represents the number of channels, and H and W represent the height and width, respectively. Each pixel at a spatial location (x, y) on the feature map is used as a center pixel, and the system will then... For each of the W center pixels, the gradient value between the center pixel and its four neighboring pixels is calculated sequentially (in a highly parallel manner on a GPU). For each center pixel, the channel allocation ratio in the four directions is dynamically calculated independently to determine the number of channels in the four neighboring pixels. The total number of channels in the feature map can be 16, 32, 64, 128, 256, etc. It should be noted that the center pixel type includes edge pixels and non-edge pixels. Edge pixels are pixels located on the four edge lines and four corners of the feature map. Center pixels on the four edge lines have three neighboring pixels, and center pixels at the four corners have two neighboring pixels. Non-edge pixels are all pixels in the feature map other than edge pixels, and non-edge pixels have four neighboring pixels. For non-edge pixels, this application can obtain channel slices corresponding to the number of channels in each of the four neighboring pixels from their neighborhood features in a predetermined order. The channel slices of the four neighboring pixels of the center pixel are then concatenated in a predetermined order to obtain the context features of the center pixel.

[0050] This application addresses edge pixels and allows for calculation in various ways. For example, edge pixels may not be included in the calculation, and their original pixel values ​​can be directly used as the pixel values ​​of the corresponding edge pixels in the restored image. Alternatively, missing neighboring pixels can be ignored, and other neighboring pixels that are not missing can be used directly. For instance, if a center pixel has three neighboring pixels, then channel slices corresponding to the number of channels of each neighboring pixel are obtained from the neighborhood features of its three neighboring pixels in a predetermined order. The channel slices of the three neighboring pixels of the center pixel are then spliced ​​together in a predetermined order to obtain the context features of the center pixel.

[0051] Here, we will only take an example with a total input channel count of Ctotal=64 to illustrate the process of obtaining channel fragments and splicing to generate contextual features: After dynamic calculation, the number of channels allocated in the four directions of up, down, left, and right are as follows: =10, =15, =20, =19.

[0052] When acquiring channel slices, they are acquired in a predetermined order. The feature data of channels 1 to 10 are acquired from the feature map of the neighboring pixels in the upper direction as channel slices. Then, the feature data of channels 11 to 25 (15 channels in total) are acquired from the feature map of the neighboring pixels in the lower direction as channel slices. Then, the feature data of channels 26 to 45 (20 channels in total) are acquired from the feature map of the neighboring pixels in the left direction as channel slices. Finally, the feature data of channels 46 to 64 (19 channels in total) are acquired from the feature map of the neighboring pixels in the right direction as channel slices. After obtaining the channel slices of each neighboring pixel, the channel slices need to be spliced ​​in a predetermined order, that is, spliced ​​in the order of upper, lower, left, and right to generate the context features of each center pixel.

[0053] In summary, this application determines the number of channels for each center pixel by dividing the input feature map according to a ratio inversely proportional to the weight values ​​of each neighboring pixel (i.e., directly proportional to the sharpness gradient value). Then, a corresponding number of channel slices are extracted from each neighboring feature map, and these channel slices from different directions are reassembled to form a new gradient-guided contextual feature that has undergone dynamic channel reorganization. Because this feature injects local sharpness priors into the feature representation through dynamic channel reorganization, it enables efficient and accurate spatial adaptive suppression of defocus blur.

[0054] In another embodiment of this application, the context features are encoded to obtain an admission gating vector and a key-value vector. The key-value vector is then used to calculate global interaction features through bidirectional weighted key-value calculation. The processed feature map obtained based on the admission gating vector and the global interaction features includes: The contextual features of each pixel are encoded into an admission gating vector, a key vector, and a value vector. The key vector and value vector of each pixel are bidirectionally weighted to calculate the global interaction features of each pixel. Based on the admission gating vector and global interaction features of each pixel, a processed feature map is generated. The processed feature map is then used as the input feature map for the next dynamic gradient weighting perception module for further processing until the restored depth feature map is obtained.

[0055] In this application, the contextual features enhanced by dynamic gradient field guidance in the above embodiments will be encoded in three fully connected layers. See also Figure 3 This is a schematic diagram of the dynamic gradient weighting sensing module structure provided in an embodiment of this application, as shown below. Figure 3As shown, the input features are first obtained by gradient perception to obtain the gradient value of each pixel, and then dynamic weight generation is performed. Based on the perceived sharpness gradient value, a dynamic weight value is generated for each neighboring pixel. Through neighborhood information weighting and global guidance, the number of channels is allocated based on the weight value of each neighboring pixel, and channel slices are obtained according to the number of channels and then concatenated to generate context features. This context feature is input to fully connected layer 1, fully connected layer 2, and fully connected layer 3 respectively. The context feature entering fully connected layer 1 is encoded into a receptance gate vector and output by combining non-linear calculation (such as the ReLU function). This receptance gate vector is used to control how much global interaction feature after global interaction is accepted by the current pixel. It is obtained through the fully connected layer and its value is made to be in the range of [0,1] and similar. In the blurred region (small gradient value), the number of channels in the context feature is small but the weight is high. The receptance gate vector may tend to be larger to allow more global information to enter. In the sharp region (large gradient), the receptance gate vector may be smaller to protect the original details of the cleaned region.

[0056] In this application, the context features entering fully connected layer 2 and fully connected layer 3 are encoded into key vectors and value vectors, respectively. The key vector identifies the feature label of the current pixel; the value vector represents the actual feature content of the current pixel. The value vector in clear areas is rich in detail, while the value vector in blurred areas is smoother. The key-value vector and the value vector need to be processed by the Bi-WKV module. Specifically, after obtaining the context features of each pixel, the Bi-WKV module transforms the independent context features of each pixel into global interactive features that fuse information from the entire image, based on the key vector and the value vector. The calculation formula of the Bi-WKV module is shown below, taking the calculation of the final output of the t-th pixel as an example: ; in, For pixel index, Represents the global interaction feature of the t-th pixel; The key vector is obtained by linearly transforming the input context features of the t-th pixel through a fully connected layer _2. The value vector is obtained by linearly transforming the input context features of the t-th pixel through a fully connected layer _3. This is the position offset vector; This is the forward molecular state vector; This is the backward molecular state vector; The forward denominator is the state vector; The backward denominator is the state vector; It is a natural constant, approximately 2.718.

[0057] The aforementioned position bias vector is determined during the training process. The Bi-WKV module initializes the value of the position bias vector at the very beginning of training, and continuously optimizes it as training progresses until the final value is determined at the end of training. The forward numerator state vector, backward numerator state vector, forward denominator state vector, and backward denominator state vector are continuously updated during model inference. This application obtains these updated vectors according to the following formula: Current state vector = (previous state vector × forget coefficient) + (features of the current pixel); Specifically, if the current state vector is the forward numerator state vector of the current pixel, then the corresponding previous state vector is the previous forward numerator state vector of the current pixel; if the current state vector is the backward numerator state vector of the current pixel, then the corresponding previous state vector is the previous backward numerator state vector of the current pixel; if the current state vector is the forward denominator state vector of the current pixel, then the corresponding previous state vector is the previous forward denominator state vector of the current pixel; if the current state vector is the backward denominator state vector of the current pixel, then the corresponding previous state vector is the previous backward denominator state vector of the current pixel; the forgetting coefficient ranges from 0 to 1, and its specific value can be customized according to the actual situation.

[0058] As can be seen from the above embodiments, this application possesses accurate and spatially adaptive restoration capabilities. Specifically, through its unique dynamic gradient weighting sensing module, this application can perceive the local sharpness of an image in real time, enabling differentiated processing of different regions. It can finely enhance blurred areas while perfectly protecting sharp areas, resulting in image fidelity and sharpness far exceeding traditional one-size-fits-all methods. This application also exhibits excellent robustness in industrial environments. Specifically, this application does not rely on a fixed blur pattern; its dynamic and content-adaptive characteristics enable it to effectively handle various degrees and forms of defocus blur caused by different batches of products, different production line locations, and different lighting environments, ensuring the stability and reliability of the industrial quality inspection system operating around the clock. Furthermore, this application also boasts leading-edge high-efficiency computing performance. Specifically, the overall architecture of this application is based on an advanced linear complexity model, and its core modules are also extremely efficient in computation, fundamentally solving the efficiency bottleneck of traditional high-performance models when processing high-resolution, high-frame-rate industrial images. This allows it to be seamlessly integrated into online quality inspection processes with stringent real-time requirements, possessing extremely high industrial deployment value.

[0059] The image restoration apparatus provided in the embodiments of this application is described below. The image restoration apparatus described below can be referred to in correspondence with the restoration method described above.

[0060] See Figure 4 , Figure 4This application provides a schematic diagram of an image restoration device, which specifically includes: Acquisition module 11 is used to acquire the out-of-focus blurred image to be restored; The first processing module 12 is used to input the defocused blurred image as a feature map into the encoder, process it through the encoder layers stacked sequentially, and use the feature map processed by the last encoder layer as the restored depth feature map. Each encoder layer obtains channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and its neighboring pixels in the input feature map. These channel slices are then concatenated to generate context features. The context features are encoded to obtain an acceptance gate vector and a key vector. The key vector is then used to calculate global interaction features through bidirectional weighted key calculation. The processed feature map is obtained based on the acceptance gate vector and the global interaction features. The gradient value of a neighboring pixel is proportional to the number of channel slices. The second processing module 13 is used to input the depth feature map into the decoder to obtain the restored image of the out-of-focus blurred image.

[0061] As an optional embodiment, the first processing module includes: The channel number determination unit is used to treat each pixel in the input feature map as a center pixel and allocate the number of channels to each neighboring pixel based on the gradient value between each center pixel and each neighboring pixel. The channel slice acquisition unit is used to acquire the channel slice of each neighboring pixel based on the number of channels allocated to each neighboring pixel.

[0062] As an optional embodiment, the channel number determination unit includes: The gradient value calculation subunit is used to calculate the gradient value between each center pixel and its corresponding neighboring pixels; The weight calculation subunit is used to determine the weight value of each neighboring pixel based on the gradient values ​​of each neighboring pixel of each center pixel; the gradient value of each neighboring pixel is inversely proportional to the weight value. The channel allocation subunit is used to allocate a corresponding number of channels to each neighboring pixel based on the weight value of each neighboring pixel and the total number of channels in the feature map; the weight value of each neighboring pixel is inversely proportional to the number of channels allocated.

[0063] As an optional embodiment, the gradient value calculation subunit is specifically used to: calculate the feature value of each center pixel and the feature difference between the feature values ​​of the corresponding neighboring pixels; and use the feature difference between each center pixel and the corresponding neighboring pixels as the gradient value between each center pixel and the corresponding neighboring pixels.

[0064] As an optional embodiment, the weight value calculation subunit is specifically used to: calculate the weight value of each neighboring pixel using the gradient values ​​of each neighboring pixel of each center pixel and the inverse proportional calculation function; wherein the inverse proportional calculation function includes a smoothing factor.

[0065] As an optional embodiment, the channel slice acquisition unit is specifically used to: acquire channel slices corresponding to the number of channels of each neighboring pixel from the neighborhood features of each neighboring pixel in a predetermined order; correspondingly, to splice the channel slices of each neighboring pixel to generate context features, including: splicing the channel slices of each neighboring pixel of each center pixel in the predetermined order to obtain the context features of each center pixel.

[0066] As an optional embodiment, the first processing module includes: The encoding unit is used to encode the contextual features of each pixel into an admission gating vector, a key vector, and a value vector; The computation unit is used to perform bidirectional weighted key-value calculation on the key vector and value vector of each pixel to obtain the global interaction features of each pixel; The processing unit is used to generate a processed feature map based on the acceptance gate vector of each pixel and the global interaction features. The processed feature map is then used as the input feature map of the next encoder layer for further processing until the restored depth feature map is obtained.

[0067] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, it includes: Memory 20 is used to store computer programs; The processor 21 is configured to implement the steps of the image restoration method as described in the above embodiments when executing a computer program.

[0068] The electronic devices provided in this embodiment may include, but are not limited to, smartphones, tablets, laptops, or desktop computers.

[0069] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0070] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the image restoration method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc.

[0071] In some embodiments, the electronic device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0072] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0073] In another exemplary embodiment, a computer storage medium is also provided, wherein the program instructions, when executed by a processor, implement the steps of the image restoration method described in any of the above method embodiments.

[0074] It is understood that if the image restoration method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0075] The various embodiments described in this specification are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” used herein may also mean the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a specific order described or illustrated, unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0076] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0077] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image restoration method, characterized in that, include: Obtain the out-of-focus blurred image to be restored; The out-of-focus blurred image is input into the encoder as a feature map, processed by each encoder layer stacked in sequence, and the feature map processed by the last encoder layer is used as the restored depth feature map. In this process, each encoder layer obtains channel slices of each neighboring pixel based on the gradient values ​​of the center pixel and each neighboring pixel in the input feature map. These channel slices are then concatenated to generate context features. The context features are encoded to obtain an acceptance gate vector and a key vector. The key vector is then used to calculate global interaction features through bidirectional weighted key value calculation. Finally, the processed feature map is obtained based on the acceptance gate vector and the global interaction features. The gradient value of a neighboring pixel is proportional to the number of channel slices. The bidirectional weighted key value calculation is implemented using a Bi-WKV module. The depth feature map is input into the decoder to obtain the restored image of the out-of-focus blurred image.

2. The image restoration method according to claim 1, characterized in that, Based on the gradient values ​​of the center pixel and its neighboring pixels in the input feature map, channel slices of each neighboring pixel are obtained, including: Each pixel in the input feature map is used as the center pixel, and the number of channels is assigned to each neighboring pixel based on the gradient value between each center pixel and each neighboring pixel. Based on the number of channels allocated to each neighboring pixel, obtain the channel slice of each neighboring pixel.

3. The image restoration method according to claim 2, characterized in that, The number of channels is allocated to each neighboring pixel based on the gradient value between each center pixel and each neighboring pixel, including: Calculate the gradient value between each center pixel and its corresponding neighboring pixels; The weight value of each neighboring pixel is determined based on the gradient value of each center pixel and the weight value of each neighboring pixel. Based on the weight values ​​of each neighboring pixel and the total number of channels in the feature map, a corresponding number of channels are assigned to each neighboring pixel; the weight value of each neighboring pixel is inversely proportional to the number of channels assigned.

4. The image restoration method according to claim 3, characterized in that, Calculate the gradient values ​​of each center pixel and its corresponding neighboring pixels, including: Calculate the feature value of each center pixel and the feature difference between it and the feature values ​​of its corresponding neighboring pixels; The feature difference between each center pixel and its corresponding neighboring pixels is used as the gradient value between each center pixel and its corresponding neighboring pixels.

5. The image restoration method according to claim 3, characterized in that, Based on the gradient values ​​of each neighboring pixel of each center pixel, determine the weight value of each neighboring pixel, including: The weight value of each neighboring pixel is calculated using the gradient values ​​of each center pixel and the inverse proportional calculation function; wherein the inverse proportional calculation function includes a smoothing factor.

6. The image restoration method according to claim 2, characterized in that, Based on the number of channels allocated to each neighboring pixel, obtain the channel slices for each neighboring pixel, including: According to a predetermined order, obtain channel slices corresponding to the number of channels of each neighboring pixel from the neighborhood features of each neighboring pixel; Accordingly, the channel slices of each neighboring pixel are concatenated to generate contextual features, including: The channels of each neighboring pixel of each center pixel are sliced ​​and spliced ​​in the predetermined order to obtain the context features of each center pixel.

7. The image restoration method according to any one of claims 1 to 6, characterized in that, The context features are encoded to obtain the admission gating vector and the key-value vector. The key-value vector is then used to calculate the global interaction features. Based on the admission gating vector and the global interaction features, the processed feature map is obtained, including: The contextual features of each pixel are encoded into an admission gating vector, a key vector, and a value vector; Bidirectional weighted key-value calculation is performed on the key vector and value vector of each pixel to obtain the global interaction features of each pixel; Based on the admission gating vector of each pixel and the global interaction features, a processed feature map is generated. The processed feature map is then used as the input feature map for the next encoder layer and processed until the restored depth feature map is obtained.

8. An image restoration device, characterized in that, include: The acquisition module is used to acquire the out-of-focus blurred image to be restored; The first processing module takes the defocused blurred image as a feature map and inputs it into the encoder. The image is then processed by sequentially stacked encoder layers, and the feature map processed by the last encoder layer is used as the restored depth feature map. Each encoder layer obtains channel slices of neighboring pixels based on the gradient values ​​of the center pixel and its neighboring pixels in the input feature map. These channel slices are then concatenated to generate context features. The context features are encoded to obtain an acceptance gate vector and a key vector. The key vector is then used to calculate global interaction features through bidirectional weighted key value calculation. The processed feature map is obtained based on the acceptance gate vector and the global interaction features. The gradient values ​​of neighboring pixels are proportional to the number of channel slices. The bidirectional weighted key value calculation is implemented using a Bi-WKV module. The second processing module is used to input the depth feature map into the decoder to obtain the restored image of the out-of-focus blurred image.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the image restoration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image restoration method as described in any one of claims 1 to 7.

Images