Image processing method based on curve recovery and related device

Abstract

The application discloses a kind of based on curve recovery's image processing method and related equipment, the method includes: obtaining the foggy image to be handled;By preset depth curve estimation network, image recovery parameter is obtained based on foggy image;Based on image recovery parameter, foggy image is carried out multiple nonlinear curve mapping recovery, and preliminary haze-free image is obtained;By preset image iterative defogging mechanism, preliminary haze-free image is handled based on image recovery parameter, and each round of iteration output is obtained;When detecting that the haze-free image corresponding to iteration output meets the clarity requirement of pre-set, target haze-free image is obtained.The application can be removed by nonlinear curve mapping and recursive iteration strategy, and self-adapting restoration is realized to image haze layering, and simple and fast defogging of outdoor foggy image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, and in particular to an image processing method based on curve recovery and related equipment. BACKGROUND

[0002] Nowadays, the actual promotion and application of intelligent vision technology still faces many challenges, among which the restriction of environmental factors on system performance is particularly prominent. In the haze environment, the visual perception system is difficult to accurately capture the information of the real world, which directly leads to obstacles in computer vision tasks such as image recognition and target tracking.

[0003] The existing image defogging method based on the atmospheric scattering model has the problem of poor parameter adaptability. The atmospheric scattering model involves multiple complex physical processes, forming a highly ill-conditioned equation set. In this case, it is difficult to accurately estimate the atmospheric scattering parameters in various complex scenes, whether relying on prior assumptions or deep neural networks. The inaccurate estimation of parameters in the atmospheric scattering model often directly leads to unnecessary visual phenomena such as image artifacts and unbalanced contrast.

[0004] To solve this problem, some research works mainly explore from two directions. On the one hand, by improving the estimation algorithm of atmospheric scattering parameters, the feature extraction and representation ability of deep neural networks is enhanced to improve the estimation accuracy of model parameters. On the other hand, starting from model structure optimization, the construction of the atmospheric scattering model is adjusted to reduce cumulative errors or improve modeling ability. Although these studies have improved the defogging effect to some extent, due to the inherent ill-conditioned nature of the atmospheric scattering model, the parameter estimation still faces great challenges, and the instability of the solution has not been fundamentally solved, making the defogging performance still have certain limitations. SUMMARY

[0005] To solve the above problems, embodiments of the present application provide an image processing method and device based on curve recovery, an electronic device, a computer readable storage medium, and a computer program product.

[0006] In a first aspect, to solve the above technical problems, the present application provides an image processing method based on curve recovery, comprising:

[0007] Obtaining a foggy image to be processed;

[0008] Obtaining image recovery parameters based on the foggy image through a preset depth curve estimation network;

[0009] Performing multi-nonlinear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image;

[0010] The preliminary haze-free image is iteratively processed based on the image restoration parameter through a preset image iterative defogging mechanism to obtain an iteration output of each round;

[0011] When it is detected that the haze-free image corresponding to the iteration output meets a preset definition requirement, a target haze-free image is obtained.

[0012] Beneficial effects are:

[0013] In the technical scheme provided in the embodiments of the present application, a haze image to be processed is obtained; an image restoration parameter is obtained based on the haze image through a preset depth curve estimation network; a preliminary haze-free image is obtained by performing multiple non-linear curve mapping restoration on the haze image based on the image restoration parameter; the preliminary haze-free image is iteratively processed based on the image restoration parameter through a preset image iterative defogging mechanism to obtain an iteration output of each round; and when it is detected that the haze-free image corresponding to the iteration output meets a preset definition requirement, a target haze-free image is obtained. In this way, the present application realizes image haze layering removal and adaptive restoration through a non-linear curve mapping and recursive iteration strategy, realizes simple and fast haze removal of outdoor haze images, ensures accurate restoration of details and colors, has good cross-scene generalization ability and robustness, and solves the problem of existing methods that rely on accurate parameter estimation to cause artifacts and color differences in the haze-removed image.

[0014] Further, the image restoration parameter includes an inverse image restoration parameter and a haze-free image restoration parameter; and the multiple non-linear curve mapping restoration on the haze image based on the image restoration parameter to obtain the preliminary haze-free image includes:

[0015] Performing non-linear enhancement processing on the haze image based on the inverse image restoration parameter to obtain a processed image;

[0016] Performing curve mapping restoration on the processed image based on the haze-free image restoration parameter to obtain the preliminary haze-free image.

[0017] Further, the non-linear enhancement processing on the haze image based on the inverse image restoration parameter to obtain the processed image includes:

[0018] Performing pixel normalization processing on the haze image to obtain a normalized image, and pixel values of the normalized image are in a preset range;

[0019] Performing pixel inversion on the normalized image to obtain an inverse image;

[0020] Performing curve mapping processing on the inverse image based on the inverse image restoration parameter to obtain the processed image, so as to realize non-linear enhancement of image brightness.

[0021] Further, the curve mapping recovery of the processed image based on the haze-free image recovery parameter is performed to obtain a preliminary haze-free image, including:

[0022] The curve mapping recovery of the processed image based on the haze-free image recovery parameter is performed to obtain a recovered image;

[0023] The pixel inversion of the recovered image is performed to obtain a preliminary haze-free image.

[0024] Further, the image recovery parameter includes an inverse image recovery parameter and a haze-free image recovery parameter;

[0025] Through a preset image iterative defogging mechanism, the preliminary haze-free image is iteratively processed based on the image recovery parameter to obtain an iterative output of each round, including:

[0026] In each round of iteration, the preliminary haze-free image or the iterative output of the previous round is nonlinearly transformed based on the inverse image recovery parameter to generate an intermediate image through a preset image iterative defogging mechanism;

[0027] The intermediate image is subjected to curve mapping recovery and pixel inversion based on the haze-free image recovery parameter to obtain a haze-free image as the iterative output of each round.

[0028] Further, the depth curve estimation network includes a residual SwinTransformer module and a large-core convolution attention module; the image recovery parameter is obtained based on the foggy image through a preset depth curve estimation network, including:

[0029] The shallow feature of the foggy image is obtained through convolution operation;

[0030] The shallow feature is subjected to feature extraction through the residual SwinTransformer block to obtain the global semantic feature of the foggy image;

[0031] The shallow feature is subjected to feature extraction through the large-core convolution attention module to obtain the texture detail feature of the foggy image;

[0032] The global semantic feature and the texture detail feature are subjected to fusion processing to obtain the image recovery parameter.

[0033] Further, the method further includes:

[0034] The image recovery parameter is subjected to feature enhancement processing through a cascaded structure of a preset local feature extraction module to obtain an enhanced parameter; the cascaded structure includes two convolution layers at the beginning and the end and two depth separable convolution layers and two depth separable dilated convolution layers in the middle;

[0035] obtain a spatial feature tensor corresponding to the attention mechanism structure of the local feature extraction module and the enhanced parameter;

[0036] obtain a corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure;

[0037] fuse the excitation tensor and the enhanced parameter to obtain an image restoration parameter after adaptive correction.

[0038] In a second aspect, the present application provides an image processing device based on curve restoration, comprising:

[0039] An acquisition unit is configured to acquire a foggy image to be processed.

[0040] A parameter unit is configured to obtain an image restoration parameter based on the foggy image through a preset depth curve estimation network.

[0041] A restoration unit is configured to perform multiple nonlinear curve mapping restoration on the foggy image based on the image restoration parameter to obtain a preliminary haze-free image.

[0042] An iteration unit is configured to perform iteration processing on the preliminary haze-free image based on the image restoration parameter through a preset image iteration defogging mechanism to obtain an iteration output of each round.

[0043] A recursion unit is configured to obtain a target haze-free image when detecting that a haze-free image corresponding to the iteration output meets a preset definition requirement.

[0044] In a third aspect, the present application further provides an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the electronic device is caused to implement the image processing method based on curve restoration as described above.

[0045] In a fourth aspect, the present application further provides a computer-readable storage medium having computer-readable instructions stored thereon, when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the image processing method based on curve restoration as described above.

[0046] In a fifth aspect, the present application further provides a computer program product or a computer program, the computer program product or the computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the image processing method based on curve restoration provided in the various optional embodiments described above.

[0047] It should be understood that the general description and detailed description that follow are merely exemplary and explanatory and are not restrictive of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. It is to be understood that the drawings are only schematic and that the figures in the following description are only some embodiments of the present application. Those skilled in the art will readily appreciate other advantages and / or embodiments of the present application upon reading this description with reference to the drawings, in which:

[0049] Figure 1 is a flow chart of a curve-based image processing method according to an example embodiment of the present application;

[0050] Figure 2 is an effect diagram of image defogging using the inverse image restoration parameter and the haze-free image restoration parameter according to an example embodiment of the present application;

[0051] Figure 3 is a diagram of implementing curve-based image processing using the inverse image restoration parameter and the haze-free image restoration parameter according to an example embodiment of the present application;

[0052] Figure 4 is a structural diagram of a depth curve estimation network according to an example embodiment of the present application;

[0053] Figure 5 is a structural diagram of a local feature extraction module according to an example embodiment of the present application;

[0054] Figure 6 is a diagram of performing 6 iterations of restoration on a synthesized uniform haze image according to an example embodiment of the present application;

[0055] Figure 7 is a diagram of comparing the effect of the curve-based image processing method according to the present application with the effect of processing outdoor haze images according to existing algorithms;

[0056] Figure 8 is a block diagram of a curve-based image processing device according to an example embodiment of the present application;

[0057] Figure 9 is a structural diagram of a computer system of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0058] The exemplary embodiments will be described in detail below with reference to the drawings. In the following description, the same numbers are used to denote the same elements throughout the several views. The following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0059] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0060] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed depending on the actual situation.

[0061] In the present application, "a plurality of" means two or more. The "and / or" describes the association between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0062] To solve the problem that the existing method depends on accurate parameter estimation, resulting in artifacts, color difference, and other problems in the defogging image, the embodiments of the present application propose an image processing method and device based on curve recovery, electronic equipment, and computer readable storage medium, which are mainly related to the image processing technology based on curve recovery included in digital image processing technology. The embodiments will be described in detail below.

[0063] First, refer to Figure 1 , Figure 1 is a flowchart of an image processing method based on curve recovery according to an exemplary embodiment of the present application. The method can be specifically executed by a server, which can be an independent server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services are not limited here.

[0064] As Figure 1As shown, in an exemplary embodiment, the image processing method based on curve recovery can include steps S101 to S105, which are described in detail as follows:

[0065] In step S101, a foggy image to be processed is obtained.

[0066] In step S102, an image recovery parameter is obtained based on the foggy image through a preset depth curve estimation network.

[0067] In step S103, a preliminary haze-free image is obtained by performing multiple nonlinear curve mapping recovery on the foggy image based on the image recovery parameter.

[0068] In step S104, an iterative output is obtained by performing iterative processing on the preliminary haze-free image based on the image recovery parameter through a preset image iterative haze removal mechanism.

[0069] In step S105, when the haze-free image corresponding to the iterative output meets the preset definition requirement, a target haze-free image is obtained.

[0070] As can be seen from the above, in the method provided in the embodiment, a foggy image to be processed is obtained; an image recovery parameter is obtained based on the foggy image through a preset depth curve estimation network; a preliminary haze-free image is obtained by performing multiple nonlinear curve mapping recovery on the foggy image based on the image recovery parameter; an iterative output is obtained by performing iterative processing on the preliminary haze-free image based on the image recovery parameter through a preset image iterative haze removal mechanism; and when the haze-free image corresponding to the iterative output meets the preset definition requirement, a target haze-free image is obtained. In this way, on the one hand, the image recovery parameter is used to recover the foggy image without making additional estimation on the physical properties of the scene, greatly reducing the prior dependence and computational complexity of the model; on the other hand, the nonlinear curve mapping and recursive iteration strategy are used to realize layered removal and adaptive restoration of the image haze, which not only ensures accurate recovery of the details and colors, but also has good cross-scene generalization ability and robustness. Therefore, the image processing method based on curve recovery provided in the application not only has a simple structure and is easy to implement, but also can be widely applied to different device platforms and complex scenes, and has important engineering application value and promotion prospects.

[0071] In an exemplary embodiment of the application, the image recovery parameter includes an inverse image recovery parameter and a haze-free image recovery parameter, which are set to be used for brightness mapping adjustment and contrast enhancement of the foggy image, respectively, to obtain a preliminary haze-free image. The specific steps can include:

[0072] Performing nonlinear enhancement processing on the foggy image based on the inverse image recovery parameter to obtain a processed image;

[0073] The foggy image is subjected to pixel normalization processing to obtain a normalized image, and pixel values of the normalized image are in a preset range.

[0074] In another example embodiment, the specific steps of performing non-linear enhancement processing on the foggy image based on the dehazing image restoration parameter to obtain the processed image can include:

[0075] The foggy image is subjected to pixel normalization processing to obtain a normalized image, and pixel values of the normalized image are in a preset range.

[0076] The normalized image is subjected to pixel inversion to obtain an inverted image.

[0077] The inverted image is subjected to curve mapping processing based on the dehazing image restoration parameter to obtain the processed image, so as to realize non-linear enhancement of image brightness.

[0078] In this embodiment, the input foggy image is given as The corresponding inverted image is which can be expressed by the formula: to emphasize the area in the image that is more heavily affected by fog. Then, the inverted image restoration parameter is introduced, and the brightness of the inverted image is subjected to non-linear enhancement through the following calculation formula:

[0079]

[0080] wherein, is the input foggy image, and pixel values of the input foggy image have been normalized to a preset range, such as ; represents a curve mapping operation, and the learnable inverted image restoration parameter , .

[0081] In this way, through the above-mentioned embodiments, the foggy image is subjected to pixel inversion after normalization processing, to obtain an inverted image, which is used to highlight the area in the original foggy image that is blocked by fog and haze, and the inverted image is subjected to non-linear adjustment by using adjustable inverted image restoration parameters, so as to restore the structural information of the original image.

[0082] In another example embodiment, the specific steps of performing curve mapping restoration on the processed image based on the dehazing image restoration parameter to obtain a preliminary dehazing image can include:

[0083] The processed image is subjected to curve mapping restoration based on the dehazing image restoration parameter to obtain a restored image.

[0084] The restored image is subjected to pixel inversion to obtain a preliminary dehazing image.

[0085] In this embodiment, the specific calculation formula of the preliminary dehazing image is as follows:

[0086]

[0087] wherein, represents a haze-free image; is a dehazing image recovered based on the inverse image recovery parameter and through curve mapping, i.e., the processed image in the above embodiment; is a haze-free image recovery parameter; represents a curve mapping operation.

[0088] Thus, the present application, through the above embodiment, further recovers the haze-free state of the image in the real scene, again performs clear image recovery on the dehazing image recovered through curve mapping, and obtains the preliminary haze-free image by taking the inverse.

[0089] In an example embodiment of the present application, the image recovery parameter includes the inverse image recovery parameter and the haze-free image recovery parameter. Corresponding to the preset image iterative dehazing mechanism, the specific steps of performing iterative processing on the preliminary haze-free image based on the image recovery parameter to obtain the iterative output of each round can include:

[0090] Through the preset image iterative dehazing mechanism, in each iteration, the preliminary haze-free image or the iterative output of the last round is subjected to nonlinear transformation based on the inverse image recovery parameter to generate an intermediate image.

[0091] The intermediate image is subjected to curve mapping recovery and pixel inversion based on the haze-free image recovery parameter to obtain a haze-free image as the iterative output of each round.

[0092] In the present embodiment, the inverse image recovery parameter is mainly used to distinguish the haze concentration area and map it to the inverted haze image, and the haze-free image recovery parameter is responsible for retaining the image details for the recovery mapping of the haze-free image. Please refer to Figure 2 , Figure 2 is an effect diagram of image dehazing using the inverse image recovery parameter and the haze-free image recovery parameter in an example embodiment of the present application. As Figure 2 shown, the present application can effectively reflect the scene depth information of the original haze image by using the inverse image recovery parameter and the haze-free image recovery parameter.

[0093] In the present embodiment, through the preset image iterative dehazing mechanism, in each iteration, the preliminary haze-free image or the iterative output of the last round is subjected to a new round of iterative processing based on the inverse image recovery parameter to remove the haze components in the image layer by layer. Preferably, in the first iteration, the haze-free image of the last round is first subjected to nonlinear transformation to generate an intermediate image, which can be represented as:

[0094] ​​​

[0095] The above curve transformation takes as input, and recovers the parameter to control the transformation strength, forming an intermediate representation of the inverse image of the fog map in the recursive process.

[0096] Then, the first layer clear image recovery result is calculated using the intermediate image, and the fog-free image corresponding to this iteration is obtained as the iteration output, expressed as follows:

[0097]

[0098] wherein, is the corresponding fog-free image recovery parameter, used to adjust the degree of suppression of residual fog components in the intermediate image; represents the curve mapping operation. Through the recursive operation of the corresponding steps of the above two calculation formulas, the gradual removal process of multiple levels of fog in the image is realized, which is formally represented as:

[0099]

[0100] In addition, it can be known that the process of each iteration of the above is the same as the operation steps in the embodiment of obtaining the processed image and the preliminary fog-free image in the foregoing, and therefore, as shown in Figure 3 , the curve recovery-based image processing is implemented using the inverse image recovery parameter and the fog-free image recovery parameter. Figure 3 is a schematic diagram of implementing curve recovery-based image processing using the inverse image recovery parameter and the fog-free image recovery parameter in an exemplary embodiment. Figure 3 In the embodiment shown, the input foggy image is pixel-inverted to obtain an inverse image, which is used to highlight the areas blocked by fog in the original foggy image; the inverse image obtained is applied with the inverse image recovery parameter obtained by the depth curve estimation network based on the input fog map, and the brightness and contrast thereof are nonlinearly adjusted to recover the original image structure information; the recovered image is used to perform curve function mapping again using the fog-free image recovery parameter obtained by the depth curve estimation network based on the input fog map, and pixel inversion is performed to obtain a preliminary fog-free image; an iteration mechanism is constructed based on the foregoing steps, and the output image of the previous iteration is used as the input image in each iteration recovery, and the inversion, curve recovery, and inverse mapping operations are repeated to gradually eliminate residual fog and improve the image clarity.

[0101] The preset image iterative defogging mechanism applied in the image processing method based on curve recovery provided in the application can use three fog image data sets of O-HAZE, Dense-Haze, NH-HAZE, and Live-500 during training. O-HAZE uses the first 40 images to train for 1000 epochs, and the remaining 5 images are tested. Dense-Haze and NH-HAZE use the first 50 images to train for 1000 epochs, and the remaining 5 images are tested. For the LIVE-500 data set, 200 epochs are trained and tested. During the training process, the ADAM (Adaptive Moment Estimation) optimizer is used, the parameter settings are and, the learning rate is fixed at 0.0001, and the training is performed on an NVIDIA GeForce 4090 graphics card. The number of iterations can be set to 8 times during application.

[0102] In this way, through the preset image iterative defogging mechanism, the output image of the previous round is used as the input image each round, and the reverse inversion, curve recovery, and inverse mapping operations are repeated to gradually eliminate residual fog and improve image clarity.

[0103] The image recovery parameters applied in the application can be obtained by the preset depth curve estimation network or manually defined to quickly remove relatively uniform haze. In an example embodiment of the application, the depth curve estimation network includes a residual SwinTransformer module and a large-core convolution attention module. The specific steps of obtaining the image recovery parameters can include:

[0104] obtaining shallow features of the foggy image through convolution operation;

[0105] extracting features of the shallow features through the residual SwinTransformer block to obtain global semantic features of the foggy image;

[0106] extracting features of the shallow features through the large-core convolution attention module to obtain texture detail features of the foggy image;

[0107] fusing and processing the global semantic features and the texture detail features to obtain the image recovery parameters.

[0108] Please refer to Figure 4 , Figure 4 is a structural schematic diagram of the depth curve estimation network in an example embodiment of the application. In this embodiment, the depth curve estimation network as shown in Figure 4 is used to output image recovery parameters , , and the foggy image is taken as input. First, shallow features of the input fog image are obtained through simple convolution operation The expression is as follows:

[0109]

[0110] Then, residual Swing Transformer blocks and large-kernel convolutional attention blocks are used respectively to obtain global semantic features of haze images. and texture detail features This allows for more accurate estimation of image restoration parameters. For simplicity, we use... and Representing the The RSTB and the The output of each LAB, the extracted intermediate features and global semantic features can be expressed by the following formula:

[0111]

[0112] in, For the first The role of the two convolutional layers following multiple RSTB methods is to enhance translation invariance and improve the fitting accuracy of curve parameters.

[0113] Meanwhile, another intermediate feature and texture detail features This can be expressed by the formula:

[0114]

[0115] In the formula, For the first A LAB method.

[0116] Finally, by fusing global semantic features and texture detail features Obtain image restoration parameters :

[0117]

[0118] in , Indicates the parameters for recovering the inverse image curve. This indicates the parameters for restoring the fog-free curve. These are weighting coefficients. The specific values ​​of these weighting coefficients are set according to the output requirements. For example, the output is... At that time, based on experience, it was set to 0.1.

[0119] Thus, through the above embodiments, this application can obtain the corresponding image restoration parameters based on the foggy image using a preset depth curve estimation network, making it more suitable for outdoor non-uniform fog scenes.

[0120] In an example embodiment of the present application, in order to more accurately restore the texture, color and other detailed information of the defogged image, a lightweight local feature extraction module based on large core attention is designed for adaptive correction of image restoration parameters, and the specific steps can include:

[0121] The image restoration parameters are subjected to feature enhancement processing through the cascade structure of the preset local feature extraction module to obtain enhanced parameters; the cascade structure includes convolution of the first and last two layers and depth separable convolution and depth separable dilated convolution of the middle two layers;

[0122] The spatial feature tensor corresponding to the attention mechanism structure of the local feature extraction module is obtained based on the enhanced parameters;

[0123] The corresponding excitation tensor is obtained based on the spatial feature tensor through the attention mechanism structure;

[0124] The image restoration parameters are subjected to feature enhancement processing through the cascade structure of the preset local feature extraction module to obtain enhanced parameters; the cascade structure includes convolution of the first and last two layers and depth separable convolution and depth separable dilated convolution of the middle two layers;

[0125] Please refer to Figure 5 , Figure 5 is a structure diagram of the local feature extraction module in an example embodiment of the present application. In the present embodiment, the image restoration parameters are subjected to adaptive correction through the local feature extraction module as shown in Figure 5 , preferably, the image restoration parameters are given as input features through the left cascade structure 0, the image restoration parameters are subjected to feature enhancement processing through the cascade structure to obtain enhanced parameters, and the process can be formalized as the following formula:

[0126]

[0127]

[0128] wherein, 0 is the input feature image restoration parameter; is the enhanced parameter. COA (Convolution Optimization Algorithm) is a convolution calculation function; MLP (Multilayer Perceptron) is a structure composed of multiple fully connected neural networks, which is mainly used for feature conversion and extraction in the convolution network. BN (Batch Normalization) is a standardization operation inserted between network layers, which aims to alleviate the problem of internal covariate shift (Internal Covariate Shift).

[0129] AsFigure 5 As shown in (a) in the middle, the LKCB (Local Kernel Convolution Block) in a cascaded structure is composed of four layers of convolution structure. The first and last layers use conventional convolution to enhance the nonlinear expression ability of features; the middle two layers are designed as depthwise separable convolution and depthwise separable expansion convolution, respectively, to effectively expand the receptive field and reduce the calculation cost. Among them, DSC (Depthwise Separable Convolution) reduces the number of parameters and improves the calculation efficiency by decomposing the standard convolution operation. And DSEC (Depthwise Separable Expansion Convolution) further introduces an expansion factor to capture larger scale fog information while maintaining high computational efficiency.

[0130] At the same time, in order to dynamically weight the extracted context information, a coordinate attention mechanism is introduced. The enhanced parameters and the spatial feature tensor corresponding to the attention mechanism structure of the local feature extraction module are obtained; the process of obtaining the corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure can be expressed by the following formula:

[0131]

[0132] Among them, and indicate the spatial feature tensor corresponding to the enhanced parameters obtained by the X-mean pooling and Y-mean pooling of the attention mechanism structure as shown in Figure 5 “ ” indicates the concatenation operation along the spatial dimension, which is equivalent to the “squeeze” process in the channel attention mechanism.

[0133] Then, the concatenated tensor is divided along the spatial dimension to obtain the divided feature maps and , and a nonlinear activation function is applied to perform the “excitation” operation. This process helps the network to learn the weight distribution in different spatial directions, thereby enhancing the attention to key areas.

[0134] Finally, the output of the attention mechanism structure is the weighted fusion result of the two excitation tensors and the enhanced parameters, that is:

[0135]

[0136] In the formula, is the Sigmoid activation function, The enhanced post-parameters output by the cascade structure.

[0137] In an embodiment of the present application, the image processing method based on curve restoration provided by the present application is applied to the dehazing restoration of the synthesized uniform haze image. In the embodiment, in order to verify the effectiveness of the iterative curve dehazing model proposed in the present application, a synthesized uniform haze image is selected as an experimental object, and the curve parameters are manually set and to control the bending degree of the curve, that is, to control the dehazing intensity of the image in each iteration. Among them, is used for inverse haze restoration, is used for haze-free image restoration. Specifically, when the haze in the image is more serious, in order to increase the image restoration intensity, the two curvature parameters are set to values far away from the coordinate origin (such as = -0.35, = 0.5); and in the case of light haze, the curvature parameters close to zero are used to prevent over-recovery and ensure the naturalness of the image.

[0138] In the embodiment, please refer to Figure 6 , Figure 6 is a schematic diagram of 6 times of iterative restoration of the synthesized uniform haze image in an embodiment of the present application. The first column of the diagram shows the change of the curve mapping relationship in each iteration, the horizontal axis represents the pixel value of the input image, and the vertical axis represents the corresponding output pixel value. As can be seen from the diagram, with the iteration, the shape of the recovery curve is gradually adjusted, which enhances the adaptability to different haze layers. IHR and HFR respectively represent the curve functions used in the inverse haze restoration (Inverse Haze Restoration) and haze-free image restoration (Haze-Free Restoration) processes. The second column and the third column of the diagram show the change process of the specific image effect, in which the haze-occluded area in the inverse haze restoration image is gradually enhanced, and the structure and details are gradually clear; and the finally output dehazing image presents a visual effect of contrast enhancement and natural color, which further verifies the effectiveness of the model.

[0139] In addition, as shown in Figure 7 , Figure 7 is a comparison diagram of the effect of the image processing method based on curve restoration provided by the present application and the existing algorithm for processing outdoor haze images. As can be seen from the diagram, the existing dehazing method based on the atmospheric scattering model is limited by the fixed assumption and model design, and it is difficult to adapt to non-ideal illumination in complex environments, so the problem of overall over-enhancement and brightness imbalance occurs. The iterative curve dehazing method proposed in the present application can better balance the image contrast and effectively reduce the haze residue, thereby having a better dehazing effect.

[0140] Figure 8is a block diagram of an image processing device 800 based on curve recovery according to an example embodiment of the present application. As shown in Figure 8 The device includes:

[0141] An acquisition unit 801 is configured to acquire a foggy image to be processed.

[0142] A parameter unit 802 is configured to obtain image recovery parameters based on the foggy image by using a preset depth curve estimation network.

[0143] A recovery unit 803 is configured to perform multi-nonlinear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image.

[0144] An iteration unit 804 is configured to perform iterative processing on the preliminary haze-free image based on the image recovery parameters by using a preset image iterative defogging mechanism to obtain an iteration output of each round.

[0145] A recursion unit 805 is configured to obtain a target haze-free image when detecting that the haze-free image corresponding to the iteration output meets a preset definition requirement.

[0146] The device applies the image processing method based on curve recovery provided by the present application, the acquisition unit 801 acquires a foggy image to be processed, the parameter unit 802 obtains image recovery parameters based on the foggy image by using a preset depth curve estimation network, the recovery unit 803 performs multi-nonlinear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image, the iteration unit 804 performs iterative processing on the preliminary haze-free image based on the image recovery parameters by using a preset image iterative defogging mechanism to obtain an iteration output of each round, and the recursion unit 805 obtains a target haze-free image when detecting that the haze-free image corresponding to the iteration output meets a preset definition requirement. In this way, the present application realizes image haze layering removal and adaptive restoration through nonlinear curve mapping and recursive iteration strategies, realizes simple and fast defogging of outdoor foggy images, ensures accurate recovery of details, structures and colors, has good cross-scene generalization ability and robustness, and solves the problem of false images, color difference and other problems caused by the dependence of existing methods on accurate parameter estimation.

[0147] In another example embodiment, the recovery unit 803 is further configured to perform nonlinear enhancement processing on the foggy image based on inverse image recovery parameters to obtain a processed image, and perform curve mapping recovery on the processed image based on haze-free image recovery parameters to obtain a preliminary haze-free image.

[0148] In another example embodiment, the recovery unit 803 is further configured to perform pixel normalization on the foggy image to obtain a normalized image, wherein pixel values of the normalized image are within a preset range; perform pixel inversion on the normalized image to obtain an inverse image; and perform curve mapping processing on the inverse image based on the inverse image recovery parameter to obtain a processed image, so as to achieve nonlinear enhancement of image brightness.

[0149] In another example embodiment, the recovery unit 803 is further configured to perform curve mapping recovery on the processed image based on the haze-free image recovery parameter to obtain a recovered image; and perform pixel inversion on the recovered image to obtain a preliminary haze-free image.

[0150] In another example embodiment, the image recovery parameter includes an inverse image recovery parameter and a haze-free image recovery parameter; and the iteration unit 804 is further configured to, through a preset image iterative defogging mechanism, in each round of iteration, perform nonlinear transformation on the preliminary haze-free image or the iteration output of the last round based on the inverse image recovery parameter to generate an intermediate image; and perform curve mapping recovery and pixel inversion on the intermediate image based on the haze-free image recovery parameter to obtain a haze-free image as the iteration output of each round.

[0151] In another example embodiment, the depth curve estimation network includes a residual SwinTransformer module and a large kernel convolution attention module; and the parameter unit 802 is further configured to obtain shallow features of the foggy image through convolution operation; extract features of the shallow features through the residual SwinTransformer block to obtain global semantic features of the foggy image; extract features of the shallow features through the large kernel convolution attention module to obtain texture detail features of the foggy image; and perform fusion processing on the global semantic features and the texture detail features to obtain the image recovery parameter.

[0152] In another example embodiment, the apparatus further includes:

[0153] The adaptive correction unit is configured to perform feature enhancement processing on the image recovery parameter through a cascaded structure of a preset local feature extraction module to obtain an enhanced parameter; the cascaded structure includes convolution of two head and tail layers and depth separable convolution and depth separable dilated convolution of two middle layers; obtain a spatial feature tensor corresponding to an attention mechanism structure of the local feature extraction module based on the enhanced parameter; obtain a corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure; and perform fusion processing on the excitation tensor and the enhanced parameter to obtain the adaptively corrected image recovery parameter.

[0154] It should be noted that the image processing apparatus based on curve recovery provided in the above embodiments and the image processing method based on curve recovery provided in the above embodiments belong to the same concept, where the specific manners in which various modules and units perform operations have been described in the method embodiments in detail, and will not be described here. The image processing apparatus based on curve recovery provided in the above embodiments can allocate the above functions to different functional modules to be completed according to actual needs in actual application, that is, the internal structure of the apparatus is divided into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0155] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the image processing method based on curve recovery provided in each of the above embodiments.

[0156] Figure 9 The structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. It should be noted that, Figure 9 The computer system 900 of the electronic device shown is only an example, and should not bring any limitation to the functions and use range of embodiments of the present application.

[0157] As Figure 9 shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or programs loaded from a storage portion 908 to a random access memory (RAM) 903, such as performing the methods in the above embodiments. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0158] The following components are connected to the I / O interface 905: an input part 906 including a keyboard, a mouse, etc.; an output part 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage part 908 including a hard disk, etc.; and a communication part 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read out therefrom is installed in the storage part 908 as necessary.

[0159] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0160] It should be noted that the computer-readable medium in the embodiments shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable signal medium can include a data signal propagating in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in connection with an instruction execution system, device or component. The computer program contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0161] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0162] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0163] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the image processing method based on curve recovery as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0164] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the image processing method based on curve recovery provided in the above embodiments.

[0165] The above is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement or improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of image processing based on curve restoration, characterized in that, The method comprises: obtaining a foggy image to be processed; obtaining image restoration parameters based on the foggy image through a preset depth curve estimation network; performing multiple non-linear curve mapping restoration on the foggy image based on the image restoration parameters to obtain a preliminary haze-free image; performing iterative processing on the preliminary haze-free image based on the image restoration parameters through a preset image iterative defogging mechanism to obtain an iteration output of each round; when it is detected that the haze-free image corresponding to the iteration output meets a preset definition requirement, obtaining a target haze-free image; wherein the image restoration parameters comprise inverse image restoration parameters and haze-free image restoration parameters; the performing multiple non-linear curve mapping restoration on the foggy image based on the image restoration parameters to obtain a preliminary haze-free image comprises: performing pixel normalization processing on the foggy image to obtain a normalized image, the pixel value of the normalized image being within a preset range; performing pixel inversion on the normalized image to obtain an inverse image; performing curve mapping processing on the inverse image based on the inverse image restoration parameters to obtain a processed image, so as to realize non-linear enhancement of image brightness; performing curve mapping restoration on the processed image based on the haze-free image restoration parameters to obtain a preliminary haze-free image.

2. The method of claim 1, wherein, The performing curve mapping restoration on the processed image based on the haze-free image restoration parameters to obtain a preliminary haze-free image comprises: performing curve mapping restoration on the processed image based on the haze-free image restoration parameters to obtain a restored image; performing pixel inversion on the restored image to obtain a preliminary haze-free image.

3. The method of claim 1, wherein, The image restoration parameters comprise inverse image restoration parameters and haze-free image restoration parameters; The performing iterative processing on the preliminary haze-free image based on the image restoration parameters through the preset image iterative defogging mechanism to obtain an iteration output of each round comprises: In each round of iteration, performing non-linear transformation on the preliminary haze-free image or the iteration output of the previous round based on the inverse image restoration parameters through the preset image iterative defogging mechanism to generate an intermediate image; performing curve mapping restoration and pixel inversion on the intermediate image based on the haze-free image restoration parameters to obtain a haze-free image as the iteration output of each round.

4. The method of claim 1, wherein, The depth curve estimation network comprises a residual SwinTransformer module and a large-core convolution attention module; the obtaining image restoration parameters based on the foggy image through the preset depth curve estimation network comprises: obtaining shallow features of the foggy image through convolution operation; obtaining global semantic features of the foggy image by performing feature extraction on the shallow features through the residual SwinTransformer block; obtaining texture detail features of the foggy image by performing feature extraction on the shallow features through the large-core convolution attention module; performing fusion processing on the global semantic features and the texture detail features to obtain image restoration parameters.

5. The method of claim 4, wherein, The method further comprises: The image recovery parameters are subjected to feature enhancement processing through a preset local feature extraction module cascade structure, to obtain enhanced parameters; the cascade structure includes convolution of two layers at the beginning and end, and depth separable convolution and depth separable dilated convolution of two layers in the middle; A spatial feature tensor corresponding to an attention mechanism structure of the local feature extraction module is obtained based on the enhanced parameters; An excitation tensor corresponding to the attention mechanism structure is obtained based on the spatial feature tensor; The excitation tensor and the enhanced parameters are subjected to fusion processing, to obtain self-adaptive corrected image recovery parameters.

6. An image processing apparatus based on curve restoration characterized by, Comprise: An acquisition unit configured to acquire a foggy image to be processed; A parameter unit configured to obtain image recovery parameters based on the foggy image through a preset depth curve estimation network; A recovery unit configured to perform multiple nonlinear curve mapping recovery on the foggy image based on the image recovery parameters, to obtain a preliminary haze-free image; An iteration unit configured to perform iteration processing on the preliminary haze-free image based on the image recovery parameters through a preset image iteration haze removal mechanism, to obtain an iteration output of each round; A recursion unit configured to obtain a target haze-free image when it is detected that a haze-free image corresponding to the iteration output meets a preset definition requirement; The image recovery parameters include inverse image recovery parameters and haze-free image recovery parameters; the recovery unit is further configured to perform pixel normalization processing on the foggy image, to obtain a normalized image, and pixel values of the normalized image are within a preset range; An inverse image is obtained by performing pixel inversion on the normalized image; A processed image is obtained by performing curve mapping processing on the inverse image based on the inverse image recovery parameters, to realize nonlinear enhancement of image brightness; A preliminary haze-free image is obtained by performing curve mapping recovery on the processed image based on the haze-free image recovery parameters.

7. An electronic device, comprising: Comprise: One or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method based on curve recovery according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer readable instruction is stored thereon, which, when executed by a processor of a computer, causes the computer to execute the image processing method based on curve recovery according to any one of claims 1 to 5.

Images