Image processing method based on curve recovery and related equipment

Abstract

The invention discloses an image processing method based on curve recovery and related equipment. The method comprises the following steps: acquiring a to-be-processed foggy day image; through a preset depth curve estimation network, obtaining an image restoration parameter based on the foggy day image; performing multiple nonlinear curve mapping recovery on the foggy day image based on the image recovery parameters to obtain a preliminary fogless image; through a preset image iteration defogging mechanism, performing iteration processing on the initial fogless image based on the image recovery parameters to obtain each iteration output; and when it is detected that the fogless image corresponding to the iterative output meets a preset definition requirement, obtaining a target fogless image. According to the method, layered removal and self-adaptive restoration of image haze can be realized through nonlinear curve mapping and a recursive iteration strategy, and simple and rapid defogging of an outdoor foggy day image is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital image processing, and in particular to an image processing method based on curve recovery and related equipment. BACKGROUND

[0002] Nowadays, the actual promotion and application of intelligent vision technology still faces many challenges, among which the restriction of environmental factors on system performance is particularly prominent. In the haze environment, the visual perception system is difficult to accurately capture the information of the real world, which directly leads to obstacles in computer vision tasks such as image recognition and target tracking.

[0003] The existing image defogging method based on the atmospheric scattering model has the problem of poor parameter adaptability. The atmospheric scattering model involves multiple complex physical processes, forming a highly ill-conditioned equation set. In this case, it is difficult to accurately estimate the atmospheric scattering parameters in various complex scenes, whether relying on prior assumptions or deep neural networks. The inaccurate estimation of parameters in the atmospheric scattering model often directly leads to unnecessary visual phenomena such as image artifacts and unbalanced contrast.

[0004] To solve this problem, some research works mainly explore from two directions. On the one hand, by improving the estimation algorithm of atmospheric scattering parameters, the feature extraction and representation ability of deep neural networks is enhanced to improve the estimation accuracy of model parameters. On the other hand, starting from model structure optimization, the construction of the atmospheric scattering model is adjusted to reduce cumulative errors or improve modeling ability. Although these researches improve the defogging effect to some extent, due to the inherent ill-conditioned nature of the atmospheric scattering model, the parameter estimation still faces great challenges, and the instability of the solution has not been fundamentally solved, which makes the defogging performance still have certain limitations. SUMMARY

[0005] In order to solve the above problems, the embodiments of the present application provide an image processing method and device based on curve recovery, an electronic device, a computer readable storage medium, and a computer program product.

[0006] In a first aspect, to solve the above technical problems, the present application provides an image processing method based on curve recovery, comprising: obtaining a foggy image to be processed; obtaining image recovery parameters based on the foggy image through a preset depth curve estimation network; performing multiple nonlinear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image; performing iterative processing on the preliminary haze-free image based on the image recovery parameters through a preset image iterative defogging mechanism to obtain each round of iteration output; When it is detected that the haze-free image corresponding to the iteration output meets the preset definition requirement, a target haze-free image is obtained.

[0007] The beneficial effects are: In the technical scheme provided in the embodiments of the present application, a haze image to be processed is obtained; an image restoration parameter is obtained based on the haze image through a preset depth curve estimation network; a preliminary haze-free image is obtained by performing multiple non-linear curve mapping recovery on the haze image based on the image restoration parameter; each round of iteration output is obtained by performing iteration processing on the preliminary haze-free image based on the image restoration parameter through a preset image iteration haze removal mechanism; and a target haze-free image is obtained when it is detected that the haze-free image corresponding to the iteration output meets the preset definition requirement. In this way, the image haze is removed and restored adaptively through the non-linear curve mapping and recursive iteration strategy, the outdoor haze image is quickly and simply removed, the accurate recovery of the details and colors is ensured, good cross-scene generalization ability and robustness are achieved, and the problems of the existing methods, such as the occurrence of artifacts and color difference in the haze-removed image due to the dependence on accurate parameter estimation, are solved.

[0008] Further, the image restoration parameter includes an inverse image restoration parameter and a haze-free image restoration parameter; and the multiple non-linear curve mapping recovery on the haze image based on the image restoration parameter to obtain the preliminary haze-free image includes: performing non-linear enhancement processing on the haze image based on the inverse image restoration parameter to obtain a processed image; performing curve mapping recovery on the processed image based on the haze-free image restoration parameter to obtain the preliminary haze-free image.

[0009] Further, the non-linear enhancement processing on the haze image based on the inverse image restoration parameter to obtain the processed image includes: performing pixel normalization processing on the haze image to obtain a normalized image, and the pixel value of the normalized image is in a preset range; performing pixel inversion on the normalized image to obtain an inverse image; performing curve mapping processing on the inverse image based on the inverse image restoration parameter to obtain the processed image, so as to realize non-linear enhancement of image brightness.

[0010] Further, the curve mapping recovery on the processed image based on the haze-free image restoration parameter to obtain the preliminary haze-free image includes: performing curve mapping recovery on the processed image based on the haze-free image restoration parameter to obtain a recovered image; performing pixel inversion on the recovered image to obtain the preliminary haze-free image.

[0011] Further, the image restoration parameter comprises an inverse image restoration parameter and a haze-free image restoration parameter. By means of a preset image iterative defogging mechanism, the preliminary haze-free image is iteratively processed based on the image restoration parameter, to obtain an iteration output of each round, comprising: By means of the preset image iterative defogging mechanism, in each round of iteration, the preliminary haze-free image or the iteration output of the last round is subjected to nonlinear transformation based on the inverse image restoration parameter, to generate an intermediate image. Based on the haze-free image restoration parameter, the intermediate image is subjected to curve mapping restoration and pixel inversion, to obtain a haze-free image as the iteration output of each round.

[0012] Further, the depth curve estimation network comprises a residual SwinTransformer module and a large-core convolution attention module; by means of a preset depth curve estimation network, the image restoration parameter is obtained based on the foggy image, comprising: The shallow feature of the foggy image is obtained by convolution operation; The shallow feature is subjected to feature extraction by the residual SwinTransformer block, to obtain the global semantic feature of the foggy image; The shallow feature is subjected to feature extraction by the large-core convolution attention module, to obtain the texture detail feature of the foggy image; The global semantic feature and the texture detail feature are subjected to fusion processing, to obtain the image restoration parameter.

[0013] Further, the method further comprises: The image restoration parameter is subjected to feature enhancement processing by means of a cascaded structure of a preset local feature extraction module, to obtain an enhanced parameter; the cascaded structure comprises two convolution layers at the beginning and the end, and two deep separable convolution layers and two deep separable dilated convolution layers in the middle; The spatial feature tensor corresponding to the attention mechanism structure of the local feature extraction module is obtained for the enhanced parameter; By means of the attention mechanism structure, the corresponding excitation tensor is obtained based on the spatial feature tensor; The excitation tensor and the enhanced parameter are subjected to fusion processing, to obtain the self-adaptive corrected image restoration parameter.

[0014] In a second aspect, the present application provides an image processing device based on curve restoration, comprising: An acquisition unit is configured to acquire a foggy image to be processed; A parameter unit is configured to obtain an image restoration parameter based on the foggy image by means of a preset depth curve estimation network; a recovery unit, configured to perform multiple non-linear curve mapping recovery on the foggy image based on the image recovery parameter to obtain a preliminary haze-free image; an iteration unit, configured to perform iteration processing on the preliminary haze-free image based on the image recovery parameter through a preset image iteration defogging mechanism to obtain an iteration output of each round; a recursion unit, configured to obtain a target haze-free image when it is detected that the iteration output corresponds to a haze-free image satisfying a preset definition requirement.

[0015] In a third aspect, the present application also provides an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the curve-recovery-based image processing method as described above.

[0016] In a fourth aspect, the present application also provides a computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor of a computer, cause the computer to perform the curve-recovery-based image processing method as described above.

[0017] In a fifth aspect, the present application also provides a computer program product or a computer program, which comprises computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the curve-recovery-based image processing method provided in the various optional embodiments described above.

[0018] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application. It is obvious that the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. In the drawings: Figure 1 is a flowchart of a curve-recovery-based image processing method according to an exemplary embodiment of the present application; Figure 2 is an effect diagram of image defogging using inverse image recovery parameters and haze-free image recovery parameters according to an exemplary embodiment of the present application; Figure 3Fig. 1 is a schematic diagram of implementing curve restoration based image processing by using the inverse image restoration parameter and the haze-free image restoration parameter in an example embodiment of the present application; Figure 4 Fig. 2 is a schematic diagram of a structure of a depth curve estimation network in an example embodiment of the present application; Figure 5 Fig. 3 is a schematic diagram of a structure of a local feature extraction module in an example embodiment of the present application; Figure 6 Fig. 4 is a schematic diagram of performing 6 times of iterative restoration on a synthesized uniform haze image in an example embodiment of the present application; Figure 7 Fig. 5 is a schematic diagram of comparing the effect of the curve restoration based image processing method provided by the present application with the effect of processing outdoor haze weather images by using the existing algorithm; Figure 8 Fig. 6 is a block diagram of a curve restoration based image processing device shown in an example embodiment of the present application; Figure 9 Fig. 7 is a schematic diagram of a structure of a computer system of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0020] The example embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are merely illustrative, and do not necessarily include all contents and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so that the actual execution order can be changed according to the actual situation.

[0023] In the present application, "multiple" refers to two or more. The association relationship of "and / or" describes the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0024] To solve the problem that the existing method depends on accurate parameter estimation, resulting in artifacts, color difference and other problems in the defogging image, an embodiment of the present application provides an image processing method and device based on curve recovery, an electronic device and a computer readable storage medium, which are mainly related to the image processing technology based on curve recovery in the digital image processing technology. The embodiments will be described in detail below.

[0025] First, please refer to Figure 1 , Figure 1 is a flowchart of an image processing method based on curve recovery according to an example embodiment of the present application. The method can be specifically executed by a server, which can be an independent server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. This place is not limited.

[0026] As shown in Figure 1 , in an example embodiment, the image processing method based on curve recovery can include steps S101 to S105, which are described in detail as follows: Step S101, obtaining a foggy image to be processed.

[0027] Step S102, obtaining image recovery parameters based on the foggy image through a preset depth curve estimation network.

[0028] Step S103, performing multiple nonlinear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image.

[0029] Step S104, performing iterative processing on the preliminary haze-free image based on the image recovery parameters through a preset image iterative defogging mechanism to obtain an iteration output of each round.

[0030] Step S105, when the haze-free image corresponding to the iteration output meets the preset clarity requirement, obtaining a target haze-free image.

[0031] It can be seen from the above that in the method provided in the embodiment, the foggy image to be processed is obtained; the image restoration parameter is obtained based on the foggy image through a preset depth curve estimation network; the foggy image is subjected to multi-nonlinear curve mapping restoration based on the image restoration parameter, and a preliminary haze-free image is obtained; the preliminary haze-free image is subjected to iterative processing based on the image restoration parameter through a preset image iterative defogging mechanism, and each round of iteration output is obtained; when it is detected that the haze-free image corresponding to the iteration output meets the preset definition requirement, the target haze-free image is obtained. In this way, on the one hand, the image is restored through the image restoration parameter, without making additional estimation on the physical properties in the scene, which greatly reduces the prior dependence and operation complexity of the model; on the other hand, the image haze is removed and restored adaptively through the nonlinear curve mapping and recursive iteration strategy, which not only ensures the accurate recovery of the details and colors, but also has good cross-scene generalization ability and robustness. Therefore, the image processing method based on curve restoration provided in the application not only has a simple structure and is easy to implement, but also can be widely applied to different device platforms and complex scenes, and has important engineering application value and promotion prospect.

[0032] In an example embodiment of the application, the image restoration parameter includes an inverse image restoration parameter and a haze-free image restoration parameter, which are set to be used for brightness mapping adjustment and contrast enhancement of the foggy image respectively, so as to obtain the preliminary haze-free image. The specific steps can include: performing nonlinear enhancement processing on the foggy image based on the inverse image restoration parameter to obtain a processed image; performing curve mapping restoration on the processed image based on the haze-free image restoration parameter to obtain the preliminary haze-free image.

[0033] In another example embodiment, the specific steps of performing nonlinear enhancement processing on the foggy image based on the inverse image restoration parameter to obtain a processed image can include: performing pixel normalization processing on the foggy image to obtain a normalized image, and the pixel value of the normalized image is in a preset range; performing pixel inversion on the normalized image to obtain an inverse image; performing curve mapping processing on the inverse image based on the inverse image restoration parameter to obtain the processed image, so as to realize nonlinear enhancement of the image brightness.

[0034] In the embodiment, the input foggy image is given , and the corresponding inverse image is which can be represented by the formula: to emphasize the areas in the image that are more affected by fog. Then, the inverse image restoration parameter is introduced, and the brightness of the inverted image is nonlinearly enhanced through the following calculation formula: wherein, is an input foggy image, whose pixel values have been normalized to a preset range, such as ; denotes a curve mapping operation, which employs a learnable inverse image restoration parameter , .

[0035] Thus, in the above embodiment, the foggy image is normalized and pixel-inverted to obtain an inverse image, which is used to highlight the area blocked by the fog in the original foggy image, and the inverse image is subjected to nonlinear adjustment by applying an adjustable inverse image restoration parameter to restore the original image structure information.

[0036] In another exemplary embodiment, the specific steps of recovering the preliminary haze-free image based on the haze-free image restoration parameter by curve mapping the processed image can include: recovering the processed image by curve mapping based on the haze-free image restoration parameter to obtain a recovered image; pixel-inverting the recovered image to obtain the preliminary haze-free image.

[0037] In this embodiment, the specific calculation formula for obtaining the preliminary haze-free image is as follows: wherein, denotes a haze-free image; is the inverse image recovered by curve mapping based on the inverse image restoration parameter, i.e., the processed image in the above embodiment; is the haze-free image restoration parameter; denotes a curve mapping operation.

[0038] Thus, in the above embodiment, for further recovery of the haze-free state of the image in the real scene, the inverse image recovered by curve mapping is subjected to clear image recovery again, and the preliminary haze-free image is obtained by negation.

[0039] In an exemplary embodiment of the present application, the image restoration parameter includes the inverse image restoration parameter and the haze-free image restoration parameter, and the specific steps of iteratively processing the preliminary haze-free image based on the image restoration parameter by a preset image iterative defogging mechanism to obtain the iterative output of each round can include: by the preset image iterative defogging mechanism, in each iteration, the preliminary haze-free image or the iterative output of the last round is subjected to nonlinear transformation based on the inverse image restoration parameter to generate an intermediate image; the intermediate image is recovered by curve mapping based on the haze-free image restoration parameter, and pixel-inverted to obtain a haze-free image as the iterative output of each round.

[0040] In this embodiment, the inverse image restoration parameters are mainly used to distinguish fog density areas and map them to the inverted haze image, while the fog-free image restoration parameters are responsible for preserving image details for use in the restoration mapping of the fog-free image. Figure 2 , Figure 2 This is a schematic diagram of the effect of image defogging using inverse image restoration parameters and fog-free image restoration parameters in an exemplary embodiment of the present application. Figure 2 As shown, the present application utilizes inverse image restoration parameters and fog-free image restoration parameters to effectively reflect the scene depth information of the original foggy image.

[0041] In this embodiment, through the preset image iterative defogging mechanism, in each round of iteration, a new round of iterative processing is performed on the preliminary fog-free image or the iterative output of the previous round based on the inverse image restoration parameters to remove the haze components in the image layer by layer. In the iteration, first the fog-free image of the previous round is Perform nonlinear transformation to generate intermediate images , which can be expressed as: The above curve transformation is As input, and the parameters are restored by inverse image Control the transformation strength to form an intermediate representation for recovering the inverse of the fog image during the recursive process.

[0042] Next, use this intermediate image to calculate the Clear image restoration results of the layer , get the fog-free image corresponding to this iteration as the iterative output, the expression is as follows: in, is the corresponding haze-free image restoration parameter, which is used to adjust the degree of suppression of the residual haze components in the intermediate image; Represents the curve mapping operation. By recursively performing the steps corresponding to the above two calculation formulas alternately, the process of gradually removing multiple levels of haze in the image is achieved, which can be formally expressed as: In addition, it can be seen that the above-mentioned iterative process is the same as the calculation steps in the embodiment of obtaining the processed image and the preliminary fog-free image in the previous text, so Figure 3 As shown, Figure 3 FIG. 1 is a schematic diagram of implementing curve-based restoration image processing using inverse image restoration parameters and haze-free image restoration parameters in an exemplary embodiment. Figure 3In the illustrated embodiment, the input foggy image is pixel-inverted to obtain an inverse image, which is used to highlight the area blocked by the haze in the original foggy image; the inverse image obtained is applied with the inverse image restoration parameters obtained by the depth curve estimation network based on the input fog image, and the brightness and contrast thereof are non-linearly adjusted to restore the original image structure information; the restored image is used to perform curve function mapping again using the haze-free image restoration parameters obtained by the depth curve estimation network based on the input fog image, and pixel inversion is performed to obtain a preliminary haze-free image; an iterative mechanism is constructed based on the foregoing steps, and each round of iteration uses the output image of the previous round as the input image, and the inversion, curve restoration and inverse mapping operations are repeated to gradually eliminate the residual haze and improve the image clarity.

[0043] The preset image iterative defogging mechanism applied in the image processing method based on curve restoration provided in the present application can use three fog image data sets of O-HAZE, Dense-Haze, NH-HAZE and Live-500 during training. O-HAZE is used to train the first 40 images for 1000 epochs, and the remaining 5 images are used for testing. Dense-Haze and NH-HAZE are used to train the first 50 images for 1000 epochs, and the remaining 5 images are used for testing. For the LIVE-500 data set, 200 epochs are trained and tested. During the training process, the ADAM (Adaptive Moment Estimation) optimizer is used, the parameter settings are and the learning rate is fixed at 0.0001, and the training is performed on an NVIDIA GeForce 4090 graphics card. The number of iterations can be set to 8 times during application.

[0044] In this way, through the preset image iterative defogging mechanism, the output image of the previous round is used as the input image in each round, and the inversion, curve restoration and inverse mapping operations are repeated, so that the residual haze can be gradually eliminated and the image clarity can be improved.

[0045] The image restoration parameters applied in the present application can be obtained by the preset depth curve estimation network or manually defined to quickly remove relatively uniform haze. In an exemplary embodiment of the present application, the depth curve estimation network includes a residual SwinTransformer module and a large-core convolution attention module. The specific steps of obtaining the image restoration parameters can include: obtaining the shallow features of the foggy image through convolution operation; extracting the shallow features through the residual SwinTransformer block to obtain the global semantic features of the foggy image; extracting the shallow features through the large-core convolution attention module to obtain the texture detail features of the foggy image; The global semantic feature and the texture detail feature are fused to obtain an image restoration parameter.

[0046] Please refer to Figure 4 , Figure 4 is a structural diagram of a depth curve estimation network in an exemplary embodiment of the present application. In the embodiment, a depth curve estimation network as shown in Figure 4 is used to output an image restoration parameter , . Taking a foggy image as input, shallow features of the input fog image are first obtained through simple convolution operations , Then, a residual SwinTransformer block and a large kernel convolution attention block are respectively used to obtain global semantic features and texture detail features of the foggy image to more accurately estimate the image restoration parameter. For convenience of representation, the outputs of the th RSTB and the th LAB are represented by and , and the extracted intermediate features and the global semantic features can be expressed as follows: wherein is the th RSTB method, and the two convolution layers after the multiple RSTBs are used to enhance the translational invariance and improve the fitting accuracy of the curve parameters.

[0047] Meanwhile, another intermediate feature and the texture detail feature can be expressed as follows: wherein is the th LAB method.

[0048] Finally, the image restoration parameter is obtained by fusing the global semantic feature and the texture detail feature : wherein , represents the inverse image curve restoration parameter, and is a weight coefficient, and a specific value of the weight coefficient is set according to output requirements, for example, when the output is 0.1 according to experience.

[0049] In this way, the image restoration parameter corresponding to the foggy image can be obtained based on the preset depth curve estimation network, so that the image restoration parameter is more suitable for the outdoor non-uniform fog scene.

[0050] In an example embodiment of the present application, in order to more accurately restore the texture, color and other detail information of the dehazed image, a lightweight local feature extraction module based on a large core attention mechanism is designed to adaptively correct the image restoration parameter, and the specific steps can include: performing feature enhancement processing on the image restoration parameter through the cascade structure of the preset local feature extraction module to obtain an enhanced parameter; the cascade structure includes two convolutional layers at the beginning and the end, and two deep separable convolutional layers and two deep separable dilated convolutional layers in the middle; obtaining a spatial feature tensor corresponding to the attention mechanism structure of the local feature extraction module based on the enhanced parameter; obtaining a corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure; performing fusion processing on the excitation tensor and the enhanced parameter to obtain an adaptively corrected image restoration parameter.

[0051] Please refer to Figure 5 , Figure 5 is a structural schematic diagram of the local feature extraction module in an example embodiment of the present application. In the present embodiment, the image restoration parameter is adaptively corrected through the local feature extraction module as shown in Figure 5 Preferably, the image restoration parameter is given as an input feature 0 through the cascade structure on the left, the image restoration parameter is processed through the cascade structure to obtain an enhanced parameter, and the process can be formalized as the following formula: wherein, 0 is the input feature image restoration parameter; To enhance the posterior parameters. COA (Convolution Optimization Algorithm) is a convolution calculation function; MLP (Multilayer Perceptron) is a structure composed of multiple fully connected neural networks, which is mainly used for feature conversion and extraction in convolutional networks. BN (Batch Normalization) is a standardization operation inserted between network layers, aiming to alleviate the problem of Internal Covariate Shift.

[0052] As shown in (a) in FIG. 1, LKCB (Local Kernel Convolution Block) as a cascade structure is composed of four layers of convolution structure. Among them, the first and last two layers use conventional convolution to enhance the nonlinear expression ability of features; the middle two layers are designed as depthwise separable convolution and depthwise separable expansion convolution respectively, to effectively expand the receptive field and reduce the calculation cost. Among them, DSC (Depthwise Separable Convolution) reduces the number of parameters and improves the calculation efficiency by decomposing the standard convolution operation. And DSEC (Depthwise Separable Expansion Convolution) further introduces a dilation factor to capture larger scale fog information while maintaining high computational efficiency. Figure 5 Meanwhile, in order to dynamically weight the extracted context information, a coordinate attention mechanism is introduced. The spatial feature tensor corresponding to the attention mechanism structure of the enhanced posterior parameters and the local feature extraction module is obtained; the process of obtaining the corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure can be expressed by the following formula:

[0053] Among them, and represent the spatial feature tensor corresponding to the enhanced posterior parameters obtained by the X-mean pooling and Y-mean pooling of the attention mechanism structure as shown in FIG. 2; “ ” represents the concatenation operation along the spatial dimension, which is equivalent to the “squeeze” process in the channel attention mechanism. Figure 5 Then the concatenated tensor is divided along the spatial dimension to obtain the divided feature maps and

[0054] ​​​and a non-linear activation function is applied to perform an Excitation operation. This process helps the network to learn the weight distribution in different spatial directions, thus enhancing the focus on key regions.

[0055] Finally, the output of the attention mechanism structure is the weighted fusion result of the two excitation tensors and the enhanced parameters, i.e., wherein is a Sigmoid activation function, is the enhanced parameter output by the cascade structure.

[0056] In an embodiment of the present application, the image processing method based on curve recovery provided by the present application is applied to the dehazing recovery of the synthesized uniform haze image. In the present embodiment, in order to verify the effectiveness of the iterative curve dehazing model proposed in the present application, a synthesized uniform haze image is selected as the experimental object, and the curve parameters are manually set as and to control the bending degree of the curve, i.e., to control the dehazing intensity of the image in each iteration. Among them, is used for inverse haze restoration, is used for haze-free image restoration. Specifically, when the haze in the image is severely blocked, in order to increase the image restoration intensity, the two curvature parameters are set to values far away from the coordinate origin (such as = -0.35, = 0.5); and in the case of light haze, the curvature parameters close to zero are used to prevent over-recovery and ensure the naturalness of the image.

[0057] In the present embodiment, please refer to Figure 6 , Figure 6 is a schematic diagram of the 6-time iterative recovery of the synthesized uniform haze image in an embodiment of the present application. The first column of the diagram shows the change of the curve mapping relationship in each iteration, wherein the horizontal axis represents the pixel value of the input image, and the vertical axis represents the corresponding output pixel value. As can be seen from the diagram, with the progress of the iteration, the shape of the recovery curve is gradually adjusted, which enhances the adaptability to different haze layers. IHR and HFR respectively represent the curve functions used in the inverse haze restoration (Inverse Haze Restoration) and haze-free image restoration (Haze-Free Restoration) processes. The second column and the third column of the diagram show the change process of the specific image effect, wherein the haze blocked area in the inverse haze restoration image is gradually enhanced, and the structure and details are gradually clear; and the dehazing image finally output presents a visual effect of enhanced contrast and natural color, which further verifies the effectiveness of the model.

[0058] In addition, as Figure 7As shown, Figure 7 is a comparison diagram of the image processing method based on curve recovery provided by the present application and the effect of the existing algorithm in processing outdoor foggy image. As can be seen from the figure, the existing dehazing method based on atmospheric scattering model is limited by fixed assumptions and model design, and it is difficult to adapt to non-ideal lighting in complex environment, thereby the problems of overall over-enhancement and brightness imbalance occur. The iterative curve dehazing method proposed by the present application can better balance the image contrast and effectively reduce the fog residue, thereby having better dehazing effect.

[0059] Figure 8 is a block diagram of an image processing device 800 based on curve recovery according to an example embodiment of the present application. As Figure 8 shown, the device comprises: An acquisition unit 801 is configured to acquire a foggy image to be processed. A parameter unit 802 is configured to obtain image recovery parameters based on the foggy image through a preset depth curve estimation network. A recovery unit 803 is configured to perform multiple non-linear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image. An iteration unit 804 is configured to perform iterative processing on the preliminary haze-free image based on the image recovery parameters through a preset image iterative dehazing mechanism to obtain an iteration output of each round. A recursion unit 805 is configured to obtain a target haze-free image when detecting that the haze-free image corresponding to the iteration output meets a preset definition requirement.

[0060] The device applies the image processing method based on curve recovery provided by the present application, the acquisition unit 801 acquires a foggy image to be processed, the parameter unit 802 obtains image recovery parameters based on the foggy image through a preset depth curve estimation network, the recovery unit 803 performs multiple non-linear curve mapping recovery on the foggy image based on the image recovery parameters to obtain a preliminary haze-free image, the iteration unit 804 performs iterative processing on the preliminary haze-free image based on the image recovery parameters through a preset image iterative dehazing mechanism to obtain an iteration output of each round, and the recursion unit 805 obtains a target haze-free image when detecting that the haze-free image corresponding to the iteration output meets a preset definition requirement. In this way, the present application realizes image haze layering removal and adaptive restoration through non-linear curve mapping and recursive iteration strategy, realizes simple and fast dehazing of outdoor foggy image, ensures accurate recovery of detail structure and color, has good cross-scene generalization ability and robustness, and solves the problems of existing methods, such as dependence on accurate parameter estimation, false image, color difference, etc.

[0061] In another example embodiment, the recovery unit 803 is further configured to perform non-linear enhancement processing on the foggy image based on the inverse image recovery parameter to obtain a processed image, and perform curve mapping recovery on the processed image based on the haze-free image recovery parameter to obtain the preliminary haze-free image.

[0062] In another example embodiment, the recovery unit 803 is further configured to perform pixel normalization processing on the foggy image to obtain a normalized image, wherein pixel values of the normalized image are within a preset range, perform pixel inversion on the normalized image to obtain an inverse image, and perform curve mapping processing on the inverse image based on the inverse image recovery parameter to obtain the processed image, so as to achieve non-linear enhancement of image brightness.

[0063] In another example embodiment, the recovery unit 803 is further configured to perform curve mapping recovery on the processed image based on the haze-free image recovery parameter to obtain a recovered image, and perform pixel inversion on the recovered image to obtain the preliminary haze-free image.

[0064] In another example embodiment, the image recovery parameter includes the inverse image recovery parameter and the haze-free image recovery parameter, and the iteration unit 804 is further configured to, through a preset image iterative defogging mechanism, in each iteration, perform non-linear transformation on the preliminary haze-free image or the iteration output of the last round based on the inverse image recovery parameter to generate an intermediate image, and perform curve mapping recovery and pixel inversion on the intermediate image based on the haze-free image recovery parameter to obtain a haze-free image as the iteration output of each round.

[0065] In another example embodiment, the depth curve estimation network includes a residual SwinTransformer module and a large kernel convolution attention module, and the parameter unit 802 is further configured to obtain shallow features of the foggy image through convolution operation, extract features of the shallow features through the residual SwinTransformer block to obtain global semantic features of the foggy image, extract features of the shallow features through the large kernel convolution attention module to obtain texture detail features of the foggy image, and perform fusion processing on the global semantic features and the texture detail features to obtain the image recovery parameter.

[0066] In another example embodiment, the device further includes: The adaptive correction unit is configured to perform feature enhancement processing on the image recovery parameter through a cascaded structure of a preset local feature extraction module to obtain an enhanced parameter, wherein the cascaded structure includes two convolution layers at the beginning and the end, and two depth separable convolution layers and two depth separable dilated convolution layers in the middle; obtain a spatial feature tensor corresponding to the attention mechanism structure of the local feature extraction module based on the enhanced parameter; obtain a corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure; and perform fusion processing on the excitation tensor and the enhanced parameter to obtain the adaptively corrected image recovery parameter.

[0067] It should be noted that the image processing apparatus based on curve recovery provided by the above-mentioned embodiments and the image processing method based on curve recovery provided by the above-mentioned embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. The image processing apparatus based on curve recovery provided by the above-mentioned embodiments can be divided into different functional modules to complete the above-described all or part of the functions according to the needs in the actual application, i.e., the internal structure of the apparatus is divided into different functional modules to complete the above-described all or part of the functions, and this is not limited herein.

[0068] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the image processing method based on curve recovery provided in each of the above-mentioned embodiments.

[0069] Figure 9 The structural schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application is shown. It should be noted that, Figure 9 The computer system 900 of the electronic device shown is only an example, and should not limit the functions and use range of the embodiments of the present application.

[0070] As Figure 9 shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or programs loaded from a storage portion 908 into a random access memory (RAM) 903, such as performing the methods in the above-mentioned embodiments. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0071] The following components are connected to the I / O interface 905: an input part 906 including a keyboard, a mouse, etc.; an output part 907 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), etc., and a speaker, etc.; a storage part 908 including a hard disk, etc.; and a communication part 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read out therefrom is installed in the storage part 908 as necessary.

[0072] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0073] It should be noted that the computer-readable medium in the embodiments shown in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable signal medium can include a data signal propagating in a baseband or as a carrier wave in a propagated data signal, in which the computer-readable computer program is carried. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit programs for use by or in connection with an instruction execution system, device or component. The computer program contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0074] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks represented in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0075] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0076] Another aspect of the present application also provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the image processing method based on curve recovery as described above. The computer readable storage medium can be included in the electronic device described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0077] Another aspect of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the image processing method based on curve recovery provided in the above embodiments.

[0078] The above is only a preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement or improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of image processing based on curve restoration, characterized in that, The method comprises: acquiring a foggy image to be processed; obtaining image restoration parameters based on the foggy image through a preset depth curve estimation network; performing multiple non-linear curve mapping restoration on the foggy image based on the image restoration parameters to obtain a preliminary haze-free image; performing iterative processing on the preliminary haze-free image based on the image restoration parameters through a preset image iterative defogging mechanism to obtain an iteration output of each round; when it is detected that the haze-free image corresponding to the iteration output meets a preset definition requirement, obtaining a target haze-free image.

2. The method of claim 1, wherein, The image restoration parameters comprise anti-image restoration parameters and haze-free image restoration parameters; the performing multiple non-linear curve mapping restoration on the foggy image based on the image restoration parameters to obtain a preliminary haze-free image comprises: performing non-linear enhancement processing on the foggy image based on the anti-image restoration parameters to obtain a processed image; performing curve mapping restoration on the processed image based on the haze-free image restoration parameters to obtain a preliminary haze-free image.

3. The method of claim 2, wherein, The performing non-linear enhancement processing on the foggy image based on the anti-image restoration parameters to obtain a processed image comprises: performing pixel normalization processing on the foggy image to obtain a normalized image, and the pixel value of the normalized image is within a preset range; performing pixel inversion on the normalized image to obtain an anti-image; performing curve mapping processing on the anti-image based on the anti-image restoration parameters to obtain a processed image, so as to realize non-linear enhancement of image brightness.

4. The method of claim 2, wherein, The performing curve mapping restoration on the processed image based on the haze-free image restoration parameters to obtain a preliminary haze-free image comprises: performing curve mapping restoration on the processed image based on the haze-free image restoration parameters to obtain a restored image; performing pixel inversion on the restored image to obtain a preliminary haze-free image.

5. The method of claim 1, wherein, The image restoration parameters comprise anti-image restoration parameters and haze-free image restoration parameters; The performing iterative processing on the preliminary haze-free image based on the image restoration parameters through a preset image iterative defogging mechanism to obtain an iteration output of each round comprises: In each round of iteration, performing non-linear transformation on the preliminary haze-free image or the iteration output of the previous round based on the anti-image restoration parameters through a preset image iterative defogging mechanism to generate an intermediate image; performing curve mapping restoration and pixel inversion on the intermediate image based on the haze-free image restoration parameters to obtain a haze-free image as the iteration output of each round.

6. The method of claim 1, wherein, The depth curve estimation network comprises a residual SwinTransformer module and a large-core convolution attention module; the obtaining image restoration parameters based on the foggy image through a preset depth curve estimation network comprises: obtaining shallow features of the foggy image through convolution operation; obtaining global semantic features of the foggy image by performing feature extraction on the shallow features through the residual SwinTransformer block; obtaining texture detail features of the foggy image by performing feature extraction on the shallow features through the large-core convolution attention module; performing fusion processing on the global semantic features and the texture detail features to obtain image restoration parameters.

7. The method of claim 6, wherein, The method further comprises: performing feature enhancement processing on the image restoration parameter through a preset local feature extraction module cascade structure to obtain an enhanced parameter; the cascade structure comprises convolution of two layers at the head and tail and depth separable convolution and depth separable dilated convolution of two layers in the middle; obtaining a spatial feature tensor corresponding to an attention mechanism structure of the local feature extraction module of the enhanced parameter; obtaining a corresponding excitation tensor based on the spatial feature tensor through the attention mechanism structure; performing fusion processing on the excitation tensor and the enhanced parameter to obtain an image restoration parameter after adaptive correction.

8. An image processing apparatus based on curve restoration, characterized by, Comprise: An acquisition unit configured to acquire a foggy image to be processed; A parameter unit configured to obtain an image restoration parameter based on the foggy image through a preset depth curve estimation network; A restoration unit configured to perform multi-nonlinear curve mapping restoration on the foggy image based on the image restoration parameter to obtain a preliminary haze-free image; An iteration unit configured to perform iteration processing on the preliminary haze-free image based on the image restoration parameter through a preset image iteration defogging mechanism to obtain an iteration output of each round; A recursion unit configured to obtain a target haze-free image when it is detected that a haze-free image corresponding to the iteration output meets a preset definition requirement.

9. An electronic device, comprising: Comprise: One or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the image processing method based on curve restoration according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer readable instruction is stored thereon, when the computer readable instruction is executed by the processor of the computer, the computer executes the image processing method based on curve restoration according to any one of claims 1 to 7.

Images