A barrel distortion image adaptive correction method and related device

Abstract

The present application provides a barrel distortion image adaptive correction method and related equipment, comprising: obtaining a distortion image to be corrected; constructing a barrel distortion correction network; inputting the distortion image into the barrel distortion correction network for adaptive correction to generate a corrected image; the barrel distortion correction network comprises an encoder connected in turn for feature extraction and feature compression of the distortion image, and a decoder for generating a pixel stream based on a distortion coefficient tensor, and performing resampling processing on the distortion image through the pixel stream to generate a corrected image; the method has the characteristics of simple operation, lightweight network structure, accurate correction, flexible migration use scene, good robustness, and can be widely applied to infrared thermal imaging cameras to correct the barrel distortion caused by wide-angle lens shooting, improve the accuracy of electrolytic cell plate short circuit / open circuit fault detection in the wet metallurgical electrolysis process, and provide accurate perception basis for intelligent optimization control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image distortion correction, in particular to a barrel-shaped distortion image adaptive correction method and related equipment. BACKGROUND

[0002] At present, due to the wide angle lens has the points of large imaging angle, wide range, therefore in the hydrometallurgy electrolysis process, is widely used in infrared thermal imaging camera to detect the electrolytic cell plate short circuit / open circuit fault. However, the physical structure of the wide angle lens determines that the magnification of the edge of the field of view is lower than the center part, forming a barrel-shaped distortion. The existence of barrel-shaped distortion causes the distortion of the image, reduces the accuracy of target detection and semantic recognition, so it is necessary to correct these infrared images with barrel-shaped distortion and reduce the influence on imaging. However, the existing barrel-shaped distortion correction method of infrared image has strict use conditions, depends on template calibration and specific texture detection, and it is difficult to realize adaptive correction in complex industrial environment. Therefore, how to solve the problem that the existing barrel-shaped distortion correction method of infrared image is limited by the use scene, has poor robustness and flexibility, and is difficult to realize adaptive correction is one of the key technical difficulties to improve the accuracy of electrolytic cell plate short circuit / open circuit fault detection in the hydrometallurgy electrolysis process, and to provide accurate perception basis for intelligent optimization control of electrolytic workshop.

[0003] The method of image distortion correction is mainly divided into two categories, which are optical method and digital image processing method. The optical method corrects distortion mainly by using a complex optical imaging system to make the photographed image close to the real object in the imaging process. However, the design and manufacturing process of the complex optical imaging system required in this method is difficult and costly, so the current distortion correction system generally does not adopt this method.

[0004] The method of using digital image processing technology to realize image distortion correction mainly uses correction algorithm to further correct the formed image after the real object has been imaged, so that the formed image is close to the real object. This method does not require strict optical process compared with the optical processing method, and has lower cost. Moreover, the adaptability of the method in various environments and under different requirements can be enhanced according to the improvement of the algorithm, so this method is now widely used for barrel-shaped distortion research of images.

[0005] In the barrel distortion correction based on the digital image processing technology, the most common method is based on template calibration, which needs to make a known type of template, then take a distorted camera to get a distorted image, and compare it with the non-distorted image of the template to obtain the mapping relationship between the GCP coordinates of the two images, so as to calculate the distortion parameters of the image and realize the distortion correction. However, this method needs to manually calibrate the control point pair, which is not only laborious but also inaccurate in positioning the control point coordinates. Therefore, an automatic calibration method of the control point pair is proposed. However, the template calibration method needs to sample the template, and only the images taken in the specific environment during the sampling of the template have significance for correction, which lacks flexibility and has a great discount in the complex industrial environment.

[0006] The method based on specific texture detection can estimate the barrel distortion parameters by detecting the texture features in the photographed object, such as vanishing points, straight lines, circles, and texts. However, this requires corresponding texture features in the image, and relies on accurate line detection or circle detection algorithms, which greatly limits the universality of the method. In the electrolytic production site, the texture features may not be detected due to the shielding of acid mist, resulting in algorithm failure. SUMMARY

[0007] The present application provides a barrel distortion image adaptive correction method and related equipment, which aims to realize the adaptive correction of infrared image barrel distortion and improve the accuracy of electrolytic cell plate short circuit / open circuit fault detection.

[0008] To achieve the above purpose, the present application provides a barrel distortion image adaptive correction method, which comprises:

[0009] Step 1: obtaining a distortion image to be corrected;

[0010] Step 2: constructing a barrel distortion correction network;

[0011] Step 3: inputting the distortion image into the barrel distortion correction network for adaptive correction to generate a corrected image;

[0012] The barrel distortion correction network comprises an encoder and a decoder connected in sequence; the encoder performs feature extraction and feature compression on the distortion image to obtain a distortion coefficient tensor, and outputs the distortion coefficient tensor to the decoder; the decoder generates a pixel stream based on the distortion coefficient tensor, and performs resampling processing on the distortion image through the pixel stream to generate a corrected image.

[0013] Further, before step 3, it further comprises:

[0014] preprocessing the distortion image to be corrected;

[0015] The preprocessed distorted image is input into a barrel distortion correction network for adaptive correction to generate a corrected image.

[0016] Further, the encoder comprises:

[0017] The first feature extraction layer, the second feature extraction layer, the third feature extraction layer, the fourth feature extraction layer, the first feature compression layer, the second feature compression layer, the third feature compression layer, the fourth feature compression layer, the adaptive pooling layer, the CBAM attention layer, the first full connection layer, and the second full connection layer;

[0018] The output end of the first feature extraction layer is connected with the input end of the adaptive pooling layer, the output end of the adaptive pooling layer is connected with the input end of the first feature compression layer, the output end of the first feature compression layer is connected with the input end of the CBAM attention layer, the output end of the CBAM attention layer is connected with the input end of the second feature extraction layer, the output end of the second feature extraction layer is connected with the output end of the CBAM attention layer and the input end of the second feature compression layer respectively, the output end of the second feature compression layer is connected with the input end of the third feature extraction layer, the output end of the third feature extraction layer is connected with the output end of the second feature compression layer and the input end of the third feature compression layer respectively, the output end of the third feature compression layer is connected with the input end of the fourth feature extraction layer, the output end of the fourth feature extraction layer is connected with the output end of the third feature compression layer and the input end of the fourth feature compression layer respectively, the output end of the fourth feature compression layer is connected with the input end of the first full connection layer, the output end of the first full connection layer is connected with the input end of the second full connection layer, and the output end of the second full connection layer is connected with the input end of the decoder.

[0019] Further, the decoder comprises:

[0020] The pixel stream generator and the resampling module;

[0021] The input end of the pixel stream generator is connected with the output end of the second full connection layer, and the output end of the pixel stream generator is connected with the input end of the resampling module.

[0022] Further, after the preprocessed distorted image is input into the first feature extraction layer, the adaptive pooling layer, the first feature compression layer, the CBAM attention layer, the second feature extraction layer, the second feature compression layer, the third feature extraction layer, the third feature compression layer, the fourth feature extraction layer, the fourth feature compression layer, the first full connection layer, and the second full connection layer are used for feature extraction and feature compression to obtain a distortion coefficient tensor.

[0023] The distortion coefficient tensor is input into the pixel stream generator to generate a pixel stream, and the pixel stream is input into the resampling module for resampling processing to obtain a corrected image.

[0024] Further, the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer each comprises:

[0025] the first two-dimensional convolution structure, the second two-dimensional convolution structure, the third two-dimensional convolution structure, the fourth two-dimensional convolution structure, the fifth two-dimensional convolution structure, the sixth two-dimensional convolution structure, the seventh two-dimensional convolution structure, the eighth two-dimensional convolution structure, the ninth two-dimensional convolution structure and the first splicing layer;

[0026] the output end of the first two-dimensional convolution structure is connected with the input end of the fifth two-dimensional convolution structure, the output end of the fifth two-dimensional convolution structure is connected with the input end of the eighth two-dimensional convolution structure, and the output end of the eighth two-dimensional convolution structure is connected with the input end of the first splicing layer;

[0027] the output end of the second two-dimensional convolution structure is connected with the input end of the sixth two-dimensional convolution structure, the output end of the sixth two-dimensional convolution structure is connected with the input end of the ninth two-dimensional convolution structure, and the output end of the ninth two-dimensional convolution structure is connected with the input end of the first splicing layer;

[0028] the output end of the third two-dimensional convolution structure is connected with the input end of the seventh two-dimensional convolution structure, and the output end of the seventh two-dimensional convolution structure is connected with the input end of the first splicing layer;

[0029] the output end of the fourth two-dimensional convolution structure is connected with the input end of the first splicing layer;

[0030] the input end of the first two-dimensional convolution structure of the second feature extraction layer, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, the input end of the fourth two-dimensional convolution structure are each connected with the output end of the CBAM attention layer, and the output end of the first splicing layer of the second feature extraction layer is connected with the input end of the second feature compression layer;

[0031] the input end of the first two-dimensional convolution structure of the third feature extraction layer, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, the input end of the fourth two-dimensional convolution structure are each connected with the output end of the second feature layer, and the output end of the first splicing layer of the third feature extraction layer is connected with the input end of the third feature compression layer;

[0032] the input end of the first two-dimensional convolution structure of the fourth feature extraction layer, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, the input end of the fourth two-dimensional convolution structure are each connected with the output end of the second feature layer, and the output end of the first splicing layer of the fourth feature extraction layer is connected with the input end of the fourth feature compression layer.

[0033] Further, the first feature compression layer, the second feature compression layer, the third feature compression layer and the fourth feature compression layer each comprises:

[0034] the twelfth two-dimensional convolutional structure, the eleventh two-dimensional convolutional structure, the tenth two-dimensional convolutional structure, the thirteenth two-dimensional convolutional structure, the fourteenth two-dimensional convolutional structure, the fifteenth two-dimensional convolutional structure, the sixteenth two-dimensional convolutional structure, the seventeenth two-dimensional convolutional structure, the eighteenth two-dimensional convolutional structure, the first two-dimensional max-pooling layer, the second two-dimensional max-pooling layer, and the second concatenation layer;

[0035] An output end of the twelfth two-dimensional convolutional structure is connected with an input end of the thirteenth two-dimensional convolutional structure, an output end of the thirteenth two-dimensional convolutional structure is connected with an input end of the seventeenth two-dimensional convolutional structure, and an output end of the seventeenth two-dimensional convolutional structure is connected with an input end of the second concatenation layer;

[0036] An output end of the eleventh two-dimensional convolutional structure is connected with an input end of the fourteenth two-dimensional convolutional structure, an output end of the fourteenth two-dimensional convolutional structure is connected with an input end of the eighteenth two-dimensional convolutional structure, and an output end of the eighteenth two-dimensional convolutional structure is connected with an input end of the second concatenation layer;

[0037] An output end of the tenth two-dimensional convolutional structure is connected with an input end of the fifteenth two-dimensional convolutional structure, an output end of the fifteenth two-dimensional convolutional structure is connected with an input end of the first two-dimensional max-pooling layer, and an output end of the first two-dimensional max-pooling layer is connected with an input end of the second concatenation layer;

[0038] An output end of the thirteenth two-dimensional convolutional structure is connected with an input end of the second two-dimensional max-pooling layer, and an output end of the second two-dimensional max-pooling layer is connected with an input end of the second concatenation layer;

[0039] An input end of the twelfth two-dimensional convolutional structure, an input end of the eleventh two-dimensional convolutional structure, an input end of the tenth two-dimensional convolutional structure, and an input end of the thirteenth two-dimensional convolutional structure of the first feature compression layer are all connected with an input end of the adaptive pooling layer, and an output end of the second concatenation layer of the first feature compression layer is connected with an input end of the CBAM attention layer;

[0040] An input end of the twelfth two-dimensional convolutional structure, an input end of the eleventh two-dimensional convolutional structure, an input end of the tenth two-dimensional convolutional structure, and an input end of the thirteenth two-dimensional convolutional structure of the second feature compression layer are all connected with an output end of the first concatenation layer of the second feature extraction layer, and an output end of the second concatenation layer of the second feature compression layer is connected with an input end of the first two-dimensional convolutional structure, an input end of the second two-dimensional convolutional structure, an input end of the third two-dimensional convolutional structure, and an input end of the fourth two-dimensional convolutional structure of the third feature extraction layer;

[0041] The input end of the twelfth two-dimensional convolution structure of the third feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure, and the input end of the thirteenth two-dimensional convolution structure are all connected with the first splicing layer of the third feature extraction layer, and the output end of the second splicing layer of the third feature compression layer is connected with the input end of the first two-dimensional convolution structure, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, and the input end of the fourth two-dimensional convolution structure of the fourth feature extraction layer.

[0042] The input end of the twelfth two-dimensional convolution structure of the fourth feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure, and the input end of the thirteenth two-dimensional convolution structure are all connected with the first splicing layer of the fourth feature extraction layer, and the output end of the second splicing layer of the fourth feature compression layer is connected with the input end of the first full connection layer.

[0043] The application further provides a barrel-shaped distortion image adaptive correction device, which comprises:

[0044] An acquisition module is configured to acquire a distortion image to be corrected.

[0045] A construction module is configured to construct a barrel-shaped distortion correction network.

[0046] A correction module is configured to input the distortion image into the barrel-shaped distortion correction network for adaptive correction to generate a corrected image.

[0047] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the barrel-shaped distortion image adaptive correction method.

[0048] The application further provides a barrel-shaped distortion image adaptive correction device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the barrel-shaped distortion image adaptive correction method when executing the computer program.

[0049] The above-mentioned scheme of the application has the following advantages:

[0050] The application obtains a distortion image to be corrected, extracts and compresses features of the distortion image through an encoder in a barrel distortion correction network constructed, obtains a distortion coefficient tensor, and outputs the distortion coefficient tensor to a decoder, the decoder generates a pixel stream based on the distortion coefficient tensor, and performs resampling processing on the distortion image through the pixel stream to generate a corrected image; the mapping relationship between the distortion image and the non-distortion image is learned through the construction of the barrel distortion correction network, the distortion image is input into the barrel distortion correction network to estimate distortion parameters by extracting global feature information of the distortion image to generate a pixel stream, and adaptive correction is realized by resampling pixels according to the pixel stream; the problems that the prior art is limited by a use scene, has poor robustness and flexibility, and is difficult to realize adaptive correction are solved, the application has the characteristics of simple operation, light network structure, accurate correction, flexible migration of use scenes, and good robustness, and can be widely applied to an infrared thermal imaging camera to correct barrel distortion caused by a wide-angle lens, improve the accuracy of electrolytic cell plate short circuit / open circuit fault detection in a hydrometallurgical electrolysis process, and provide a precise perception basis for intelligent optimization control.

[0051] Other benefits of the present application will be described in detail in the subsequent specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 The flowchart of the embodiment of the present application is shown in the figure;

[0053] Figure 2 The structure diagram of the barrel distortion correction network in the embodiment of the present application is shown in the figure;

[0054] Figure 3 The structure diagram of the first feature extraction layer in the barrel distortion correction network in the embodiment of the present application is shown in the figure;

[0055] Figure 4 The structure diagram of the first feature compression layer in the barrel distortion correction network in the embodiment of the present application is shown in the figure;

[0056] Figure 5 The effect diagram of correcting a multi-scene ordinary barrel distortion image through the embodiment of the present application is shown in the figure;

[0057] Figure 6 The effect diagram of correcting an infrared barrel distortion image through the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0058] To make the technical problems, technical solutions and advantages of the present application clearer, specific embodiments will be described in detail below with reference to the drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0059] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0060] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be a locking connection, or a detachable connection, or an integral connection; it can be a mechanical connection, or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0061] In addition, the technical features involved in the different embodiments of the application described below can be combined with each other as long as there is no conflict.

[0062] The present application aims at the existing problems, and provides a barrel-shaped distortion image adaptive correction method and related equipment.

[0063] As shown in Figure 1 The embodiment of the present application provides a barrel-shaped distortion image adaptive correction method, which comprises the following steps:

[0064] Step 1, obtaining a distortion image to be corrected;

[0065] Step 2, constructing a barrel-shaped distortion correction network;

[0066] Step 3, inputting the distortion image into the barrel-shaped distortion correction network for adaptive correction to generate a corrected image;

[0067] The barrel-shaped distortion correction network comprises an encoder and a decoder connected in sequence; the encoder performs feature extraction and feature compression on the distortion image to obtain a distortion coefficient tensor, and outputs the distortion coefficient tensor to the decoder; the decoder generates a pixel stream based on the distortion coefficient tensor, and performs resampling processing on the distortion image through the pixel stream to generate a corrected image.

[0068] Specifically, before step 3, it further comprises:

[0069] The preprocessed distorted image is treated, and a distorted image with a size of 1024*1280*3 is obtained;

[0070] The preprocessed distorted image is input into a barrel-shaped distortion correction network for adaptive correction, and a corrected image is generated.

[0071] In the embodiment of the application, the barrel-shaped distortion correction network is trained by using multi-scene ordinary images and infrared images, and the training is as follows:

[0072] In the embodiment of the application, a multi-scene ordinary image with a size of 1024*1280*3 is obtained from a Place365-CNNs public picture data set, a distortion coefficient β is randomly generated, and a multi-scene ordinary non-distorted image is inversely distorted according to a standard radial distortion model, and the model is as follows:

[0073]

[0074] wherein (x d , y d ) is a pixel point coordinate after distortion, (x u , y u ) is a pixel point coordinate before distortion (after correction), (x c , y c ) is the center of a camera distortion model, usually the center of an image, r is the Euclidean distance of a pixel point from the center (x c , y c ) of the distortion model, and L(r) is a magnification function.

[0075]

[0076] L(r) = 1 + k1r 2 +k2r 4 +…+k n r 2n

[0077] wherein the set k = (k1, k2, …, k n-1 , k n ) is a distortion coefficient, and in the embodiment of the application, only the first distortion coefficient k1 is considered for the purpose of reducing the calculation cost.

[0078] The displacement of the pixels before and after distortion is calculated according to the following formula, and a pixel flow matrix F(u, v) is generated.

[0079]

[0080] The distorted image is taken as the data of a distorted data set, and the pixel flow matrix F(u, v) is taken as the data label, and a multi-scene ordinary image distortion data set is prepared.

[0081] Infrared images are acquired from an infrared thermal imaging camera used in practical applications for electrode short-circuit / open-circuit detection. The above steps are repeated to create an infrared image distortion dataset.

[0082] Specifically, such as Figure 2 As shown, the encoder includes: a first feature extraction layer with dimensions of 1024×1280×24, a second feature extraction layer with dimensions of 128×128×64, a third feature extraction layer with dimensions of 32×32×128, a fourth feature extraction layer with dimensions of 8×8×256, a first feature compression layer with dimensions of 128×128×64, a second feature compression layer with dimensions of 32×32×128, a third feature compression layer with dimensions of 8×8×256, a fourth feature compression layer with dimensions of 2×2×256, an adaptive pooling layer with dimensions of 512×512×24, a CBAM attention layer with dimensions of 128×128×64, a first fully connected layer with dimensions of 2048, and a second fully connected layer with dimensions of 512.

[0083] The output of the first feature extraction layer is connected to the input of the adaptive pooling layer. The output of the adaptive pooling layer is connected to the input of the first feature compression layer. The output of the first feature compression layer is connected to the input of the CBAM attention layer. The output of the CBAM attention layer is connected to the input of the second feature extraction layer. The output of the second feature extraction layer is connected to both the output of the CBAM attention layer and the input of the second feature compression layer. The output of the second feature compression layer is connected to the input of the third feature extraction layer. The output of the third feature extraction layer is connected to both the output of the second feature compression layer and the input of the third feature compression layer. The output of the third feature compression layer is connected to the input of the fourth feature extraction layer. The output of the fourth feature extraction layer is connected to both the output of the third feature compression layer and the input of the fourth feature compression layer. The output of the fourth feature compression layer is connected to the input of the first fully connected layer. The output of the first fully connected layer is connected to the input of the second fully connected layer. The output of the second fully connected layer is connected to the input of the decoder.

[0084] In this embodiment of the invention, the output of the second feature extraction layer and the output of the CBAM attention layer stack the parameters four times; the output of the third feature extraction layer and the output of the second feature compression layer stack the parameters four times; the output of the fourth feature extraction layer and the output of the third feature compression layer stack the parameters four times, thereby realizing a parameter sharing mechanism. This deepens the network without increasing the number of network parameters, effectively reducing the total number of network parameters and making the correction network easier to deploy and apply.

[0085] Specifically, the decoder includes: a pixel stream generator and a resampling module;

[0086] The input end of the pixel flow generator is connected with the output end of the second full connection layer, and the output end of the pixel flow generator is connected with the input end of the resampling module.

[0087] Specifically, the distorted image obtained after preprocessing is input into the first feature extraction layer, and then feature extraction and feature compression are performed through the adaptive pooling layer, the first feature compression layer, the CBAM attention layer, the second feature extraction layer, the second feature compression layer, the third feature extraction layer, the third feature compression layer, the fourth feature extraction layer, the fourth feature compression layer, the first full connection layer and the second full connection layer, to obtain the distortion coefficient tensor β.

[0088] The distortion coefficient tensor β is input into the pixel flow generator to generate a 1024x1280x2 pixel flow F as follows:

[0089]

[0090] The pixel flow F is input into the resampling module for resampling processing to obtain a corrected image.

[0091] Specifically, as shown in Figure 3 The first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer have the same structure, and each includes: a first two-dimensional convolution structure of 1x1, a second two-dimensional convolution structure of 1x1, a third two-dimensional convolution structure of 1x1, a fourth two-dimensional convolution structure of 1x1, a fifth two-dimensional convolution structure of 1x5, a sixth two-dimensional convolution structure of 1x3, a seventh two-dimensional convolution structure of 3x3, an eighth two-dimensional convolution structure of 5x1, a ninth two-dimensional convolution structure of 3x1, and a first splicing layer.

[0092] The output end of the first two-dimensional convolution structure is connected with the input end of the fifth two-dimensional convolution structure, the output end of the fifth two-dimensional convolution structure is connected with the input end of the eighth two-dimensional convolution structure, and the output end of the eighth two-dimensional convolution structure is connected with the input end of the first splicing layer.

[0093] The output end of the second two-dimensional convolution structure is connected with the input end of the sixth two-dimensional convolution structure, the output end of the sixth two-dimensional convolution structure is connected with the input end of the ninth two-dimensional convolution structure, and the output end of the ninth two-dimensional convolution structure is connected with the input end of the first splicing layer.

[0094] The output end of the third two-dimensional convolution structure is connected with the input end of the seventh two-dimensional convolution structure, and the output end of the seventh two-dimensional convolution structure is connected with the input end of the first splicing layer.

[0095] The output end of the fourth two-dimensional convolution structure is connected with the input end of the first splicing layer.

[0096] The input end of the first two-dimensional convolution structure, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, and the input end of the fourth two-dimensional convolution structure of the second feature extraction layer are connected with the output end of the CBAM attention layer, and the output end of the first splicing layer of the second feature extraction layer is connected with the input end of the second feature compression layer.

[0097] The input end of the first two-dimensional convolution structure, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, and the input end of the fourth two-dimensional convolution structure of the third feature extraction layer are connected with the output end of the second feature layer, and the output end of the first splicing layer of the third feature extraction layer is connected with the input end of the third feature compression layer.

[0098] The input end of the first two-dimensional convolution structure, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, and the input end of the fourth two-dimensional convolution structure of the fourth feature extraction layer are connected with the output end of the second feature layer, and the output end of the first splicing layer of the fourth feature extraction layer is connected with the input end of the fourth feature compression layer.

[0099] The first feature extraction layer, the second feature extraction layer, the third feature extraction layer, and the fourth feature extraction layer use different convolution kernels on multiple branches, and finally the feature maps of each branch are spliced together. The first two-dimensional convolution structure, the second two-dimensional convolution structure, the third two-dimensional convolution structure, the fourth two-dimensional convolution structure, the fifth two-dimensional convolution structure, the sixth two-dimensional convolution structure, the seventh two-dimensional convolution structure, the eighth two-dimensional convolution structure, and the ninth two-dimensional convolution structure are mainly composed of a convolution layer, a batch normalization layer, and a ReLU activation function, and the step is 1. Among them, the padding of the seventh two-dimensional convolution structure is 1, the padding of the sixth two-dimensional convolution structure is (0, 1), the padding of the ninth two-dimensional convolution structure is (1, 0), the padding of the fifth two-dimensional convolution structure is (0, 2), and the padding of the eighth two-dimensional convolution structure is (2, 0).

[0100] Specifically, as shown in FIG. 1, the structure of the first feature compression layer is shown. Figure 4 The structure of the first feature compression layer, the second feature compression layer, the third feature compression layer, and the fourth feature compression layer is consistent and includes a twelfth two-dimensional convolution structure of 1x1, an eleventh two-dimensional convolution structure of 1x1, a tenth two-dimensional convolution structure of 1x1, a thirteenth two-dimensional convolution structure of 1x1, a fourteenth two-dimensional convolution structure of 1x5, a fifteenth two-dimensional convolution structure of 1x3, a sixteenth two-dimensional convolution structure of 3x3, a seventeenth two-dimensional convolution structure of 5x1, an eighteenth two-dimensional convolution structure of 3x1, a first two-dimensional max pooling layer, a second two-dimensional max pooling layer, and a second splicing layer.

[0101] Different from the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer is that: 4x4 first two-dimensional maximum pooling layer and 2x2 second two-dimensional maximum pooling layer are used on the branch respectively, wherein the step length of the 1x1 twelfth two-dimensional convolution structure, the eleventh two-dimensional convolution structure, the tenth two-dimensional convolution structure and the thirteenth two-dimensional convolution structure is 1, and the step length of the remaining convolution is 2.

[0102] The output end of the twelfth two-dimensional convolution structure is connected with the input end of the thirteenth two-dimensional convolution structure, the output end of the thirteenth two-dimensional convolution structure is connected with the input end of the seventeenth two-dimensional convolution structure, and the output end of the seventeenth two-dimensional convolution structure is connected with the input end of the second splicing layer.

[0103] The output end of the eleventh two-dimensional convolution structure is connected with the input end of the fourteenth two-dimensional convolution structure, the output end of the fourteenth two-dimensional convolution structure is connected with the input end of the eighteenth two-dimensional convolution structure, and the output end of the eighteenth two-dimensional convolution structure is connected with the input end of the second splicing layer.

[0104] The output end of the tenth two-dimensional convolution structure is connected with the input end of the fifteenth two-dimensional convolution structure, the output end of the fifteenth two-dimensional convolution structure is connected with the input end of the first two-dimensional maximum pooling layer, and the output end of the first two-dimensional maximum pooling layer is connected with the input end of the second splicing layer.

[0105] The output end of the thirteenth two-dimensional convolution structure is connected with the input end of the second two-dimensional maximum pooling layer, and the output end of the second two-dimensional maximum pooling layer is connected with the input end of the second splicing layer.

[0106] The input end of the twelfth two-dimensional convolution structure of the first feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure and the input end of the thirteenth two-dimensional convolution structure are all connected with the input end of the adaptive pooling layer, and the output end of the second splicing layer of the first feature compression layer is connected with the input end of the CBAM attention layer.

[0107] The input end of the twelfth two-dimensional convolution structure of the second feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure and the input end of the thirteenth two-dimensional convolution structure are all connected with the output end of the first splicing layer of the second feature extraction layer, and the output end of the second splicing layer of the second feature compression layer is connected with the input end of the first two-dimensional convolution structure, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure and the input end of the fourth two-dimensional convolution structure of the third feature extraction layer.

[0108] The input end of the twelfth two-dimensional convolution structure of the third feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure, and the input end of the thirteenth two-dimensional convolution structure are all connected with the first splicing layer of the third feature extraction layer, and the output end of the second splicing layer of the third feature compression layer is connected with the input end of the first two-dimensional convolution structure, the input end of the second two-dimensional convolution structure, the input end of the third two-dimensional convolution structure, and the input end of the fourth two-dimensional convolution structure of the fourth feature extraction layer.

[0109] The input end of the twelfth two-dimensional convolution structure of the fourth feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure, and the input end of the thirteenth two-dimensional convolution structure are all connected with the first splicing layer of the fourth feature extraction layer, and the output end of the second splicing layer of the fourth feature compression layer is connected with the input end of the first full connection layer.

[0110] The multi-scene ordinary image distortion data set is divided into a training set and a test set, the training set is used to train and update the barrel-shaped distortion correction network constructed, and a plurality of versions of neural network weights are obtained; the test set is used to test the plurality of versions of neural network weights, and the optimal neural network weight is selected.

[0111] In the embodiment of the application, the pixel flow generator generates a pixel flow F according to the distortion parameter β, a standard radial distortion model and a formula.

[0112] In the embodiment of the application, the resampling module adjusts the position of the pixel according to the formula based on the pixel flow and the distortion image generated by calculation, and resamples the pixel to correct the distortion image, and the calculation step is consistent with the step of generating the distortion data.

[0113] It needs to be explained that the calculation process of the pixel flow generator and the resampling module does not depend on the deep learning process, but only depends on the distortion model, and therefore the parameters therein do not have learnability.

[0114] In the embodiment of the application, the barrel-shaped distortion correction network prevents overfitting through an EPE loss function, the EPE loss function is modified by using an L2 regularization method, a zero-mean Gaussian prior distribution is set for the network parameters to constrain the network, the stability of the learning process is ensured, overfitting of the network is prevented, and the formula is as follows:

[0115] Loss=EPE+λ||Θ||

[0116] Wherein, λ is a penalty coefficient of the L2 regularization term, and ||·|| is an L2 norm of a matrix.

[0117] The embodiment of the present application trains the barrel distortion correction network through the input of multi-scene ordinary distorted images, extracts image information through an encoder, and gradually compresses to 1 distortion coefficient tensor input into a decoder, generates a pixel stream through a pixel stream generator and a distortion coefficient tensor, resamples the distorted image according to the pixel stream, and generates a corrected image, the purpose of which is to enable the barrel distortion correction network to have the ability to extract barrel distortion features of images in multiple scenes.

[0118] Due to the low resolution, single tone and low contrast of infrared images, which are quite different from ordinary images, the barrel distortion correction network is not ideal, the embodiment of the present application further freezes all structures in the barrel distortion correction network except the first full connection layer and the second full connection layer, and continues to train the network using the infrared image distortion dataset, and finely adjusts the network, the purpose of which is to keep the learned feature extraction structure of the barrel distortion correction network unchanged, not to be disturbed by scene migration, and to improve the fitting effect of the barrel distortion correction network in the infrared image scene, and the specific process is as follows:

[0119] The infrared image distortion dataset is divided into a training set and a test set, the training set is used for training, multiple iterations are updated, and multiple versions of neural network weights are obtained, the test set is used for testing, each version of neural network weight is quantitatively tested, and the globally optimal neural network weight is selected.

[0120] The barrel distortion correction result of the ordinary barrel distortion image through the embodiment of the present application is shown in Figure 5 , which can effectively and accurately correct the barrel distortion of images in different scenes and different distortion degrees (i.e., with different distortion coefficients), proving the effectiveness of the method in ordinary images.

[0121] The barrel distortion correction result of the infrared barrel distortion image through the embodiment of the present application is shown in Figure 6 , after the targeted fine tuning and optimization of the infrared image through transfer learning, the barrel distortion of the infrared image can be effectively and accurately corrected, proving the effectiveness of the method in infrared images.

[0122] The embodiment of the present application obtains a distortion image to be corrected, extracts and compresses features of the distortion image through an encoder in a barrel distortion correction network constructed, obtains a distortion coefficient tensor, and outputs the distortion coefficient tensor to a decoder, so that the decoder generates a pixel stream based on the distortion coefficient tensor, and performs resampling processing on the distortion image through the pixel stream to generate a corrected image; without calculating distortion parameters based on a pre-set checkerboard template or other shape templates, or estimating internal and external parameters of a camera model through a mapping relationship between pixel points between real textures and textures in an image, the mapping relationship between the distortion image and a non-distortion image is learned through the construction of the barrel distortion correction network, the distortion image is input into the barrel distortion correction network to estimate distortion parameters to generate a pixel stream through the extraction of global feature information of the distortion image, and adaptive correction is realized through resampling of pixel points according to the pixel stream; the problems that the prior art is limited by a use scene, has poor robustness and flexibility, and is difficult to realize adaptive correction are solved, the present application has the characteristics of simple operation, lightweight network structure, accurate correction, flexible use scene, and good robustness, through targeted fine tuning and optimization, the correction network can also achieve good correction effect in a difficult infrared image correction task, and can be widely applied to infrared thermal imaging cameras to correct barrel distortion caused by a wide-angle lens, improve the accuracy of electrolytic cell plate short circuit / open circuit fault detection in a hydrometallurgical electrolysis process, and provide a precise perception basis for intelligent optimization control.

[0123] The embodiment of the present application also provides a barrel distortion image adaptive correction device.

[0124] The acquisition module is used to acquire a distortion image to be corrected.

[0125] The construction module is used to construct a barrel distortion correction network.

[0126] The correction module is used to input the distortion image into the barrel distortion correction network for adaptive correction to generate a corrected image.

[0127] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the barrel distortion image adaptive correction method.

[0128] The embodiment of the present application also provides a barrel distortion image adaptive correction device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the barrel distortion image adaptive correction method.

[0129] The above is the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the principles described in the present application, can also be made several improvements and refinements, these improvements and refinements should also be considered the scope of protection of the present application.

Claims

1. A barrel distortion image adaptive correction method, characterized by, The method comprises the following steps: Step 1, obtaining a distortion image to be corrected; Step 2, constructing a barrel distortion correction network; Step 3, inputting the distortion image into the barrel distortion correction network for adaptive correction to generate a corrected image; The barrel distortion correction network comprises an encoder and a decoder connected in sequence; the encoder performs feature extraction and feature compression on the distortion image to obtain a distortion coefficient tensor, and outputs the distortion coefficient tensor to the decoder; the decoder generates a pixel stream based on the distortion coefficient tensor, and performs resampling processing on the distortion image through the pixel stream to generate a corrected image; The encoder comprises: a first feature extraction layer, a second feature extraction layer, a third feature extraction layer, a fourth feature extraction layer, a first feature compression layer, a second feature compression layer, a third feature compression layer, a fourth feature compression layer, an adaptive pooling layer, a CBAM attention layer, a first full connection layer and a second full connection layer; The output end of the first feature extraction layer is connected with the input end of the adaptive pooling layer, the output end of the adaptive pooling layer is connected with the input end of the first feature compression layer, the output end of the first feature compression layer is connected with the input end of the CBAM attention layer, the output end of the CBAM attention layer is connected with the input end of the second feature extraction layer, the output end of the second feature extraction layer is connected with the output end of the CBAM attention layer and the input end of the second feature compression layer respectively, the output end of the second feature compression layer is connected with the input end of the third feature extraction layer, the output end of the third feature extraction layer is connected with the output end of the second feature compression layer and the input end of the third feature compression layer respectively, the output end of the third feature compression layer is connected with the input end of the fourth feature extraction layer, the output end of the fourth feature extraction layer is connected with the output end of the third feature compression layer and the input end of the fourth feature compression layer respectively, the output end of the fourth feature compression layer is connected with the input end of the first full connection layer, the output end of the first full connection layer is connected with the input end of the second full connection layer, and the output end of the second full connection layer is connected with the input end of the decoder; The decoder comprises: a pixel stream generator and a resampling module; The input end of the pixel stream generator is connected with the output end of the second full connection layer, and the output end of the pixel stream generator is connected with the input end of the resampling module; After the preprocessed distortion image is input into the first feature extraction layer, feature extraction and feature compression are performed through the adaptive pooling layer, the first feature compression layer, the CBAM attention layer, the second feature extraction layer, the second feature compression layer, the third feature extraction layer, the third feature compression layer, the fourth feature extraction layer, the fourth feature compression layer, the first full connection layer and the second full connection layer to obtain a distortion coefficient tensor; The distortion coefficient tensor is input into the pixel stream generator to generate a pixel stream, and the pixel stream is input into the resampling module to perform resampling processing to obtain a corrected image.

2. The barrel distortion image self-adaptive rectification method according to claim 1, characterized in that, Before the step 3, the method further comprises the following steps: Preprocessing the to-be-corrected distorted image; inputting the preprocessed distorted image into the barrel distortion correction network for adaptive correction to generate a corrected image.

3. The barrel distortion image adaptive correction method according to claim 2, characterized in that, the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer each comprise: a first two-dimensional convolution structure, a second two-dimensional convolution structure, a third two-dimensional convolution structure, a fourth two-dimensional convolution structure, a fifth two-dimensional convolution structure, a sixth two-dimensional convolution structure, a seventh two-dimensional convolution structure, an eighth two-dimensional convolution structure, a ninth two-dimensional convolution structure and a first splicing layer; an output end of the first two-dimensional convolution structure is connected with an input end of the fifth two-dimensional convolution structure, an output end of the fifth two-dimensional convolution structure is connected with an input end of the eighth two-dimensional convolution structure, and an output end of the eighth two-dimensional convolution structure is connected with an input end of the first splicing layer; an output end of the second two-dimensional convolution structure is connected with an input end of the sixth two-dimensional convolution structure, an output end of the sixth two-dimensional convolution structure is connected with an input end of the ninth two-dimensional convolution structure, and an output end of the ninth two-dimensional convolution structure is connected with an input end of the first splicing layer; an output end of the third two-dimensional convolution structure is connected with an input end of the seventh two-dimensional convolution structure, and an output end of the seventh two-dimensional convolution structure is connected with an input end of the first splicing layer; an output end of the fourth two-dimensional convolution structure is connected with an input end of the first splicing layer; an input end of the first two-dimensional convolution structure, an input end of the second two-dimensional convolution structure, an input end of the third two-dimensional convolution structure and an input end of the fourth two-dimensional convolution structure of the second feature extraction layer are connected with an output end of the CBAM attention layer, and an output end of the first splicing layer of the second feature extraction layer is connected with an input end of the second feature compression layer; an input end of the first two-dimensional convolution structure, an input end of the second two-dimensional convolution structure, an input end of the third two-dimensional convolution structure and an input end of the fourth two-dimensional convolution structure of the third feature extraction layer are connected with an output end of the second feature layer, and an output end of the first splicing layer of the third feature extraction layer is connected with an input end of the third feature compression layer; an input end of the first two-dimensional convolution structure, an input end of the second two-dimensional convolution structure, an input end of the third two-dimensional convolution structure and an input end of the fourth two-dimensional convolution structure of the fourth feature extraction layer are connected with an output end of the second feature layer, and an output end of the first splicing layer of the fourth feature extraction layer is connected with an input end of the fourth feature compression layer.

4. The barrel distortion image adaptive correction method according to claim 3, characterized in that, the first feature compression layer, the second feature compression layer, the third feature compression layer and the fourth feature compression layer each comprise: the twelfth two-dimensional convolutional structure, the eleventh two-dimensional convolutional structure, the tenth two-dimensional convolutional structure, the thirteenth two-dimensional convolutional structure, the fourteenth two-dimensional convolutional structure, the fifteenth two-dimensional convolutional structure, the sixteenth two-dimensional convolutional structure, the seventeenth two-dimensional convolutional structure, the eighteenth two-dimensional convolutional structure, the first two-dimensional max-pooling layer, the second two-dimensional max-pooling layer, and the second concatenation layer; an output end of the twelfth two-dimensional convolutional structure is connected with an input end of the thirteenth two-dimensional convolutional structure, an output end of the thirteenth two-dimensional convolutional structure is connected with an input end of the seventeenth two-dimensional convolutional structure, and an output end of the seventeenth two-dimensional convolutional structure is connected with an input end of the second concatenation layer; an output end of the eleventh two-dimensional convolutional structure is connected with an input end of the fourteenth two-dimensional convolutional structure, an output end of the fourteenth two-dimensional convolutional structure is connected with an input end of the eighteenth two-dimensional convolutional structure, and an output end of the eighteenth two-dimensional convolutional structure is connected with an input end of the second concatenation layer; an output end of the tenth two-dimensional convolutional structure is connected with an input end of the fifteenth two-dimensional convolutional structure, an output end of the fifteenth two-dimensional convolutional structure is connected with an input end of the first two-dimensional max-pooling layer, and an output end of the first two-dimensional max-pooling layer is connected with an input end of the second concatenation layer; an output end of the thirteenth two-dimensional convolutional structure is connected with an input end of the second two-dimensional max-pooling layer, and an output end of the second two-dimensional max-pooling layer is connected with an input end of the second concatenation layer; an input end of the twelfth two-dimensional convolutional structure, an input end of the eleventh two-dimensional convolutional structure, an input end of the tenth two-dimensional convolutional structure, and an input end of the thirteenth two-dimensional convolutional structure of the first feature compression layer are all connected with an input end of the adaptive pooling layer, and an output end of the second concatenation layer of the first feature compression layer is connected with an input end of the CBAM attention layer; an input end of the twelfth two-dimensional convolutional structure, an input end of the eleventh two-dimensional convolutional structure, an input end of the tenth two-dimensional convolutional structure, and an input end of the thirteenth two-dimensional convolutional structure of the second feature compression layer are all connected with an output end of the first concatenation layer of the second feature extraction layer, and an output end of the second concatenation layer of the second feature compression layer is connected with an input end of the first two-dimensional convolutional structure, an input end of the second two-dimensional convolutional structure, an input end of the third two-dimensional convolutional structure, and an input end of the fourth two-dimensional convolutional structure of the third feature extraction layer; an input end of the twelfth two-dimensional convolutional structure, an input end of the eleventh two-dimensional convolutional structure, an input end of the tenth two-dimensional convolutional structure, and an input end of the thirteenth two-dimensional convolutional structure of the third feature compression layer are all connected with the first concatenation layer of the third feature extraction layer, and an output end of the second concatenation layer of the third feature compression layer is connected with an input end of the first two-dimensional convolutional structure, an input end of the second two-dimensional convolutional structure, an input end of the third two-dimensional convolutional structure, and an input end of the fourth two-dimensional convolutional structure of the fourth feature extraction layer; The input end of the twelfth two-dimensional convolution structure of the fourth feature compression layer, the input end of the eleventh two-dimensional convolution structure, the input end of the tenth two-dimensional convolution structure, and the input end of the thirteenth two-dimensional convolution structure are all connected with the first splicing layer of the fourth feature extraction layer, and the output end of the second splicing layer of the fourth feature compression layer is connected with the input end of the first full connection layer.

5. A barrel distortion image adaptive correction apparatus for performing the barrel distortion image adaptive correction method according to claim 1, characterized by Comprise: An acquisition module is used for acquiring a distortion image to be corrected. A construction module is used for constructing a barrel distortion correction network. A correction module is used for inputting the distortion image into the barrel distortion correction network for adaptive correction to generate a corrected image.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program is executed by the processor to implement the barrel distortion image adaptive correction method of any one of claims 1-4.

7. A barrel distortion image adaptive correction device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the barrel distortion image adaptive correction method of any one of claims 1-4.

Images