Visual fusion enhanced perception method for complex underground environment based on the visual tectum mechanism of rattlesnake

Abstract

The present invention discloses a visual fusion enhanced perception method for underground complex environment imitating the visual tectum mechanism of rattlesnakes, comprising the following steps: S1, respectively collecting visible light source images and infrared source images by visible light camera and infrared camera; S2, obtaining visible light restored images and infrared restored images by image restoration technology; S3, after registering and cropping enhanced images, image preprocessing is completed based on the center-surround antagonistic receptor field model of six bimodal cell mechanisms of bionic rattlesnakes to obtain four bionic antagonistic enhancer images; S4, classifying the image acquisition environment into harsh levels by environmental recognition technology, and performing image fusion according to different degrees of harshness; S5, enhancing and transforming the fused image by image enhancement technology to obtain the final image. The present invention can increase the effective information of low-quality images in complex and harsh scenes such as underground, and improve the monitoring capability of personnel, equipment and environment at the operation site.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a visual fusion enhanced perception method for complex underground environments that imitates the visual tectum mechanism of rattlesnakes. Background Art

[0002] With the rapid development of computer technology, visual perception technology has been widely used in monitoring personnel and equipment in underground excavation, comprehensive mining, transportation and lifting. Images are the most basic carriers of visual information. Visible light images have rich scene texture details, high clarity and contrast, and good visual effects, but are affected by occlusion and have poor imaging at night; infrared images have prominent targets and high pixel values, strong penetration and can detect hidden targets, but the texture information is fuzzy. However, since the underground working environment is usually very harsh, the images obtained often have problems such as low contrast and blurred targets, which seriously affect the accuracy and efficiency of high-level visual tasks such as target detection and pedestrian recognition, and thus lead to equipment failure or untimely personnel safety monitoring.

[0003] By extracting and fusing the information of visible light and infrared images, a more accurate, comprehensive and reliable image description of the same scene or target can be obtained, which can better meet the requirements of upper-level visual tasks. The bionic visual mechanism in nature has the potential to solve the above problems and meet the relevant perception requirements. At present, some image fusion algorithms have emerged, including some fusion algorithms of bionic rattlesnake imaging mechanisms, which achieve image fusion by imitating the dual-mode cell mechanism that can fuse infrared and visible light information. However, the existing bionic rattlesnake fusion algorithm cannot fully utilize the dual-source information, the fused image quality is poor, the contrast is low, and the fusion strategy cannot be adjusted according to the image acquisition environment, so it cannot meet actual needs. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a visual fusion enhanced perception method for complex underground environments that imitates the visual tectonic mechanism of a rattlesnake. By performing non-weighted fusion and transformation enhancement on images, effective fusion of visible light and infrared information is achieved, so that it can overcome the influence of complex environmental factors such as low illumination, high dust, and high water vapor underground, thereby achieving effective monitoring of personnel, equipment and environment at the work site.

[0005] Technical solution: A visual fusion enhanced perception method for underground complex environment imitating the visual tectum mechanism of rattlesnakes, comprising the following steps:

[0006] S1, collect visible light source images and infrared source images through visible light camera and infrared camera respectively;

[0007] S2, obtaining a visible light restored image and an infrared restored image through image restoration technology;

[0008] S3, after registering and cropping the enhanced image, the image preprocessing was completed based on the center-surround antagonistic receptor field model of six bimodal cell mechanisms of bionic rattlesnakes to obtain four bionic antagonistic enhancer images;

[0009] S4, classify the image acquisition environment into different levels of severity through environmental recognition technology, and perform image fusion according to the severity;

[0010] S5, enhancing and transforming the fused image through image enhancement technology to obtain the final image.

[0011] Furthermore, in step S2, it is assumed that the pixel points meet the statistical standard of Poisson distribution, and based on Bayesian theory, the deconvolution iterative deduction calculation is performed on the degraded image through maximum likelihood estimation, and the iterative formula is as follows:

[0012]

[0013] In the formula, F n+1 (i, j) is the image of the n+1th iteration; F n (i, j) is the image of the nth iteration; G(i, j) is the input image; H(i, j) is the point spread function of the system; H(-i, -j) is the spatial flip of H(i, j).

[0014] Further, in step S3, the center-surround antagonistic receptor field model of the six bimodal cell mechanisms of the bionic rattlesnake is divided into an ON center antagonistic model and an OFF center antagonistic model, and their mathematical models are respectively expressed as follows:

[0015] ON center antagonism model:

[0016]

[0017] OFF center antagonism model:

[0018] f OFF (m,n)=f center (m,n)·log(f surround (m,n)+1)

[0019] Among them, (m, n) is the pixel coordinate, f center (m,n) is the image input to the central area, f surround (m,n) is the image input to the edge surround area, f ON (m,n) is the output image of the ON center antagonistic model, f OFF (m,n) is the output image of the OFF center antagonistic model.

[0020] Further, in step S3, the four bionic antagonistic enhancer images include inputting the visible light restored image into the ON center antagonistic model fcenter (m,n) and f surround (m,n) The generated VLI_ON image; the visible light restored image is input into the OFF center antagonism model f center (m,n) and f surround (m,n) generated VLI_OFF image; input the infrared restored image into the ON center antagonism model f center (m,n) and f surround (m,n) IRI_ON image generated; input the infrared restored image into the OFF center antagonistic model f center (m,n) and f surround (m,n) The generated IRI_OFF image.

[0021] Further, in step S4, the image sampling environment is divided based on the difference in infrared image energy between the normal environment and the harsh environment, and the division process is as follows:

[0022] S41, slide the 3×3 window and record the window pixel energy E each time it moves W (p,q), the formula is as follows:

[0023]

[0024] S42, calculate the total energy E of the entire infrared image, the formula is as follows:

[0025]

[0026] S43, calculating the global energy average value E of the infrared image moving window, the formula is as follows:

[0027]

[0028] S44, setting the discrimination coefficient k to divide the normal environment area S of the infrared image normal and harsh environment areas S harsh :

[0029]

[0030] S45, calculate the proportion of harsh environment areas:

[0031]

[0032] S46, classify the harsh environment level according to the proportion of harsh environment areas:

[0033]

[0034] Among them, p is the horizontal coordinate of the pixel point, p = 1, 2, ..., h-2, h is the total number of columns in the image; q is the vertical coordinate of the pixel point, q = 1, 2, ..., w-2, w is the total number of rows in the image, G(p, q) is the grayscale value of the pixel point; k is selected according to the actual working conditions.

[0035] Further, in step S4, the image fusion strategy is as follows: when it is determined to be an extremely harsh environment, the visible light image is not fused and the IRI_ON image is directly output; when it is determined to be a normal, slightly harsh, slightly harsh, moderately harsh, or severely harsh environment, non-uniform weighted fusion is performed.

[0036] Furthermore, the implementation steps of the non-uniform weighted fusion are as follows:

[0037] S61, extracting the common information of VLI_OFF and IRI_OFF, then subtracting the common information from IRI_OFF to obtain infrared-specific information, and mapping it to the R channel of the RGB channel;

[0038] S62, respectively input VLI_ON and IRI_OFF into the f of the ON center antagonist model center (m,n) and f surround (m,n), and get VLI_ON-IRI_OFF; input IRI_ON and VLI_OFF into f in the ON center antagonism model respectively. center (m,n) and f surround (m,n), and IRI_ON-VLI_OFF is obtained; then VLI_ON-IRI_OFF and IRI_ON-VLI_OFF are non-uniformly weighted fused and mapped to the G channel of the RGB channel; the weight distribution of VLI_ON-IRI_OFF and IRI_ON-VLI_OFF is based on the proportion of harsh environment area P harsh to divide;

[0039] S63, map VLI_ON to the B channel of the RGB channels.

[0040] Further, in step S5, the enhanced conversion process is as follows:

[0041] S51, continuously dividing the image into a plurality of 3×3 non-overlapping sub-regions;

[0042] S52, calculating the grayscale histogram and cumulative distribution function of each sub-region, and constructing a grayscale mapping table;

[0043] S53, setting a clipping threshold to limit contrast, intercepting pixel values ​​outside the threshold of each sub-region and reallocating pixels;

[0044] S54, performing histogram equalization processing on the cropped area, traversing the input image, and performing bilinear interpolation processing based on the grayscale value of each point and its location and surrounding associated blocks to complete the reconstruction of the grayscale value of the pixel point.

[0045] Compared with the prior art, the present invention has the following significant effects:

[0046] 1. The present invention corrects the image quality degradation caused by the camera or other external factors by applying an image restoration algorithm, imitates the bimodal cell mechanism of the optic tectum of rattlesnakes, and adopts a center-surround antagonistic receptive field model to pre-process the image to enhance contrast and edge information; on the basis of pre-processing, the environment is identified according to the infrared image, and then the image is non-weightedly fused and transformed to enhance, so as to achieve effective fusion of visible light and infrared information, so as to overcome the influence of complex environmental factors such as low illumination, high dust, high water vapor, etc. in the mine, and then achieve effective monitoring of personnel, equipment and environment at the operation site;

[0047] 2. The present invention reduces the weakening of the image quality of visible light and infrared sources caused by the internal structure of the camera and the external environment by setting up image restoration processing, thereby ensuring the high quality of the images before and after fusion;

[0048] 3. The present invention identifies the environment based on the difference in infrared image energy between normal environment and harsh environment, and performs non-uniform weighted fusion of visible light information and infrared information based on the identification results, ensuring that the fusion perception algorithm is applicable to environments of different harshness;

[0049] 4. The present invention extracts infrared-specific information and maps it to the R channel, highlighting the target features; maps the visible light image enhanced by the ON center antagonism model to the B channel, retaining the environmental features; maps the non-uniform weighted fusion image to the G channel, and selectively enhances the visible light information or infrared information according to the environment with the help of non-uniform weighting, so as to make it more in line with actual needs;

[0050] 5. The present invention improves image quality and image contrast by setting up image enhancement processing, so that effective information is further highlighted. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is the overall flow chart of the underground complex environment fusion enhanced perception method of the present invention;

[0052] Figure 2 It is a flow chart of the resistance enhancement method of the present invention;

[0053] Figure 3 This is a schematic diagram of the ON center antagonism model of the present invention;

[0054] Figure 4This is a schematic diagram of the OFF center antagonism model of the present invention;

[0055] Figure 5 is a flow chart of the image fusion method of the present invention;

[0056] Figure 6 Column (a) is a schematic diagram of a visible light source image according to an embodiment of the present invention, column (b) is a schematic diagram of an infrared source image, and column (c) is a schematic diagram of a final fused image. DETAILED DESCRIPTION

[0057] The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0058] like Figure 1 As shown, the main steps of the visual fusion enhanced perception method for complex underground environments provided by the present invention that imitates the visual tectum mechanism of rattlesnakes are as follows:

[0059] Step 1: collect visible light source images and infrared source images through a visible light camera and an infrared camera respectively.

[0060] Step 2, obtaining a visible light restored image and an infrared restored image through image restoration technology;

[0061] like Figure 2 As shown, image restoration technology is used to eliminate image blur caused by the camera or other external factors, and obtain visible light restored images and infrared restored images.

[0062] For image restoration technology, it is assumed that the pixel points meet the statistical standards of Poisson distribution. Based on Bayesian theory, the deconvolution iterative deduction calculation of the degraded image is performed through maximum likelihood estimation, and a clear restored image is obtained when the blur convolution kernel is unknown. The iterative formula is as follows:

[0063]

[0064] Among them, F n+1 (i, j) is the image of the n+1th iteration; F n (i, j) is the image of the nth iteration; G(i, j) is the input image; H(i, j) is the point spread function of the system; H(-i, -j) is the spatial flip of H(i, j).

[0065] Step 3, after registering and cropping the enhanced image, image preprocessing is completed based on the center-surround antagonistic receptor field model of six bimodal cell mechanisms of bionic rattlesnakes to obtain four bionic antagonistic enhancer images;

[0066] like Figure 3 and Figure 4As shown in the figure, the center-surround antagonistic receptor field model of the six bimodal cell mechanisms of bionic rattlesnake is divided into two types: ON center antagonistic model and OFF center antagonistic model, and their mathematical models are expressed as follows:

[0067] ON center antagonism model:

[0068]

[0069] OFF center antagonism model:

[0070] f OFF (m,n)=f center (m,n)·log(f surround (m,n)+1) (3)

[0071] Among them, (m,n) is the pixel coordinate, f center (m,n) is the image input to the central area, f surround (m,n) is the image input to the edge surround area, f ON (m,n) is the output image of the ON center antagonistic model, f OFF (m,n) is the output image of the OFF center antagonistic model.

[0072] Four biomimetic antagonistic enhancer images include inputting the visible light restored image into the ON center antagonistic model f center (m,n) and f surround (m,n) The generated VLI_ON image; the visible light restored image is input into the OFF center antagonism model f center (m,n) and f surround (m,n) generated VLI_OFF image; input the infrared restored image into the ON center antagonism model f center (m,n) and f surround (m,n) IRI_ON image generated; input the infrared restored image into the OFF center antagonistic model f center (m,n) and f surround (m,n) The generated IRI_OFF image.

[0073] Step 4: Use environmental recognition technology to classify the severity of the image sampling environment, and perform image fusion according to the severity.

[0074] Specifically, the environment recognition technology divides the image sampling environment based on the difference in infrared image energy between normal environment and harsh environment. The division process is as follows:

[0075] Step 41, slide the 3×3 window and record the window pixel energy E of each movement W (p,q), the formula is as follows:

[0076]

[0077] Step 42, calculate the total energy E of the entire infrared image, the formula is as follows:

[0078]

[0079] Step 43, calculating the global energy average value of the infrared image moving window The formula is as follows:

[0080]

[0081] Step 44, setting the discrimination coefficient k to divide the normal environment area S of the infrared image normal and harsh environment areas S harsh :

[0082]

[0083] Step 45, calculate the proportion of harsh environment areas P harsh :

[0084]

[0085] Step 46: according to the proportion of harsh environment area P harsh Classification of harsh environment levels:

[0086]

[0087] Among them, p is the horizontal coordinate of the pixel point, p = 1, 2, ..., h-2, h is the total number of columns in the image; q is the vertical coordinate of the pixel point, q = 1, 2, ..., w-2, w is the total number of rows in the image, G(p, q) is the grayscale value of the pixel point, and k is reasonably selected according to the actual working conditions, generally 0.25 to 0.5.

[0088] Image fusion adopts the following strategy: when it is judged to be an extremely harsh environment, the visible light image contains almost no effective information, and no fusion is performed at this time, and the IRI_ON image is directly output; when it is judged to be a normal, slightly harsh, slightly harsh, moderately harsh, or severely harsh environment, non-uniform weighted fusion is performed.

[0089] like Figure 5 As shown, the non-uniform weighted fusion method includes the following steps:

[0090] Step D1, extracting the common information of VLI_OFF and IRI_OFF, then subtracting the common information from IRI_OFF to obtain infrared-specific information, and mapping it to the R channel of the RGB channel;

[0091] Step D2, input VLI_ON and IRI_OFF into the f in the ON center antagonism model respectively. center (m,n) and f surround (m,n), and get VLI_ON-IRI_OFF; input IRI_ON and VLI_OFF into f in the ON center antagonism model respectively. center (m,n) and f surround (m,n), get IRI_ON-VLI_OFF, then perform non-uniform weighted fusion of VLI_ON-IRI_OFF and IRI_ON-VLI_OFF, and map them to the G channel of the RGB channel;

[0092] The weighting strategy for VLI_ON-IRI_OFF and IRI_ON-VLI_OFF is based on the proportion of harsh environment areas P harsh The specific strategies are shown in Table 1:

[0093] Table 1 VLI_ON-IRI_OFF and IRI_ON-VLI_OFF weighting strategy table

[0094]

[0095]

[0096] Step D3, map VLI_ON to the B channel of the RGB channel. Figure 6 Columns (a) and (b) are the visible light source image and infrared source image respectively. After processing from step 1 to step 4, the fused image is as follows: Figure 6 As shown in column (c) of ; it can be seen that the fused image well integrates the background information of the visible light image and the target features of the infrared image. Compared with the source image, the fused image has better quality and higher contrast, which can provide more effective information for upstream visual tasks.

[0097] Step 5: Enhance the fused image by using image enhancement technology to improve image contrast, highlight effective information, and obtain the final image;

[0098] The enhanced conversion process is as follows:

[0099] Step 51, continuously dividing the image into a number of 3×3 non-overlapping sub-regions;

[0100] Step 52, calculating the grayscale histogram and cumulative distribution function of each sub-region, and constructing a grayscale mapping table;

[0101] Step 53, setting a clipping threshold to limit contrast, intercepting pixel values ​​outside the threshold of each sub-region and redistributing pixels;

[0102] Step 54, performing histogram equalization processing on the cropped area, traversing the input image, and performing bilinear interpolation processing based on the grayscale value of each point and its location and surrounding associated blocks to complete the reconstruction of the pixel grayscale value and obtain an enhanced image with high contrast and prominent features.

[0103] The above descriptions are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual fusion enhanced perception method for complex underground environments that imitates the visual tectum mechanism of rattlesnakes, characterized in that: The following steps are involved: S1, collect visible light source images and infrared source images through a visible light camera and an infrared camera respectively; S2, obtaining a visible light restored image and an infrared restored image through image restoration technology; S3, after registering and cropping the enhanced image, the image preprocessing is completed based on the center-surround antagonistic receptor field model of six bimodal cell mechanisms of bionic rattlesnakes, and four bionic antagonistic enhancer images of VLI_ON, VLI_OFF, IRI_ON and IRI_OFF are obtained; S4, using environmental recognition technology to classify the image acquisition environment into normal, slightly bad, slightly bad, moderately bad, severely bad and extremely bad levels; the strategies for image fusion according to the severity are as follows: when it is determined to be an extremely bad environment, the visible light image is not fused and the IRI_ON image is directly output; when it is determined to be normal, slightly bad, slightly bad, moderately bad, or severely bad, non-uniform weighted fusion is performed; S5, enhancing and transforming the fused image through image enhancement technology to obtain the final image.

2. The method for enhancing perception of complex underground environment visual fusion by imitating the visual cover mechanism of a rattlesnake according to claim 1 is characterized in that: In step S2, it is assumed that the pixel points meet the statistical standard of Poisson distribution, and based on Bayesian theory, the deconvolution iterative deduction calculation is performed on the degraded image through maximum likelihood estimation, and the iterative formula is as follows: In the formula, F n+1 (i, j) is the image of the n+1th iteration; F n (i, j) is the image of the nth iteration; G(i, j) is the input image; H(i, j) is the point spread function of the system; H(-i, -j) is the spatial flip of H(i, j).

3. The method for visual fusion enhancement perception of complex underground environments by imitating the visual hood mechanism of a rattlesnake according to claim 1 is characterized in that: In step S3, the center-surround antagonistic receptor field model of the six bimodal cell mechanisms of bionic rattlesnake is divided into an ON center antagonistic model and an OFF center antagonistic model, and their mathematical models are respectively expressed as follows: ON center antagonism model: OFF center antagonism model: f OFF (m,n)=f center (m,n)·log(f surround (m,n)+1) Among them, (m, n) is the pixel coordinate, f center (m,n) is the image input to the central area, f surround (m,n) is the image input to the edge surround area, f ON (m,n) is the output image of the ON center antagonistic model, f OFF (m,n) is the output image of the OFF center antagonistic model.

4. The method for enhancing perception of complex underground environment visual fusion by imitating the visual hood mechanism of a rattlesnake according to claim 3 is characterized in that: In step S3, the four bionic antagonistic enhancer images include inputting the visible light restored image into the ON center antagonistic model f center (m,n) and f surround (m,n) The generated VLI_ON image; the visible light restored image is input into the OFF center antagonism model f center (m,n) and f surround (m,n) generated VLI_OFF image; input the infrared restored image into the ON center antagonism model f center (m,n) and f surround (m,n) IRI_ON image generated; input the infrared restored image into the OFF center antagonistic model f center (m,n) and f surround (m,n) The generated IRI_OFF image.

5. The method for enhancing perception of complex underground environment visual fusion by imitating the visual hood mechanism of a rattlesnake according to claim 1 is characterized in that: In step S4, the image sampling environment is divided based on the difference in infrared image energy between a normal environment and a harsh environment. The division process is as follows: S41, slide the 3×3 window and record the window pixel energy E each time it moves W (p,q), the formula is as follows: S42, calculate the total energy E of the entire infrared image, the formula is as follows: S43, calculating the global energy average value of the infrared image moving window The formula is as follows: S44, setting the discrimination coefficient k to divide the normal environment area S of the infrared image normal and harsh environment areas S harsh : S45, calculate the proportion of harsh environment areas: S46, classify the harsh environment level according to the proportion of harsh environment areas: Among them, p is the horizontal coordinate of the pixel point, p = 1, 2, ..., h-2, h is the total number of columns in the image; q is the vertical coordinate of the pixel point, q = 1, 2, ..., w-2, w is the total number of rows in the image, G(p, q) is the grayscale value of the pixel point; k is selected according to the actual working conditions.

6. The method for enhancing perception of complex underground environment visual fusion by imitating the visual tectum mechanism of rattlesnake according to claim 1 is characterized in that: The implementation steps of the non-uniform weighted fusion are as follows: S61, extracting the common information of VLI_OFF and IRI_OFF, then subtracting the common information from IRI_OFF to obtain infrared-specific information, and mapping it to the R channel of the RGB channel; S62, respectively input VLI_ON and IRI_OFF into the f of the ON center antagonist model center (m,n) and f surround (m,n), and get VLI_ON-IRI_OFF; input IRI_ON and VLI_OFF into f in the ON center antagonism model respectively. center (m,n) and f surround (m,n), and IRI_ON-VLI_OFF is obtained; then VLI_ON-IRI_OFF and IRI_ON-VLI_OFF are non-uniformly weighted fused and mapped to the G channel of the RGB channel; the weight distribution of VLI_ON-IRI_OFF and IRI_ON-VLI_OFF is based on the proportion of harsh environment area P harsh to divide; S63, map VLI_ON to the B channel of the RGB channels.

7. The method for enhancing perception of complex underground environment visual fusion by imitating the visual hood mechanism of a rattlesnake according to claim 1 is characterized in that: In step S5, the enhanced conversion process is as follows: S51, continuously dividing the image into a plurality of 3×3 non-overlapping sub-regions; S52, calculating the grayscale histogram and cumulative distribution function of each sub-region, and constructing a grayscale mapping table; S53, setting a clipping threshold to limit contrast, intercepting pixel values ​​outside the threshold of each sub-region and redistributing pixels; S54, performing histogram equalization processing on the cropped area, traversing the input image, and performing bilinear interpolation processing based on the grayscale value of each point and its location and surrounding associated blocks to complete the reconstruction of the grayscale value of the pixel point.

Images