Low-light full-color micro-light night vision reconnaissance system

Abstract

The application discloses a low-light full-color low-light-level night survey system, and particularly relates to the field of image processing technology, comprising a low-light imaging module, a dynamic range expansion module, a multispectral and infrared fusion module, a three-dimensional reconstruction and modeling module and an image fusion module; the low-light imaging module acquires visible light and infrared images under low-light environment and transmits the images to the dynamic range expansion module; the dynamic range expansion module adjusts bright and dark details of the images by using a high dynamic range algorithm, and transmits the adjusted image data to the multispectral and infrared fusion module; the multispectral and infrared fusion module performs multispectral fusion on the visible light and infrared images by using an image processing algorithm, creates a comprehensive image, and transmits the image to the three-dimensional reconstruction and modeling module; the three-dimensional reconstruction and modeling module combines depth data with image data to generate a three-dimensional model, and transmits the model to the image fusion module; and the image fusion module fuses data of all the modules to generate high-quality images and data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and more particularly, to a low-light full-color low-light night vision survey system. BACKGROUND

[0002] In the night or low-light environment, the traditional optical survey equipment relies on natural light source or artificial lighting for work. However, due to insufficient light, the traditional equipment often cannot effectively capture enough light signals, resulting in blurred images, missing details, and even complete inability to identify target objects. This limitation brings great challenges to many survey tasks, especially in scenarios that require high-precision measurement, accurate navigation, three-dimensional photogrammetry or environmental monitoring.

[0003] In addition, the visual effect in low-light environment is often not as clear as during the day, and the contrast between dark and bright parts is too large, resulting in compression of the dynamic range of the image, further affecting the presentation of details. More importantly, accurate geographic information, morphological characteristics of target objects and rapid identification of complex scenes often depend on color restoration of images, and traditional night vision equipment often only provides monochrome or grayscale images, which cannot accurately reflect the actual situation, bringing great inconvenience to survey personnel. SUMMARY

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present application provide a low-light full-color low-light night vision survey system to solve the problems raised in the above background.

[0005] To achieve the above-mentioned purpose, the present application provides the following technical scheme, a low-light full-color low-light night vision survey system, comprising a low-light imaging module, a dynamic range expansion module, a multispectral and infrared fusion module, a three-dimensional reconstruction and modeling module, and an image fusion module;

[0006] The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire visible light images and infrared images in a low-light environment, and transmits them to the image fusion module and the dynamic range expansion module;

[0007] The dynamic range expansion module receives the visible light and infrared images transmitted by the low-light imaging module, and performs image preprocessing, adjusts the bright and dark details of the image using a high dynamic range algorithm, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multispectral and infrared fusion module;

[0008] The multispectral and infrared fusion module uses image processing algorithms to perform multispectral fusion on the visible light images and infrared images, creates a comprehensive image, and transmits it to the image fusion module and the three-dimensional reconstruction and modeling module;

[0009] The three-dimensional reconstruction and modeling module obtains the accurate distance of the target object by laser ranging, generates depth data, combines the depth data with image data to generate a three-dimensional model, and transmits the three-dimensional model to the image fusion module.

[0010] The image fusion module fuses the data of all modules to generate high-quality images and data.

[0011] In a preferred embodiment, the low-light imaging module uses a low-light image sensor and an infrared sensor to obtain visible light images and infrared images in a low-light environment, and transmits the visible light images and infrared images to the image fusion module and the dynamic range expansion module. The specific steps are as follows:

[0012] Step A1, ambient light monitoring: using a photosensitive sensor to monitor the light intensity of the current environment, and judging whether it is in a low-light environment according to the monitoring result. In a low-light environment, the low-light imaging module activates the low-light image sensor and the infrared sensor;

[0013] Step A2, image acquisition: using a low-light image sensor to convert light signals in the environment into digital images, acquiring visible light images in a low-light environment, detecting thermal radiation generated by objects of different temperatures through an infrared sensor, generating infrared images, and automatically adjusting the exposure time according to the change of ambient light to obtain multiple visible light and infrared images under different exposure times;

[0014] Step A3, image transmission: integrating the acquired visible light images and infrared images to form a complete data packet, the data packet including exposure time information of the images, and transmitting the integrated image data in the form of digital signals to the image fusion module and the dynamic range expansion module.

[0015] In a preferred embodiment, the dynamic range expansion module receives the visible light images and infrared images obtained by the low-light imaging module, performs image preprocessing, adjusts the bright and dark details of the images using a high dynamic range algorithm, expands the dynamic range of the images, and transmits the adjusted image data to the image fusion module and the multispectral and infrared fusion module. The specific steps are as follows:

[0016] Step C1, image preprocessing: representing the visible light images and infrared images received by the low-light imaging module as I visible and I IR respectively, and removing noise from the received two images to improve image quality;

[0017] Step C2, brightness calculation: calculating the brightness values of the visible light images and infrared images, the brightness value of the visible light image being: visible (x,y)=0.2999·R visible (x,y)+0.587·G visible(x, y) + 0.114 * B visible (x, y); the luminance value of the infrared image is: L IR (x, y) = I IR (x, y); wherein, L visible (x, y) represents the luminance value of the visible light image at position (x, y), R visible (x, y), G visible (x, y), B visible (x, y) respectively represent the pixel values of the red channel, green channel and blue channel in the visible light image at position (x, y); L IR (x, y) represents the luminance value of the infrared image at position (x, y), I IR (x, y) represents the pixel value of the infrared image at position (x, y), according to the infrared image without color channel, the pixel value of the infrared image is directly used as its luminance value;

[0018] Step C3, HDR synthesis: using multiple visible light and infrared images with different exposure times for HDR synthesis, enhancing dark details and compressing bright areas by dynamic range expansion of the images, and transmitting the adjusted HDR image data to the image fusion module and the multispectral and infrared fusion module, further comprising the following steps:

[0019] Step C301, calculating weight: the visible light and infrared images under different exposure times are respectively denoted as and wherein, m and n respectively represent the indexes of the visible light image and the infrared image, the weight is calculated according to the luminance and exposure time of each image, and the specific calculation formula is as follows:

[0020]

[0021] wherein, and are the variances of the luminance values of the visible light and infrared images, and are the weights of the visible light and infrared images, L max is the maximum value of the luminance of all visible light images, T m is the exposure time of the mth visible light image, is the luminance value of the mth visible light image, L max,IR is the maximum value of the luminance of all infrared images, is the luminance value of the nth infrared image, T n is the exposure time of the nth infrared image;

[0022] Step C302, Synthesis of image: Synthesize the high dynamic range visible light communication and infrared image using the calculated weight, the high dynamic range visible light image expression is: The high dynamic range infrared image expression is: Wherein, I HDR,visible (x,y) represents the pixel value of the synthesized visible light HDR image at position (x,y), I HDR,IR (x,y) represents the pixel value of the synthesized infrared HDR image at position (x,y), And Wvis and WIR are the weights of the visible light and infrared images respectively, Imis the pixel value of the mth visible light image at position (x,y), Inis the pixel value of the nth infrared image at position (x,y);

[0023] Step C303, Detail enhancement: Perform local contrast enhancement on the synthesized HDR image using the adaptive histogram equalization method to improve dark details and compress bright regions, the specific calculation formula is as follows:

[0024] I enhanced (x,y) = I HDR (x,y) + c·(I base (x,y) - I HDR (x,y))

[0025] Wherein, I enhanced (x,y) is the image value after detail enhancement, I HDR (x,y) is the pixel value of the synthesized HDR image at position (x,y), I bace is the original image, and c is a constant that controls the enhancement intensity.

[0026] In a preferred embodiment, the multispectral and infrared fusion module is to perform multispectral fusion on the visible light image and the infrared image using an image processing algorithm to create a comprehensive image, which is transmitted to the image fusion module and the three-dimensional reconstruction and modeling module, the specific steps are as follows:

[0027] Step D1, Feature extraction: Receive the synthesized visible light HDR image I HDR,visible (x,y) and infrared HDR image I HDR,IR (x,y) from the dynamic range expansion module, register the two images, and extract color information, texture features and contour information from the visible light image; extract thermal features from the infrared image;

[0028] Step D2, Feature fusion: According to the feature information extracted in step D1, calculate the feature values of the visible light image and the infrared image, and use the feature values to calculate the fusion weight, the specific calculation formula is as follows:

[0029] f visible (x,y) = a · C(x,y) + b · T(x,y) + g · E(x,y)

[0030] f IR (x,y) = d · TH(x,y)

[0031]

[0032] where f visible (x,y) and f IR (x,y) are the feature values of the visible light image and the infrared image respectively, C(x,y) is the color feature, T(x,y) is the texture feature, E(x,y) is the contour feature, a, b, g, d are weight coefficients, TH(x,y) is the thermal feature, and w(x,y) is the fusion weight;

[0033] Step D3, generating a comprehensive image: using the calculated fusion weight for weighted average to generate a comprehensive image, and transmitting the comprehensive image to the image fusion module and the three-dimensional reconstruction and modeling module, the comprehensive image expression is: fused (x,y) = w(x,y) · I HDR,visible (x,y) + [1-w(x,y)] · I HDR,IR (x,y), where I fused (x,y) is the pixel value of the comprehensive image at position (x,y), w(x,y) is the fusion weight, I HDR,visible (x,y) and I HDR,IR (x,y) are the pixel values of the visible light HDR image and the infrared HDR image at position (x,y) respectively.

[0034] In a preferred embodiment, the three-dimensional reconstruction and modeling module obtains the accurate distance of the target object by laser ranging, generates depth data, combines the depth data with the image data to generate a three-dimensional model, and transmits the three-dimensional model to the image fusion module. The specific steps are as follows:

[0035] Step E1, obtaining the distance D between the target object and the sensor by laser emission and receiving the return signal, and converting the obtained distance D into depth data D depth (x,y), representing the depth value of the target object corresponding to each pixel position (x,y);

[0036] Step E2, constructing a three-dimensional point cloud: generating a three-dimensional point cloud data through the pixel position and the depth value, the three-dimensional coordinates of each pixel (x,y) are and combining all the three-dimensional coordinate points into a point cloud set where f x and fy (x, y, z) is the focal length of the camera in the x and y directions, and (x, y, z) are the coordinates in three-dimensional space. depth (x,y) is the depth value at pixel (x,y);

[0037] Step E3, 3D Reconstruction: The generated point cloud data is denoised, and a triangle is generated by connecting three non-collinear points in the point cloud to form a 3D mesh T. j =(P a ,P b ,P c ), where P a P b P c It selects three non-collinear points from the point cloud to form the three vertices of a triangle, and then extracts the color information from the composite image I. fused (x,y) is mapped onto each triangle of the 3D mesh to generate a 3D model that includes spatial data and texture information, further including the following steps:

[0038] Step E301, Texture Mapping: For each vertex P of a triangle a P b P c The corresponding pixel coordinates are from the composite image I fused Obtain color information from (x,y): C a =I fused (x a ,y a ), C b =I fused (x b ,y b ), C c =I fused (x c ,y c ), where (x a ,y a ), (x a ,y a ) and (x c ,y c ) is related to the vertex P of the triangle a P b P c Associated image coordinates;

[0039] Step E302, Texture Coordinates: Calculate the texture coordinates (u,v) = λ1(u) for each point using the barycentric coordinates. a ,v a )+λ2(u b ,v b )+λ3(u c ,v c), wherein the barycentric coordinates (lambda1, lambda2, lambda3) satisfy lambda1+lambda2+lambda3=1, and a three-dimensional model expression including the spatial data and the texture information is obtained as: S={T j =(P a ,P b ,P c ),(u a ,v a ),(u b ,v b ),(u c ,v c ),(C a ,C b ,C c )|j=1,2,...,K}, K is the total number of triangles generated by connecting all non-collinear points, (C a ,C b ,C c ) is a color value of each point on the triangle, extracted from the image I fused (x,y), (u a ,v a ), (u b ,v b ), (u c ,v c ) are texture coordinates of each vertex.

[0040] In a preferred embodiment, the image fusion module fuses the data of all modules to generate high-quality images and data, and the specific steps are as follows:

[0041] Step S1, data receiving: receiving data from the low-light imaging module, the dynamic range expansion module, the multispectral and infrared fusion module, and the three-dimensional reconstruction module, and standardizing the data transmitted by each module;

[0042] Step S2, multi-modal data fusion: fusing image data and spatial data, generating final high-quality images and three-dimensional models according to the fused data, and displaying full-color night vision images on the screen, providing depth and spatial positioning information, wherein the image data includes visible light images, infrared images, and fused images, and the spatial data includes depth data and three-dimensional models.

[0043] The present application has the advantages that: by combining high-sensitivity sensors, image enhancement technology, three-dimensional modeling, and efficient data processing and fusion technology, high-quality full-color images and accurate measurement results can be provided in low-light environments, and the present application uses high dynamic range, accurate positioning, three-dimensional reconstruction, and data fusion functions to enable complex geographic information acquisition, target recognition, and environmental monitoring in low-light environments, overcoming the shortcomings of traditional night vision equipment in low-light environments. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A system flowchart of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present application.

[0046] In the description of the present application, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0047] In the description of the present application, the term "for example" is used to indicate "as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily interpreted as more preferred or more advantageous than other embodiments. The following description is given in order to enable any person skilled in the art to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person skilled in the art can realize the present application without using these specific details. In other examples, well-known structures and processes will not be described in detail in order to avoid unnecessary details making the description of the present application obscure. Therefore, the present application is not intended to be limited to the shown embodiments, but is consistent with the broadest scope in accordance with the principles and characteristics disclosed in the present application.

[0048] Embodiment 1

[0049] The present embodiment provides a low-light full-color low-light night reconnaissance system as shown in Figure 1 The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire visible light images and infrared images in a low-light environment, and transmits them to the image fusion module and the dynamic range expansion module.

[0050] The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire visible light images and infrared images in a low-light environment, and transmits them to the image fusion module and the dynamic range expansion module.

[0051] The dynamic range expansion module receives the visible light and infrared images transmitted by the low-light imaging module, and performs image preprocessing, adjusts the bright and dark details of the image using a high dynamic range algorithm, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multispectral and infrared fusion module;

[0052] The multispectral and infrared fusion module performs multispectral fusion on the visible light image and the infrared image using an image processing algorithm to create a comprehensive image, which is transmitted to the image fusion module and the three-dimensional reconstruction and modeling module;

[0053] The three-dimensional reconstruction and modeling module obtains the accurate distance of the target object through laser ranging to generate depth data, combines the depth data with the image data to generate a three-dimensional model, and transmits the three-dimensional model to the image fusion module;

[0054] The image fusion module fuses the data of all modules to generate high-quality images and data.

[0055] In this embodiment, the low-light imaging module is specifically described. The low-light imaging module uses a low-light image sensor and an infrared sensor to obtain visible light images and infrared images in a low-light environment, and transmits them to the image fusion module and the dynamic range expansion module to improve the usability and accuracy of the images. The specific steps are as follows:

[0056] Step A1, ambient light monitoring: use a photosensitive sensor to monitor the light intensity of the current environment, and determine whether it is in a low-light environment according to the monitoring result. In a low-light environment, the low-light imaging module activates the low-light image sensor and the infrared sensor;

[0057] Step A2, image acquisition: use the low-light image sensor to convert the light signal in the environment into a digital image, acquire a visible light image in a low-light environment, detect the thermal radiation generated by objects of different temperatures through the infrared sensor to generate an infrared image, and automatically adjust the exposure time according to the change of the ambient light to obtain multiple visible light and infrared images under different exposure times;

[0058] Step A3, image transmission: integrate the acquired visible light image and infrared image to form a complete data packet, which includes the exposure time information of the image. The integrated image data is transmitted to the image fusion module and the dynamic range expansion module in the form of digital signals.

[0059] In this embodiment, it is particularly necessary to explain the dynamic range expansion module. The dynamic range expansion module receives the visible light image and the infrared image acquired by the low-light imaging module, and performs image preprocessing. The high dynamic range algorithm is used to adjust the bright part and dark part details of the image, expand the dynamic range of the image, and transmit the adjusted image data to the image fusion module and the multispectral and infrared fusion module. The specific steps are as follows:

[0060] Step C1, image preprocessing: the visible light image and the infrared image received by the low-light imaging module are respectively denoted as I visible and I IR . The noise of the received two images is removed to improve the image quality.

[0061] Step C2, brightness calculation: the brightness values of the visible light image and the infrared image are calculated. The visible light image brightness value is L visible (x,y) = 0.2999·R visible (x,y) + 0.587·G visible (x,y) + 0.114·B visible (x,y); and the infrared image brightness value is L IR (x,y) = I IR (x,y); wherein L visible (x,y) represents the brightness value of the visible light image at position (x, y), R visible (x,y), G visible (x,y), and B visible (x,y) represent the pixel values of the red channel, the green channel, and the blue channel of the visible light image at position (x, y), respectively; and L IR (x,y) represents the brightness value of the infrared image at position (x, y), I IR (x,y) represents the pixel value of the infrared image at position (x, y). According to the infrared image without color channels, the pixel value of the infrared image is directly used as its brightness value.

[0062] Step C3, HDR synthesis: using multiple visible light and infrared images with different exposure times for HDR synthesis, enhancing dark details and compressing bright regions by expanding the dynamic range of the image, and transmitting the adjusted HDR image data to the image fusion module and the multispectral and infrared fusion module, further comprising the following steps:

[0063] Step C301, weight calculation: the visible light and infrared images under different exposure times are respectively denoted as and wherein m and n respectively represent the indexes of the visible light image and the infrared image. The weight is calculated according to the brightness and exposure time of each image. The specific calculation formula is as follows:

[0064]

[0065] where, and are the variances of the visible and infrared image luminance values, respectively, and are the weights of the visible and infrared images, respectively, L max is the maximum value of all visible image luminances, T m is the exposure time of the mth visible image, is the luminance value of the mth visible image, L max,IR is the maximum value of all infrared image luminances, is the luminance value of the nth infrared image, T n is the exposure time of the nth infrared image;

[0066] Step C302, Synthesis of images: Synthesize the high dynamic range visible light communication and infrared images using the calculated weights, the high dynamic range visible light image expression is: The high dynamic range infrared image expression is: where, HDR,visible (x, y) represents the pixel value of the synthesized visible light HDR image at position (x, y), I HDR,IR (x, y) represents the pixel value of the synthesized infrared HDR image at position (x, y), and are the weights of the visible and infrared images, respectively, is the pixel value of the mth visible image at position (x, y), is the pixel value of the nth infrared image at position (x, y);

[0067] Step C303, Detail enhancement: Perform local contrast enhancement on the synthesized HDR image using an adaptive histogram equalization method to improve dark details and compress bright regions, the specific calculation formula is as follows:

[0068] I enhanced (x, y) = I HDR (x, y) + c · (I base (x, y) - I HDR (x, y))

[0069] where, enhanced (x, y) is the image value after detail enhancement, I HDR (x, y) is the pixel value of the synthesized HDR image at position (x, y), I bace is the original image, and c is a constant that controls the enhancement intensity.

[0070] In this embodiment, it is particularly necessary to explain the multispectral and infrared fusion module. The multispectral and infrared fusion module is used for multispectral fusion of a visible light image and an infrared image by using an image processing algorithm, creating a comprehensive image, so that more feature information is obtained in the same image, and the recognition and positioning capability of a target is improved. The comprehensive image is transmitted to an image fusion module and a three-dimensional reconstruction and modeling module. The specific steps are as follows:

[0071] Step D1, feature extraction: receiving the synthesized visible light HDR image I HDR,visible (x,y) and the infrared HDR image I HDR,IR (x,y) from the dynamic range expansion module, registering the two images to ensure that the two images are aligned, and extracting color information, texture features and contour information from the visible light image and extracting thermal features from the infrared image;

[0072] Step D2, feature fusion: calculating the feature values of the visible light image and the infrared image according to the feature information extracted in step D1, and calculating the fusion weight by using the feature values. The specific calculation formula is as follows:

[0073] f visible (x,y) = a*C(x,y) + β*T(x,y) + γ*E(x,y)

[0074] f IR (x,y) = δ*TH(x,y)

[0075]

[0076] wherein f visible (x,y) and f IR (x,y) are the feature values of the visible light image and the infrared image respectively, C(x,y) is the color feature, T(x,y) is the texture feature, E(x,y) is the contour feature, ε, β, γ, δ are weight coefficients, TH(x,y) is the thermal feature, and ω(x,y) is the fusion weight;

[0077] Step D3, generating a comprehensive image: performing weighted averaging by using the calculated fusion weight to generate a comprehensive image, and transmitting the comprehensive image to the image fusion module and the three-dimensional reconstruction and modeling module. The comprehensive image expression is as follows: I fused (x,y) = ω(x,y)*I HDR,visible (x,y) + [1-ω(x,y)]*I HDR,IR (x,y), wherein I fused (x,y) is the pixel value of the comprehensive image at the position (x,y), ω(x,y) is the fusion weight, I HDR,visible (x,y) and I HDR,IR(x,y) are the pixel values of the visible light HDR image and the infrared HDR image at position (x,y), respectively.

[0078] In this embodiment, it is particularly necessary to explain the three-dimensional reconstruction and modeling module. The three-dimensional reconstruction and modeling module obtains the accurate distance of the target object through laser ranging, generates depth data, combines the depth data with image data, and generates a three-dimensional model. The three-dimensional model is transmitted to the image fusion module, thereby improving the accurate positioning of the object in space. The specific steps are as follows:

[0079] Step E1, obtaining the distance D between the target object and the sensor through laser emission and receiving the return signal, and converting the obtained distance D into depth data D depth (x,y), representing the depth value of the target object corresponding to each pixel position (x,y);

[0080] Step E2, constructing a three-dimensional point cloud: generating three-dimensional point cloud data through pixel positions and depth values, and the three-dimensional coordinates of each pixel (x,y) are and all three-dimensional coordinate points are combined into a point cloud set where f x and f y are the focal lengths of the camera in the x and y directions, (X,Y,Z) are the coordinates in the three-dimensional space, D depth (x,y) is the depth value at the pixel (x,y);

[0081] Step E3, three-dimensional reconstruction: denoising the generated point cloud data, and generating a triangle by connecting three non-collinear points in the point cloud, forming a three-dimensional mesh T j =(P a ,P b ,P c ), where P a , P b , P c are three non-collinear points selected from the point cloud, constituting three vertices of the triangle, and the color information of the image is mapped from the composite image I fused (x,y) to each triangle of the three-dimensional mesh, generating a three-dimensional model including spatial data and texture information, further comprising the following steps:

[0082] Step E301, texture mapping: for each vertex P a , P b , P c of the triangle, the corresponding pixel coordinates are obtained from the composite image I fused (x,y) to obtain color information: C a =I fused (x a ,y a ), Cb = I fused (x b ,y b ), C c = I fused (x c ,y c ), wherein (x a ,y a ), (x a ,y a ) and (x c ,y c ) are image coordinates associated with triangle vertices P a , P b , P c ;

[0083] Step E302, texture coordinates: calculate the texture coordinates (u, v) corresponding to each point through the barycentric coordinates (λ1, λ2, λ3) = λ1(u a ,v a ) + λ2(u b ,v b ) + λ3(u c ,v c ), wherein the barycentric coordinates (λ1, λ2, λ3) satisfy λ1 + λ2 + λ3 = 1 and are related to the position of the point in the triangle, and the three-dimensional model expression including spatial data and texture information is obtained as: S = {T j = (P a , P b , P c ), (u a ,v a ), (u b ,v b ), (u c ,v c ), (C a , C b , C c ) | j = 1, 2, …, K}, K is the total number of triangles generated by connecting all non-collinear points, (C a , C b , C c ) is the color value of each point on the triangle, extracted from the image I fused (x, y), (u a , v a ), (u b , v b ), (u c , v c ) are the texture coordinates of each vertex.

[0084] In this embodiment, it is specifically required to explain the image fusion module, the image fusion module fuses data of all modules to generate high-quality images and data, and the specific steps are as follows:

[0085] Step S1, data receiving: receiving data from the low-light imaging module, the dynamic range expansion module, the multispectral and infrared fusion module and the three-dimensional reconstruction module, and performing standardized processing on the data transmitted by each module to ensure that the data format is uniform and the resolution is consistent;

[0086] Step S2, multi-modal data fusion: fusing image data and spatial data to ensure the consistency of image brightness, contrast and color, generating final high-quality images and three-dimensional models according to the fused data, and displaying full-color night vision images on the screen to provide depth and spatial positioning information, the image data includes visible light images, infrared images and fused images, and the spatial data includes depth data and three-dimensional models.

[0087] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0088] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0089] The present application is described with reference to flowcharts and / or block diagrams according to the method, device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of the flows and / or blocks in the flowchart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a machine that implements the functions specified in the flowchart and / or block diagram. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0090] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions that are executed on the computer or other programmable apparatus provide steps for implementing the Figure 1 function specified in the flow or flows and / or blocks Figure 1 of the block or blocks.

[0092] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such additional variations and modifications as fall within the scope of the application.

[0093] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A low light full color micro light night vision reconnaissance system characterized by: The low-light imaging module, the dynamic range expansion module, the multispectral and infrared fusion module, the three-dimensional reconstruction and modeling module, and the image fusion module are included. The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire visible light images and infrared images in a low-light environment and transmits the images to the image fusion module and the dynamic range expansion module. The dynamic range expansion module receives the visible light and infrared images transmitted by the low-light imaging module, performs image preprocessing, adjusts the bright and dark details of the images using a high dynamic range algorithm, expands the dynamic range of the images, and transmits the adjusted image data to the image fusion module and the multispectral and infrared fusion module. The multispectral and infrared fusion module uses image processing algorithms to perform multispectral fusion on the visible light images and the infrared images, creating a comprehensive image, which is transmitted to the image fusion module and the three-dimensional reconstruction and modeling module. The infrared fusion module also includes HDR synthesis: using multiple visible light and infrared images with different exposure times for HDR synthesis, enhancing dark details and compressing bright areas by expanding the dynamic range of the images, and transmitting the adjusted HDR image data to the image fusion module and the multispectral and infrared fusion module. Compute the weights: the visible and infrared images at different exposure times are denoted as and where m and n represent the indices of the visible and infrared images, respectively, and the weights are computed according to the brightness and exposure time of each image, with the specific formula as follows: wherein, and are the variances of the visible and infrared image luminance values, respectively, and are the weights of the visible and infrared images, respectively, L max is the maximum value of all visible image luminances, T m is the exposure time of the mth visible image, is the luminance value of the mth visible image, L max,IR is the maximum value of all infrared image luminances, is the luminance value of the nth infrared image, T n is the exposure time of the nth infrared image; The three-dimensional reconstruction and modeling module acquires the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a three-dimensional model, and transmits the model to the image fusion module. The image fusion module fuses the data from all modules to generate high-quality images and data.

2. A low-light full-color micro-light night vision reconnaissance system according to claim 1, characterized in that: The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire visible light images and infrared images in a low-light environment and transmits the images to the image fusion module and the dynamic range expansion module, with the following specific steps: Step A1, ambient light monitoring: use a photosensitive sensor to monitor the light intensity of the current environment, and determine whether it is in a low-light environment according to the monitoring results. In a low-light environment, the low-light imaging module activates the low-light image sensor and the infrared sensor. Step A2, image acquisition: use the low-light image sensor to convert the light signal in the environment into a digital image, acquire a visible light image in a low-light environment, and generate an infrared image by detecting the thermal radiation of objects with different temperatures through the infrared sensor. Adjust the exposure time automatically according to the change of ambient light to acquire multiple visible light and infrared images with different exposure times. Step A3, image transmission: integrate the acquired visible light image and infrared image to form a complete data packet, which includes the exposure time information of the image. Transmit the integrated image data in digital signal form to the image fusion module and the dynamic range expansion module.

3. A low-light full-color micro-light night vision reconnaissance system according to claim 2, characterized in that: The dynamic range expansion module receives the visible light image and infrared image acquired by the low-light imaging module, performs image preprocessing, adjusts the bright and dark details of the images using a high dynamic range algorithm, expands the dynamic range of the images, and transmits the adjusted image data to the image fusion module and the multispectral and infrared fusion module, with the following specific steps: Step C1, image pre-processing: the visible light image and the infrared image received by the low-light imaging module are respectively denoted as I visible and I IR , and noise removal is performed on the two received images; Step C2, luminance calculation: calculate the luminance values of the visible light image and the infrared image, the visible light image luminance value is: L visible (x,y) = 0.2999 · R visible (x,y) + 0.587 · G visible (x,y) + 0.114 · B visible (x,y); the infrared image luminance value is: L IR (x,y) = I IR (x,y); wherein, L visible (x,y) represents the luminance value of the visible light image at position (x, y), R visible (x,y), G visible (x,y), B visible (x,y) respectively represent the pixel values of the red channel, the green channel, the blue channel in the visible light image at position (x, y); L IR (x,y) represents the luminance value of the infrared image at position (x, y), I IR (x,y) represents the pixel value of the infrared image at position (x, y).

4. A low-light full-color micro-light night vision reconnaissance system according to claim 3, characterized in that: The HDR synthesis using multiple different exposure time visible light and infrared images enhances dark details and compresses bright regions by dynamic range expansion, and transmits the adjusted HDR image data to the image fusion module and the multispectral and infrared fusion module, further comprising the following steps: Step C 302, Synthesis image: Synthesize the high dynamic range visible light communication and infrared images using the computed weights, the high dynamic range visible light image expression: The high dynamic range infrared image expression is: where I HDR,visible (x,y) represents the pixel value of the synthesized visible light HDR image at position (x,y), I HDR,IR (x,y) represents the pixel value of the synthesized infrared HDR image at position (x,y), and are the weights of the visible light and infrared images, respectively, is the pixel value of the mth visible light image at position (x,y), is the pixel value of the nth infrared image at position (x,y); Step C303, detail enhancement: using adaptive histogram equalization method to enhance the local contrast of the synthesized HDR image to improve dark details and compress bright regions, the specific calculation formula is as follows: I enhanced (x,y) = I HDR (x,y) + c · (I base (x,y) - I HDR (x,y)) where I enhanced (x, y) is the detail enhanced image value, I HDR (x, y) is the pixel value of the synthesized HDR image at position (x, y), I bace is the original image, and c is a constant that controls the enhancement intensity.

5. A low-light full-color micro-light night vision reconnaissance system according to claim 4, characterized in that: The multispectral and infrared fusion module uses image processing algorithms to perform multispectral fusion on visible light images and infrared images to create a comprehensive image, which is transmitted to the image fusion module and the three-dimensional reconstruction and modeling module, the specific steps are as follows: Step D1, feature extraction: receive the synthesized visible light HDR image I from the dynamic range expansion module HDR,visible (x,y) and the infrared HDR image I HDR,IR (x,y), register the two images and extract color information, texture features and contour information from the visible light image; extract thermal features from the infrared image; Step D2, feature fusion: according to the feature information extracted in step D1, the characteristic value of the visible light image and the infrared image is calculated, and the characteristic value is used to calculate the fusion weight, the specific calculation formula is as follows: f visible (x,y) = a - C(x,y) + β - T(x,y) + γ - E(x,y) f IR (x,y) = δ - TH(x,y) where f visible (x,y) and f IR (x,y) are the feature values of the visible light image and the infrared image, respectively, C(x,y) is the color feature, T(x,y) is the texture feature, E(x,y) is the contour feature, ε, β, γ, δ are weight coefficients, TH(x,y) is the thermal feature, and ω(x,y) is the fusion weight. Step D3, generating a comprehensive image: using the calculated fusion weight to perform a weighted average to generate a comprehensive image, and transmitting the comprehensive image to the image fusion module and the three-dimensional reconstruction and modeling module, the comprehensive image expression is: fused (x, y) = ω(x, y) · I HDR,visible (x, y) + [1 - ω(x, y)] · I HDR,IR (x, y), wherein I fused (x, y) is the pixel value of the comprehensive image at position (x, y), ω(x, y) is the fusion weight, I HDR,visible (x, y) and I HDR,IR (x, y) are the pixel values of the visible light HDR image and the infrared HDR image at position (x, y), respectively.

6. A low-light full-color micro-light night vision reconnaissance system according to claim 5, characterized in that: The three-dimensional reconstruction and modeling module obtains the accurate distance of the target object by laser ranging to generate depth data, combines the depth data with the image data to generate a three-dimensional model, and transmits it to the image fusion module, the specific steps are as follows: Step E1, obtain the distance D between the target object and the sensor by laser emission and reception of return signals, and convert the obtained distance D into depth data D depth (x,y), representing the depth value of the target object corresponding to each pixel position (x,y); Step E2, constructing a three-dimensional point cloud: generating three-dimensional point cloud data by pixel position and depth value, the three-dimensional coordinates of each pixel (x, y) are and all three-dimensional coordinate points are combined into a point cloud set where f x and f y are the focal lengths of the camera in the x and y directions, (X, Y, Z) is the coordinate in the three-dimensional space, D depth (x, y) is the depth value at the pixel (x, y). Step E3, three-dimensional reconstruction: the generated point cloud data is denoised, and a triangle is generated by connecting three non-collinear points in the point cloud to form a three-dimensional mesh T j = (P a , P b , P c ), wherein P a , P b , P c are three non-collinear points selected from the point cloud, constituting three vertices of the triangle, and the color information of the image is mapped from the composite image I fused (x,y) to each triangle of the three-dimensional mesh to generate a three-dimensional model including spatial data and texture information.

7. A low-light full-color micro-light night vision reconnaissance system according to claim 6, characterized in that: In the step E3 of the three-dimensional reconstruction, the color information of the image is removed from the integrated image I fused (x,y) is mapped onto each triangle of the three-dimensional grid, generating a three-dimensional model comprising spatial data and texture information, further comprising the following steps: Step E301, Texture mapping: for each vertex P of the triangle a , P b , P c , the corresponding pixel coordinates are obtained from the composite image I fused (x,y) a from which the color information is obtained: C fused = I a (x a ,y b ), C fused = I b (x b ,y c ), C fused = I c (x c ,y a ), where (x a ,y a ), (x a ,y c ) and (x c ,y a ) are the image coordinates associated with the triangle vertices P b , P c , Step E302, texture coordinates: the texture coordinates (u, v) corresponding to each point are calculated by the barycentric coordinates (λ1, λ2, λ3) = λ1(u a ,v a )+λ2(u b ,v b )+λ3(u c ,v c ), wherein the barycentric coordinates (λ1, λ2, λ3) satisfy λ1+λ2+λ3=1, and a three-dimensional model expression including spatial data and texture information is obtained as: S={T j =(P a ,P b ,P c ),(u a ,v a ),(u b ,v b ),(u c ,v c ),(C a ,C b ,C c )|j=1,2,...,K},K is the total number of triangles generated by connecting all non-collinear points, (C a ,C b ,C c ) is the color value of each point on the triangle, extracted from the image I fused (x,y), and (u a ,v a ), (u b ,v b ), (u c ,v c ) are the texture coordinates of each vertex.

8. A low-light full-color micro-light night vision reconnaissance system according to claim 7, characterized in that: The image fusion module fuses the data of all modules to generate high-quality images and data, the specific steps are as follows: Step S1, data receiving: receiving data from low-light imaging module, dynamic range expansion module, multispectral and infrared fusion module and three-dimensional reconstruction module, and standardizing the data transmitted by each module; Step S2, multi-modal data fusion: fuse image data and spatial data, generate final high-quality images and three-dimensional models according to the fused data, and display full-color night vision images on the screen, provide depth and spatial positioning information, the image data includes visible light image, infrared image, fused image, the spatial data includes depth data, three-dimensional model.

Images