Low-light full-color low-light night vision survey system

Abstract

The invention, which relates to the technical field of image processing, discloses a low-light full-color low-light night vision survey system comprising a low-light imaging module, a dynamic range expansion module, a multispectral and infrared fusion module, a three-dimensional reconstruction and modeling module and an image fusion module. The low-light imaging module obtains visible light and infrared images in a low-light environment and transmits the visible light and infrared images to the dynamic range expansion module; the dynamic range expansion module uses a high dynamic range algorithm to adjust details of a bright part and a dark part of the image, and transmits the adjusted image data to the multispectral and infrared fusion module; the multispectral and infrared fusion module performs multispectral fusion on the visible light and infrared images by using an image processing algorithm, creates a comprehensive image and transmits the comprehensive image to the three-dimensional reconstruction and modeling module; the three-dimensional reconstruction and modeling module combines the depth data with the image data to generate a three-dimensional model, and transmits the three-dimensional model to the image fusion module; and the image fusion module fuses the data of all the modules to generate high-quality images and data.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically to a low-light full-color low-light night vision survey system. Background Art

[0002] At night or in low-light environments, traditional optical survey equipment relies on natural light or artificial lighting to work. However, due to insufficient light, traditional equipment often cannot effectively capture enough light signals, resulting in blurred images, missing details, or even complete inability to identify target objects. This limitation poses a huge challenge to many survey tasks, especially in scenarios that require high-precision measurement, precise navigation, 3D photogrammetry, or environmental monitoring.

[0003] In addition, the visual effect in low-light environment is often not as clear as that in daytime, and the contrast between dark and bright parts is too large, which leads to the compression of the dynamic range of the image, further affecting the presentation of details. More importantly, accurate geographic information, morphological characteristics of target objects, and rapid identification of complex scenes often rely on the color restoration of images, while traditional night vision equipment often only provides monochrome or grayscale images, which cannot accurately reflect the actual situation, bringing great inconvenience to surveyors. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a low-light full-color low-light night vision survey system to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a low-light full-color low-light night vision survey system, comprising a low-light imaging module, a dynamic range extension module, a multi-spectral and infrared fusion module, a three-dimensional reconstruction and modeling module, and an image fusion module;

[0006] The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire a visible light image and an infrared image in a low-light environment, and transmits them to the image fusion module and the dynamic range extension module;

[0007] The dynamic range extension module receives the visible light and infrared images transmitted by the low light imaging module, performs image preprocessing, uses a high dynamic range algorithm to adjust the bright and dark details of the image, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multi-spectral and infrared fusion module;

[0008] The multispectral and infrared fusion module uses an image processing algorithm to perform multispectral fusion on the visible light image and the infrared image, creates a composite image, and transmits it to the image fusion module and the three-dimensional reconstruction and modeling module;

[0009] The 3D reconstruction and modeling module obtains the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a 3D model, and transmits it to the image fusion module;

[0010] The image fusion module fuses the data of all modules to generate high-quality images and data.

[0011] In a preferred embodiment, the low-light imaging module uses a low-light image sensor and an infrared sensor to obtain a visible light image and an infrared image in a low-light environment, and transmits them to an image fusion module and a dynamic range extension module. The specific steps are as follows:

[0012] Step A1, ambient light monitoring: using a light-sensitive sensor to monitor the light intensity of the current environment, and judging whether it is in a low-light environment according to the monitoring result. In a low-light environment, the low-light imaging module activates the low-light image sensor and the infrared sensor;

[0013] Step A2, image acquisition: using a low-light image sensor to convert light signals in the environment into digital images, acquiring visible light images in the low-light environment, detecting thermal radiation generated by objects of different temperatures through an infrared sensor, generating infrared images, and automatically adjusting the exposure time according to changes in ambient light, to obtain multiple visible light and infrared images at different exposure times;

[0014] Step A3, image transmission: Integrate the collected visible light image and infrared image to form a complete data packet, which includes the exposure time information of the image, and transmit the integrated image data to the image fusion module and the dynamic range extension module in the form of digital signals.

[0015] In a preferred embodiment, the dynamic range extension module receives the visible light image and the infrared image acquired by the low light imaging module, performs image preprocessing, uses a high dynamic range algorithm to adjust the bright and dark details of the image, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multi-spectral and infrared fusion module. The specific steps are as follows:

[0016] Step C1, image preprocessing: The visible light image and infrared image received by the low-light imaging module are represented as I visible and I IR , and remove noise from the two received images to improve image quality;

[0017] Step C2, brightness calculation: Calculate the brightness values ​​of the visible light image and the infrared image. The brightness value of the visible light image is: L visible (x,y)=0.2999●R visible (x,y)+0.587·G visible(x,y)+0.114●B visible (x, y); the brightness value of the infrared image is: L IR (x,y)=I IR (x,y); where L visible (x, y) represents the brightness value of the visible light image at position (x, y), R visible (x,y),G visible (x,y),B visible (x, y) represents the pixel value of the red channel, green channel, and blue channel in the visible light image at the position (x, y); L IR (x, y) represents the brightness value of the infrared image at the position (x, y), I IR (x, y) represents the pixel value of the infrared image at the position (x, y). Since the infrared image does not distinguish between color channels, the pixel value of the infrared image is directly used as its brightness value;

[0018] Step C3, HDR synthesis: using multiple visible light and infrared images with different exposure times to perform HDR synthesis, by expanding the dynamic range of the image, enhancing dark details and compressing bright areas, and transmitting the adjusted HDR image data to the image fusion module and the multi-spectral and infrared fusion module, further comprising the following steps:

[0019] Step C301, calculate weights: record the visible light and infrared images at different exposure times as and Among them, m and n represent the indexes of visible light images and infrared images respectively. The weight is calculated according to the brightness and exposure time of each image. The specific calculation formula is as follows:

[0020]

[0021] in, and are the variances of the brightness values ​​of visible light and infrared images, and are the weights of visible light and infrared images, respectively, L max is the maximum brightness of all visible light images, T m is the exposure time of the mth visible light image, is the brightness value of the mth visible light image, L max,IR is the maximum brightness of all infrared images, is the brightness value of the nth infrared image, T n is the exposure time of the nth infrared image;

[0022] Step C302, synthesizing an image: synthesizing a high dynamic range visible light communication and infrared image using the calculated weights, wherein the high dynamic range visible light image expression is: The expression of high dynamic range infrared image is: Among them, I HDR,visible (x, y) represents the pixel value of the synthesized visible light HDR image at position (x, y), I HDR,IR (x,y) represents the pixel value of the synthesized infrared HDR image at position (x,y), and are the weights of visible light and infrared images, respectively. is the pixel value of the mth visible light image at position (x, y), is the pixel value of the nth infrared image at position (x, y);

[0023] Step C303, detail enhancement: Use the adaptive histogram equalization method to perform local contrast enhancement on the synthesized HDR image to improve dark details and compress bright areas. The specific calculation formula is as follows:

[0024] I enhanced (x,y)=I HDR (x,y)+c●(I base (x,y)-I HDR (x,y)

[0025] Among them, I enhanced (x, y) is the image value after detail enhancement, I HDR (x, y) is the pixel value of the synthesized HDR image at position (x, y), I bace is the original image, and c is a constant that controls the enhancement strength.

[0026] In a preferred embodiment, the multispectral and infrared fusion module uses an image processing algorithm to perform multispectral fusion on the visible light image and the infrared image to create a comprehensive image, which is transmitted to the image fusion module and the 3D reconstruction and modeling module. The specific steps are as follows:

[0027] Step D1, feature extraction: receiving the synthesized visible light HDR image I from the dynamic range extension module HDR,visible (x,y) and infrared HDR image I HDR,IR (x, y), register the two images, and extract color information, texture features, and contour information from the visible light image; extract thermal features from the infrared image;

[0028] Step D2, feature fusion: According to the feature information extracted in step D1, the feature values ​​of the visible light image and the infrared image are calculated, and the fusion weight is calculated using the feature values. The specific calculation formula is as follows:

[0029] f visible (x,y)=α·C(x,y)+β·T(x,y)+γ·E(x,y)

[0030] f IR (x,y)=δ·TH(x,y)

[0031]

[0032] Among them, f visible (x,y) and f IR (x, y) are the feature values ​​of visible light image and infrared image respectively, C(x, y) is the color feature, T(x, y) is the texture feature, E(x, y) is the contour feature, ε, β, γ, δ are weight coefficients, TH(x, y) is the thermal feature, ω(x, y) is the fusion weight;

[0033] Step D3, generating a comprehensive image: using the calculated fusion weights for weighted averaging to generate a comprehensive image, and transmitting the comprehensive image to the image fusion module and the 3D reconstruction and modeling module. The comprehensive image expression is: fused (x,y)=ω(x,y)·I HDR,visible (x,y)+[1-ω(x,y)]·I HDR,IR (x,y), where I fused (x, y) is the pixel value of the integrated image at position (x, y), ω(x, y) is the fusion weight, I HDR,visible (x,y) and I HDR,IR (x, y) are the pixel values ​​of the visible light HDR image and the infrared HDR image at position (x, y), respectively.

[0034] In a preferred embodiment, the 3D reconstruction and modeling module obtains the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a 3D model, and transmits it to the image fusion module. The specific steps are as follows:

[0035] Step E1: Obtain the distance D between the target object and the sensor by emitting laser light and receiving the return signal, and convert the obtained distance D into depth data D. depth (x, y), represents the depth value of the target object corresponding to each pixel position (x, y);

[0036] Step E2, constructing a 3D point cloud: Generate 3D point cloud data through pixel position and depth value. The 3D coordinates of each pixel (x, y) are And combine all three-dimensional coordinate points into a point cloud set Among them, f x and fy is the focal length of the camera in the x and y directions, (X, Y, Z) is the coordinate in three-dimensional space, D depth (x,y) is the depth value at pixel (x,y);

[0037] Step E3, 3D reconstruction: De-noise the generated point cloud data, and generate triangles by connecting three non-collinear points in the point cloud to form a 3D mesh T j =(P a ,P b ,P c ), where P a , P b , P c are three non-collinear points selected from the point cloud, forming the three vertices of the triangle, and the color information of the image is taken from the comprehensive image I fused (x, y) is mapped onto each triangle of the three-dimensional mesh to generate a three-dimensional model including spatial data and texture information, further comprising the following steps:

[0038] Step E301, texture mapping: For each triangle vertex P a , P b , P c The corresponding pixel coordinates are from the integrated image I fused Get color information from (x,y): C a =I fused (x a ,y a ), C b =I fused (x b ,y b ), C c =I fused (x c ,y c ), where (x a ,y a )、(x a ,y a ) and (x c ,y c ) is the triangle vertex P a , P b , P c The associated image coordinates;

[0039] Step E302, texture coordinates: Calculate the texture coordinates corresponding to each point through the barycentric coordinates (u, v) = λ 1 (u a ,v a )+λ 2 (u b ,v b )+λ 3(u c ,v c ), where the barycentric coordinates (λ 1 ,λ 2 ,λ 3 ) satisfies λ 1 +λ 2 +λ 3 = 1, the three-dimensional model expression including spatial data and texture information is obtained as follows: S = {T j =(P a ,P b ,P c ),(u a ,v a ),(u b ,v b ),(u c ,v c ),(C a ,C b ,C c )|j=1,2,...,K}, K is the total number of triangles generated by connecting all non-collinear points, (C a ,C b ,C c ) is the color value of each point on the triangle, from image I fused (x,y) extraction, (u a ,v a )、(u b ,v b )、(u c ,v c ) are the texture coordinates of each vertex.

[0040] In a preferred embodiment, the image fusion module fuses the data of all modules to generate high-quality images and data. The specific steps are as follows:

[0041] Step S1, data reception: receiving data from the low-light imaging module, the dynamic range extension module, the multi-spectral and infrared fusion module, and the three-dimensional reconstruction module, and performing standardization processing on the data transmitted by each module;

[0042] Step S2, multimodal data fusion: fuse the image data and the spatial data, generate a final high-quality image and a three-dimensional model based on the fused data, and display a full-color night vision image on the screen to provide a sense of depth and spatial positioning information, wherein the image data includes a visible light image, an infrared image, and a fused image, and the spatial data includes depth data and a three-dimensional model.

[0043] The beneficial effects of the present invention are: by combining high-sensitivity sensors, image enhancement technology, three-dimensional modeling, and efficient data processing and fusion technology, it is possible to provide high-quality full-color images and accurate measurement results in low-light environments. The present invention utilizes high dynamic range, precise positioning, three-dimensional reconstruction and data fusion functions to enable it to perform complex geographic information acquisition, target recognition and environmental monitoring in low-light environments, overcoming the shortcomings of traditional night vision equipment in low-light environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0046] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0047] In the description of the present application, the term "for example" is used to mean "used as an example, illustration or description". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0048] Example 1

[0049] This embodiment provides Figure 1 A low-light full-color low-light night vision survey system is shown, which specifically includes a low-light imaging module, a dynamic range extension module, a multi-spectral and infrared fusion module, a three-dimensional reconstruction and modeling module, and an image fusion module;

[0050] The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire a visible light image and an infrared image in a low-light environment, and transmits them to the image fusion module and the dynamic range extension module;

[0051] The dynamic range extension module receives the visible light and infrared images transmitted by the low light imaging module, performs image preprocessing, uses a high dynamic range algorithm to adjust the bright and dark details of the image, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multi-spectral and infrared fusion module;

[0052] The multispectral and infrared fusion module uses an image processing algorithm to perform multispectral fusion on the visible light image and the infrared image, creates a composite image, and transmits it to the image fusion module and the three-dimensional reconstruction and modeling module;

[0053] The 3D reconstruction and modeling module obtains the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a 3D model, and transmits it to the image fusion module;

[0054] The image fusion module fuses the data of all modules to generate high-quality images and data.

[0055] In this embodiment, the low-light imaging module specifically needs to be explained. The low-light imaging module uses a low-light image sensor and an infrared sensor to obtain a visible light image and an infrared image in a low-light environment, and transmits them to an image fusion module and a dynamic range extension module to improve the usability and accuracy of the image. The specific steps are as follows:

[0056] Step A1, ambient light monitoring: using a light-sensitive sensor to monitor the light intensity of the current environment, and judging whether it is in a low-light environment according to the monitoring result. In a low-light environment, the low-light imaging module activates the low-light image sensor and the infrared sensor;

[0057] Step A2, image acquisition: using a low-light image sensor to convert light signals in the environment into digital images, acquiring visible light images in the low-light environment, detecting thermal radiation generated by objects of different temperatures through an infrared sensor, generating infrared images, and automatically adjusting the exposure time according to changes in ambient light, to obtain multiple visible light and infrared images at different exposure times;

[0058] Step A3, image transmission: Integrate the collected visible light image and infrared image to form a complete data packet, which includes the exposure time information of the image, and transmit the integrated image data to the image fusion module and the dynamic range extension module in the form of digital signals.

[0059] In this embodiment, the dynamic range extension module specifically needs to be explained. The dynamic range extension module receives the visible light image and the infrared image acquired by the low light imaging module, performs image preprocessing, uses a high dynamic range algorithm to adjust the bright and dark details of the image, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multi-spectral and infrared fusion module. The specific steps are as follows:

[0060] Step C1, image preprocessing: The visible light image and infrared image received by the low-light imaging module are represented as I visible and I IR , and remove noise from the two received images to improve image quality;

[0061] Step C2, brightness calculation: Calculate the brightness values ​​of the visible light image and the infrared image. The brightness value of the visible light image is: L visible (x,y)=0.2999·R visible (x,y)+0.587·G visible (x,y)+0.114·B visible (x, y); the brightness value of the infrared image is: L IR (x,y)=I IR (x,y); where L visible (x, y) represents the brightness value of the visible light image at position (x, y), R visible (x,y),G visible (x,y),B visible (x, y) represents the pixel value of the red channel, green channel, and blue channel in the visible light image at the position (x, y); L IR (x, y) represents the brightness value of the infrared image at the position (x, y), I IR (x, y) represents the pixel value of the infrared image at the position (x, y). Since the infrared image does not distinguish between color channels, the pixel value of the infrared image is directly used as its brightness value;

[0062] Step C3, HDR synthesis: using multiple visible light and infrared images with different exposure times to perform HDR synthesis, by expanding the dynamic range of the image, enhancing dark details and compressing bright areas, and transmitting the adjusted HDR image data to the image fusion module and the multi-spectral and infrared fusion module, further comprising the following steps:

[0063] Step C301, calculate weights: record the visible light and infrared images at different exposure times as and Among them, m and n represent the indexes of visible light images and infrared images respectively. The weight is calculated according to the brightness and exposure time of each image. The specific calculation formula is as follows:

[0064]

[0065] in, and are the variances of the brightness values ​​of visible light and infrared images, and are the weights of visible light and infrared images, respectively, L max is the maximum brightness of all visible light images, T m is the exposure time of the mth visible light image, is the brightness value of the mth visible light image, L max,IR is the maximum brightness of all infrared images, is the brightness value of the nth infrared image, T n is the exposure time of the nth infrared image;

[0066] Step C302, synthesizing an image: synthesizing a high dynamic range visible light communication and infrared image using the calculated weights, wherein the high dynamic range visible light image expression is: The expression of high dynamic range infrared image is: Among them, I HDR,visible (x, y) represents the pixel value of the synthesized visible light HDR image at position (x, y), I HDR,IR (x,y) represents the pixel value of the synthesized infrared HDR image at position (x,y), and are the weights of visible light and infrared images, respectively. is the pixel value of the mth visible light image at position (x, y), is the pixel value of the nth infrared image at position (x, y);

[0067] Step C303, detail enhancement: Use the adaptive histogram equalization method to perform local contrast enhancement on the synthesized HDR image to improve dark details and compress bright areas. The specific calculation formula is as follows:

[0068] I enhanced (x,y)=I HDR (x,y)+c·(I base (x,y)-I HDR (x,y)

[0069] Among them, I enhanced (x, y) is the image value after detail enhancement, I HDR (x, y) is the pixel value of the synthesized HDR image at position (x, y), I bace is the original image, and c is a constant that controls the enhancement strength.

[0070] In this embodiment, the multi-spectral and infrared fusion module is specifically required to be explained. The multi-spectral and infrared fusion module uses an image processing algorithm to perform multi-spectral fusion on a visible light image and an infrared image to create a comprehensive image, so that more feature information can be obtained in the same image, which can improve the recognition and positioning capabilities of the target, and transmit it to the image fusion module and the three-dimensional reconstruction and modeling module. The specific steps are as follows:

[0071] Step D1, feature extraction: receiving the synthesized visible light HDR image I from the dynamic range extension module HDR,visible (x,y) and infrared HDR image I HDR,IR (x, y), register the two images to ensure that the two images are aligned, and extract color information, texture features and contour information from the visible light image; extract thermal features from the infrared image;

[0072] Step D2, feature fusion: According to the feature information extracted in step D1, the feature values ​​of the visible light image and the infrared image are calculated, and the fusion weight is calculated using the feature values. The specific calculation formula is as follows:

[0073] f visible (x,y)=α·C(x,y)+β·T(x,y)+γ·E(x,y)

[0074] f IR (x,y)=δ·TH(x,y)

[0075]

[0076] Among them, f visible (x,y) and f IR (x, y) are the feature values ​​of visible light image and infrared image respectively, C(x, y) is the color feature, T(x, y) is the texture feature, E(x, y) is the contour feature, ε, β, γ, δ are weight coefficients, TH(x, y) is the thermal feature, ω(x, y) is the fusion weight;

[0077] Step D3, generating a comprehensive image: using the calculated fusion weights for weighted averaging to generate a comprehensive image, and transmitting the comprehensive image to the image fusion module and the 3D reconstruction and modeling module. The comprehensive image expression is: fused (x,y)=ω(x,y)·I HDR,visible (x,y)+[1-ω(x,y)]·I HDR,IR (x,y), where I fused (x, y) is the pixel value of the integrated image at position (x, y), ω(x, y) is the fusion weight, I HDR,visible (x,y) and I HDR,IR(x, y) are the pixel values ​​of the visible light HDR image and the infrared HDR image at position (x, y), respectively.

[0078] In this embodiment, the three-dimensional reconstruction and modeling module specifically needs to be explained. The three-dimensional reconstruction and modeling module obtains the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a three-dimensional model, and transmits it to the image fusion module, thereby improving the accurate positioning of the object in space. The specific steps are as follows:

[0079] Step E1: Obtain the distance D between the target object and the sensor by emitting laser light and receiving the return signal, and convert the obtained distance D into depth data D. depth (x, y), represents the depth value of the target object corresponding to each pixel position (x, y);

[0080] Step E2, constructing a 3D point cloud: Generate 3D point cloud data through pixel position and depth value. The 3D coordinates of each pixel (x, y) are And combine all three-dimensional coordinate points into a point cloud set Among them, f x and f y is the focal length of the camera in the x and y directions, (X, Y, Z) is the coordinate in three-dimensional space, D depth (x,y) is the depth value at pixel (x,y);

[0081] Step E3, 3D reconstruction: De-noise the generated point cloud data, and generate triangles by connecting three non-collinear points in the point cloud to form a 3D mesh T j =(P a ,P b ,P c ), where P a , P b , P c are three non-collinear points selected from the point cloud, forming the three vertices of the triangle, and the color information of the image is taken from the comprehensive image I fused (x, y) is mapped onto each triangle of the three-dimensional mesh to generate a three-dimensional model including spatial data and texture information, further comprising the following steps:

[0082] Step E301, texture mapping: For each triangle vertex P a , P b , P c The corresponding pixel coordinates are from the integrated image I fused Get color information from (x,y): C a =I fused (x a ,y a ), Cb =I fused (x b ,y b ), C c =I fused (x c ,y c ), where (x a ,y a )、(x a ,y a ) and (x c ,y c ) is the triangle vertex P a , P b , P c The associated image coordinates;

[0083] Step E302, texture coordinates: Calculate the texture coordinates corresponding to each point through the barycentric coordinates (u, v) = λ 1 (u a ,v a )+λ 2 (u b ,v b )+λ 3 (u c ,v c ), where the barycentric coordinates (λ 1 ,λ 2 ,λ 3 ) satisfies λ 1 +λ 2 +λ 3 =1, and is related to the position of the point in the triangle, the three-dimensional model expression including spatial data and texture information is obtained as follows: S = {T j =(P a ,P b ,P c ),(u a ,v a ),(u b ,v b ),(u c ,v c ),(C a ,C b ,C c )|j=1,2,...,K}, K is the total number of triangles generated by connecting all non-collinear points, (C a ,C b ,C c ) is the color value of each point on the triangle, from image I fused (x,y) extraction, (u a ,v a )、(u b ,vb )、(u c ,v c ) are the texture coordinates of each vertex.

[0084] In this embodiment, the image fusion module specifically needs to be explained. The image fusion module fuses the data of all modules to generate high-quality images and data. The specific steps are as follows:

[0085] Step S1, data reception: receiving data from the low-light imaging module, the dynamic range extension module, the multi-spectral and infrared fusion module, and the three-dimensional reconstruction module, and performing standardization on the data transmitted by each module to ensure a unified data format and consistent resolution;

[0086] Step S2, multimodal data fusion: fuse the image data and spatial data to ensure the consistency of image brightness, contrast and color, generate the final high-quality image and three-dimensional model based on the fused data, and display the full-color night vision image on the screen to provide depth perception and spatial positioning information. The image data includes visible light images, infrared images, and fused images, and the spatial data includes depth data and three-dimensional models.

[0087] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0088] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0089] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0093] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A low-light full-color night vision survey system, characterized by: It includes low-light imaging module, dynamic range extension module, multi-spectral and infrared fusion module, 3D reconstruction and modeling module, and image fusion module; The low-light imaging module uses a low-light image sensor and an infrared sensor to acquire a visible light image and an infrared image in a low-light environment, and transmits them to the image fusion module and the dynamic range extension module; The dynamic range extension module receives the visible light and infrared images transmitted by the low light imaging module, performs image preprocessing, uses a high dynamic range algorithm to adjust the bright and dark details of the image, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multi-spectral and infrared fusion module; The multispectral and infrared fusion module uses an image processing algorithm to perform multispectral fusion on the visible light image and the infrared image, creates a composite image, and transmits it to the image fusion module and the three-dimensional reconstruction and modeling module; The 3D reconstruction and modeling module obtains the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a 3D model, and transmits it to the image fusion module; The image fusion module fuses the data of all modules to generate high-quality images and data.

2. The low-light full-color night vision survey system according to claim 1, characterized in that: The low-light imaging module uses a low-light image sensor and an infrared sensor to obtain a visible light image and an infrared image in a low-light environment, and transmits them to an image fusion module and a dynamic range extension module. The specific steps are as follows: Step A1, ambient light monitoring: using a light-sensitive sensor to monitor the light intensity of the current environment, and judging whether it is in a low-light environment according to the monitoring result. In a low-light environment, the low-light imaging module activates the low-light image sensor and the infrared sensor; Step A2, image acquisition: using a low-light image sensor to convert light signals in the environment into digital images, acquiring visible light images in the low-light environment, detecting thermal radiation generated by objects of different temperatures through an infrared sensor, generating infrared images, and automatically adjusting the exposure time according to changes in ambient light, to obtain multiple visible light and infrared images at different exposure times; Step A3, image transmission: Integrate the collected visible light image and infrared image to form a complete data packet, which includes the exposure time information of the image, and transmit the integrated image data to the image fusion module and the dynamic range extension module in the form of digital signals.

3. The low-light full-color night vision survey system according to claim 2, characterized in that: The dynamic range extension module receives the visible light image and infrared image acquired by the low light imaging module, performs image preprocessing, uses a high dynamic range algorithm to adjust the bright and dark details of the image, expands the dynamic range of the image, and transmits the adjusted image data to the image fusion module and the multi-spectral and infrared fusion module. The specific steps are as follows: Step C1, image preprocessing: The visible light image and infrared image received by the low-light imaging module are represented as I visible and I IR , and remove noise from the two received images; Step C2, brightness calculation: Calculate the brightness values ​​of the visible light image and the infrared image. The brightness value of the visible light image is: L visible (x,y)=0.2999●R visible (x,y)+0.587·G visible (x,y)+0.114·B visible (x, y); the brightness value of the infrared image is: L IR (x,y)=I IR (x,y); where L visible (x, y) represents the brightness value of the visible light image at position (x, y), R visible (x,y),G visible (x,y),B visible (x, y) represents the pixel value of the red channel, green channel, and blue channel in the visible light image at the position (x, y); L IR (x, y) represents the brightness value of the infrared image at the position (x, y), I IR (x,y) represents the pixel value of the infrared image at position (x,y); Step C3, HDR synthesis: Use multiple visible light and infrared images with different exposure times to perform HDR synthesis, expand the dynamic range of the image, enhance the dark details and compress the bright area, and transmit the adjusted HDR image data to the image fusion module and the multi-spectral and infrared fusion module.

4. The low-light full-color night vision survey system according to claim 3, characterized in that: The method of using a plurality of visible light and infrared images with different exposure times to perform HDR synthesis, expanding the dynamic range of the image, enhancing dark details and compressing bright areas, and transmitting the adjusted HDR image data to the image fusion module and the multi-spectral and infrared fusion module further includes the following steps: Step C301, calculate weights: record the visible light and infrared images at different exposure times as and Among them, m and n represent the indexes of visible light images and infrared images respectively. The weight is calculated according to the brightness and exposure time of each image. The specific calculation formula is as follows: in, and are the variances of the brightness values ​​of visible light and infrared images, and are the weights of visible light and infrared images, respectively, L max is the maximum brightness of all visible light images, T m is the exposure time of the mth visible light image, is the brightness value of the mth visible light image, L max,IR is the maximum brightness of all infrared images, is the brightness value of the nth infrared image, T n is the exposure time of the nth infrared image; Step C302, synthesizing an image: synthesizing a high dynamic range visible light communication and infrared image using the calculated weights, wherein the high dynamic range visible light image expression is: The expression of high dynamic range infrared image is: Among them, I HDR,visible (x, y) represents the pixel value of the synthesized visible light HDR image at position (x, y), I HDR,IR (x,y) represents the pixel value of the synthesized infrared HDR image at position (x,y), and are the weights of visible light and infrared images, respectively. is the pixel value of the mth visible light image at position (x, y), is the pixel value of the nth infrared image at position (x, y); Step C303, detail enhancement: Use the adaptive histogram equalization method to perform local contrast enhancement on the synthesized HDR image to improve dark details and compress bright areas. The specific calculation formula is as follows: I enhanced (x,y)=I HDR (x,y)+c·(I base (x,y)-I HDR (x,y)) Among them, I enhanced (x, y) is the image value after detail enhancement, I HDR (x, y) is the pixel value of the synthesized HDR image at position (x, y), I bace is the original image, and c is a constant that controls the enhancement strength.

5. The low-light full-color night vision survey system according to claim 4, characterized in that: The multi-spectral and infrared fusion module uses an image processing algorithm to perform multi-spectral fusion on the visible light image and the infrared image to create a comprehensive image, which is transmitted to the image fusion module and the 3D reconstruction and modeling module. The specific steps are as follows: Step D1, feature extraction: receiving the synthesized visible light HDR image I from the dynamic range extension module HDR,visible (x,y) and infrared HDR image I HDR,IR (x, y), register the two images, and extract color information, texture features, and contour information from the visible light image; extract thermal features from the infrared image; Step D2, feature fusion: According to the feature information extracted in step D1, the feature values ​​of the visible light image and the infrared image are calculated, and the fusion weight is calculated using the feature values. The specific calculation formula is as follows: f visible (x,y)=α·C(x,y)+β·T(x,y)+γ·E(x,y) f IR (x,y)=δ·TH(x,y) Among them, f visible (x,y) and f IR (x, y) are the feature values ​​of visible light image and infrared image respectively, C(x, y) is the color feature, T(x, y) is the texture feature, E(x, y) is the contour feature, ε, β, γ, δ are weight coefficients, TH(x, y) is the thermal feature, ω(x, y) is the fusion weight; Step D3, generating a comprehensive image: using the calculated fusion weights for weighted averaging to generate a comprehensive image, and transmitting the comprehensive image to the image fusion module and the 3D reconstruction and modeling module. The comprehensive image expression is: fused (x,y)=ω(x,y)●I HDR,visible (x,y)+[1-ω(x,y)]●I HDR,IR (x,y), where I fused (x, y) is the pixel value of the integrated image at position (x, y), ω(x, y) is the fusion weight, I HDR,visible (x,y) and I HDR,IR (x, y) are the pixel values ​​of the visible light HDR image and the infrared HDR image at position (x, y), respectively.

6. The low-light full-color night vision survey system according to claim 5, characterized in that: The 3D reconstruction and modeling module obtains the precise distance of the target object through laser ranging, generates depth data, combines the depth data with the image data to generate a 3D model, and transmits it to the image fusion module. The specific steps are as follows: Step E1: Obtain the distance D between the target object and the sensor by emitting laser light and receiving the return signal, and convert the obtained distance D into depth data D. depth (x, y), represents the depth value of the target object corresponding to each pixel position (x, y); Step E2, constructing a 3D point cloud: Generate 3D point cloud data through pixel position and depth value. The 3D coordinates of each pixel (x, y) are And combine all three-dimensional coordinate points into a point cloud set Among them, f x and f y is the focal length of the camera in the x and y directions, (X, Y, Z) is the coordinate in three-dimensional space, D depth (x,y) is the depth value at pixel (x,y); Step E3, 3D reconstruction: De-noise the generated point cloud data, and generate triangles by connecting three non-collinear points in the point cloud to form a 3D mesh T j =(P a ,P b ,P c ), where P a , P b , P c are three non-collinear points selected from the point cloud, forming the three vertices of the triangle, and the color information of the image is taken from the comprehensive image I fused (x, y) is mapped to each triangle of the 3D mesh to generate a 3D model including spatial data and texture information.

7. The low-light full-color night vision survey system according to claim 6, characterized in that: In the three-dimensional reconstruction step E3, the color information of the image is obtained from the comprehensive image I fused (x, y) is mapped onto each triangle of the three-dimensional mesh to generate a three-dimensional model including spatial data and texture information, further comprising the following steps: Step E301, texture mapping: For each triangle vertex P a , P b , P c The corresponding pixel coordinates are from the integrated image I fused Get color information from (x,y): C a =I fused (x a ,y a ), C b =I fused (x b ,y b ), C c =I fused (x c ,y c ), where (x a ,y a )、(x a ,y a ) and (x c ,y c ) is the triangle vertex P a , P b , P c The associated image coordinates; Step E302, texture coordinates: Calculate the texture coordinates (u, v) corresponding to each point by using the barycentric coordinates = λ1(u a ,v a )+λ2(u b ,v b )+λ3(u c ,v c ), where the barycentric coordinates (λ1, λ2, λ3) satisfy λ1+λ2+λ3=1, and the expression of the three-dimensional model including spatial data and texture information is: S={T j =(P a ,P b ,P c ),(u a ,v a ),(u b ,v b ),(u c ,v c ),(C a ,C b ,C c )|j=1,2,...,K}, K is the total number of triangles generated by connecting all non-collinear points, (C a ,C b ,C c ) is the color value of each point on the triangle, from image I fused (x,y) extraction, (u a ,v a )、(u b ,v b )、(u c ,v c ) are the texture coordinates of each vertex.

8. The low-light full-color night vision survey system according to claim 7, characterized in that: The image fusion module fuses the data of all modules to generate high-quality images and data. The specific steps are as follows: Step S1, data reception: receiving data from the low-light imaging module, the dynamic range extension module, the multi-spectral and infrared fusion module, and the three-dimensional reconstruction module, and performing standardization processing on the data transmitted by each module; Step S2, multimodal data fusion: fuse the image data and the spatial data, generate a final high-quality image and a three-dimensional model based on the fused data, and display a full-color night vision image on the screen to provide a sense of depth and spatial positioning information, wherein the image data includes a visible light image, an infrared image, and a fused image, and the spatial data includes depth data and a three-dimensional model.

Images