Multispectral depth camera and depth measurement method

Abstract

The invention discloses a multispectral depth camera, which comprises a combined beam splitting system, a multispectral imaging module, a depth imaging module, a precision lens and a data processing unit, the combined light splitting system comprises a dichroscope which is arranged in a light path and is used for dividing incident light into two paths, and the two paths are respectively guided to the main RGB sensor and the multispectral imaging module; the multispectral imaging module comprises a special mosaic optical filter and an image sensor; the depth imaging module comprises a main RGB sensor and an auxiliary RGB sensor; and the data processing unit is arranged behind the multispectral imaging module and the depth imaging module and is used for receiving image electric signals output by the multispectral imaging module and the depth imaging module. The method has the advantages and benefits that the multispectral information and the depth information can be accurately and quickly fused in real time, and the problems that after an independent sensor obtains the multispectral information and the depth information, image registration fusion processing is complex and low in precision are well solved.

Description

Technical Field

[0001] The present invention relates to the field of optical measurement technology, and in particular to a multispectral depth camera and a multispectral and depth information fusion method. Background Art

[0002] With the rapid development of science and technology, image acquisition and processing technology is increasingly used in various fields, especially in machine vision, remote sensing, biomedical imaging, and autonomous driving, which puts forward higher requirements for the richness, accuracy, and real-time performance of image data. Traditional single color or grayscale cameras can no longer meet the needs of these advanced applications. In order to overcome this limitation, multispectral imaging technology has emerged.

[0003] Multispectral imaging can provide richer image information by capturing the reflection or emission characteristics of objects in different spectral bands, which helps to more accurately identify objects, analyze materials, and monitor environmental changes. However, a single multispectral camera usually lacks depth information, which is particularly insufficient in applications that require three-dimensional spatial perception. On the other hand, stereo vision technology, especially depth measurement methods based on binocular stereo vision, has achieved remarkable results in the fields of robot navigation and three-dimensional reconstruction. The binocular stereo vision system can calculate the depth information of each point in the scene, providing a powerful tool for understanding and reconstructing the three-dimensional world.

[0004] However, existing multispectral imaging systems and stereo vision systems are often independent. They collect data separately and then perform complex image registration and fusion processing. This not only increases the processing difficulty and computational cost, but also may affect the accuracy of the results due to registration errors. Therefore, the development of an integrated camera system that can simultaneously achieve multispectral imaging and depth measurement has become a technical problem that needs to be solved urgently. The new method proposed in the present invention can not only simplify the data acquisition process and improve processing efficiency, but also ensure the spatial consistency of image data by sharing the same optical path, thereby simplifying subsequent image registration and fusion work, and providing a more convenient and accurate data basis for advanced image processing and analysis. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a multi-spectral depth camera.

[0006] The present invention provides a multispectral depth camera, comprising: a combined light splitting system, a multispectral imaging module, a depth imaging module, a precision lens and a data processing unit; the combined light splitting system comprises a dichroic mirror, which is arranged in an optical path and is used to split incident light into two paths, which are respectively directed to a main RGB sensor and a multispectral imaging module; the multispectral imaging module comprises a special mosaic filter and an image sensor; the special mosaic filter is arranged in the optical path and has a plurality of partitions, each partition allowing light of a specific spectral band to pass through; the image sensor is located after the special mosaic filter and receives multispectral image information decomposed by the filter; the depth imaging module comprises a main RGB sensor and an auxiliary RGB sensor; the precision lens corresponds to a shared optical path of the main RGB sensor and the multispectral imaging module and an optical path of the auxiliary RGB sensor respectively; the data processing unit is arranged after the multispectral imaging module and the depth imaging module, and receives electrical signals output by the multispectral imaging module and the depth imaging module.

[0007] Furthermore, the combined light splitting system enables the main RGB sensor and the multi-spectral imaging module to share the same optical path through a dichroic mirror, so as to ensure that the images captured by the two are highly consistent in space.

[0008] Furthermore, the optical path of the multi-spectral imaging module includes a special mosaic filter having preset spectral band and transmittance characteristics.

[0009] Furthermore, the precision lenses are respectively arranged at the shared optical path of the main RGB sensor and the multi-spectral imaging module and the optical path entrance of the auxiliary RGB sensor to optimize the clarity of image acquisition.

[0010] Furthermore, the data processing unit further includes processing algorithms such as parallax calculation, auxiliary imaging, image denoising, and spectral correction to complete the synthesis of depth images and improve the quality of multi-spectral images and depth images.

[0011] Furthermore, the multispectral depth camera also includes a synchronization control device for controlling the synchronous operation of the main RGB sensor, the auxiliary RGB sensor and the multispectral imaging module.

[0012] The present invention aims to provide a multispectral depth camera, which, through innovative optical design and component configuration, can simultaneously capture multispectral images and depth information on the same platform, significantly improving the richness and accuracy of image data and simplifying the subsequent data processing process. The core of the present invention is to integrate multispectral imaging technology with binocular vision technology to build an efficient and compact multispectral depth camera system.

[0013] The present invention also provides a depth measurement method comprising the following steps:

[0014] First, the two cameras need to be calibrated to ensure that the imaging parameters of the two cameras (such as focal length, imaging origin, distortion parameters, etc.) have been accurately calibrated; due to the characteristics of the optical lens, the camera has radial distortion in the imaging, which can be determined by three parameters k1, k2, and k3; due to assembly errors, the sensor and the optical lens are not completely parallel, so there is tangential distortion in the imaging, which can be determined by two parameters p1 and p2. The calibration of a single camera is mainly to calculate the camera's internal parameters (focal length f and imaging origin cx, cy, five distortion parameters (generally only k1, k2, p1, p2 need to be calculated, and k3 needs to be calculated for fisheye lenses with particularly large radial distortion)) and external parameters (world coordinates of the calibration object). The calibration of a binocular camera not only requires the internal parameters of each camera, but also requires the measurement of the relative position between the two cameras through calibration (i.e. the rotation matrix R and translation vector t of the right camera relative to the left camera);

[0015] Binocular correction is to eliminate distortion and align the left and right views respectively based on the monocular internal parameter data (focal length, imaging origin, distortion coefficient) and the binocular relative position relationship (rotation matrix and translation vector) obtained after camera calibration, so that the imaging origin coordinates of the left and right views are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned. In this way, any point on one image and its corresponding point on the other image must have the same row number, and the corresponding point can be matched by performing a one-dimensional search in the row;

[0016] Use stereo vision technology to match the images captured by the left and right cameras and find corresponding feature points or feature areas in the two images;

[0017] By matching the corresponding points, calculate the disparity between the left and right images;

[0018] Using the parallax information and the geometric relationship between the cameras, the baseline length of the cameras, and the intrinsic parameters of the cameras, the distance of the object is calculated using the triangulation principle;

[0019] According to the disparity information and camera parameters, the depth is estimated to obtain the distance information from the object to the camera, that is, the depth value;

[0020] Let P be a point on the object to be measured, O R With O T are the optical centers of the two cameras respectively, the imaging points of point P on the two camera sensors are P and P' respectively, f is the focal length, B is the distance between the centers of the two cameras, Z is the required depth information, and d is the parallax of point P in the left and right cameras, then:

[0021] d=|X R -X T|

[0022] According to the principle of similar triangles:

[0023]

[0024] We can get:

[0025]

[0026] In the formula, the focal length f and the camera center distance B can be obtained through calibration. As long as X is obtained R -X T (i.e., the value of parallax d) can be used to obtain the depth value information;

[0027] The purpose of binocular matching is to match the corresponding image points of the same scene in the left and right views in order to obtain a disparity map. Binocular matching is generally considered to be the most difficult and most critical problem in stereo vision. After obtaining the disparity data, the depth value information can be easily calculated using the formula in the above principle.

[0028] Beneficial effects of the present invention

[0029] Compared with the prior art, the advantages and benefits of the present invention are that it can accurately, quickly and in real time fuse multispectral information with depth information (the multispectral image and the depth image come from the same precision lens, the obtained image resolution and the captured picture are completely consistent, and no additional algorithms such as feature point matching are required for fusion, so the captured image is already registered and fused), which well solves the problem of complex and low-precision image registration and fusion processing after independent sensors obtain multispectral information and depth information respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of the structure of a multispectral depth camera provided by an embodiment of the present invention;

[0031] Figure 2 is a three-dimensional structural diagram of a multi-spectral depth camera provided by an embodiment of the present invention;

[0032] Figure 3 is a structural diagram of a front view of a multi-spectral depth camera provided by an embodiment of the present invention;

[0033] Figure 4 is a structural diagram of a side view of a multispectral depth camera provided by an embodiment of the present invention;

[0034] Figure 5 is a structural diagram of a top view of a multispectral depth camera provided by an embodiment of the present invention;

[0035] Figure 6is a structural diagram of a rear view of a multispectral depth camera provided by an embodiment of the present invention;

[0036] The figure marks designed in the above drawings are: 100, precision lens; 110, camera housing; 200, dichroic mirror; 300, mosaic filter; 310, image sensor; 400, main RGB image sensor; 410, auxiliary RGB image sensor; 500, data processing unit; 600, target object. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0038] Multispectral imaging can provide richer image information by capturing the reflection or emission characteristics of objects in different spectral bands. Some applications can be realized by analyzing the spectral information in spectral images. For example, in the field of crop growth detection, the analysis of crop phenotypic parameters can be realized based on multispectral information. However, a single multispectral camera usually lacks depth information, which is particularly insufficient in applications that require three-dimensional spatial perception.

[0039] In the related art, it is difficult to synchronously obtain multispectral information and depth information. Multispectral information and depth information can only be obtained separately, which is complicated and requires matching between the obtained multispectral image and the depth image.

[0040] Based on the problems existing in the related art, the present invention develops an integrated camera system that can simultaneously realize multi-spectral imaging and depth measurement. The implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0041] Reference Figure 1-Figure 6 The figure shows a preferred embodiment of the present invention.

[0042] Example:

[0043] The structure of a multispectral depth camera is as follows: Figure 1 As shown, it includes a precision lens 100, a combined spectroscopic system (implemented by a dichroic mirror 200), a multi-spectral imaging module (including a mosaic filter 300 and an image sensor 310), a depth imaging module (including a main RGB image sensor 400 and an auxiliary RGB image sensor 410), and a data processing unit 500.

[0044] 100. Precision lens: used to focus the incident light to form a clear image. Both lenses are precisely optically designed and manufactured (preferred models are: Onsemi AR0234, OmniVision OV9782) to ensure high transmittance, low distortion and excellent imaging quality, corresponding to the shared optical path of the main RGB sensor and the multispectral imaging module and the optical path of the auxiliary RGB sensor.

[0045] The core component of the combined light splitting system is the dichroic mirror 200; when light enters the camera from the target object 600 through the precision lens 100, the dichroic mirror 200 will separate the light into two parts according to the wavelength of the light: a part of the light with a specific wavelength (such as a part of the visible light band) is reflected to the main RGB image sensor 400; and the other part of the light (including other spectral bands, such as infrared, ultraviolet, etc.) passes through the dichroic mirror 200 and continues to enter the multi-spectral imaging module.

[0046] The multi-spectral imaging module consists of a mosaic filter 300 and an image sensor 310. The mosaic filter 300 is densely covered with filter units of various wavelengths, which are arranged in a specific pattern so that each pixel can capture spectral information of a specific wavelength. When the light passing through the dichroic mirror 200 reaches the mosaic filter 300, light of different wavelengths is absorbed or transmitted by different filter units and finally converted into electrical signals by the image sensor 310. The image sensor 310 is a CMOS sensor with high sensitivity and low noise characteristics to ensure high-quality acquisition of multi-spectral images.

[0047] The depth imaging module consists of a main RGB image sensor 400 and an auxiliary RGB image sensor 410, which are respectively located behind two precision lenses 100 to form a binocular stereo vision system. The main RGB image sensor 400 is responsible for capturing the basic color information of the scene, receiving the light after being split by a dichroic mirror, and converting it into an electrical signal. The auxiliary RGB image sensor 410 maintains a certain baseline distance from the main RGB sensor 400 to obtain a slightly different perspective image. By comparing the corresponding points in the two images and using the principle of triangulation, the depth information of each point on the target object 600 can be calculated.

[0048] The data processing unit 500 is the core control and processing component of the camera. It receives the raw data from the multispectral imaging module and the depth imaging module, and performs complex operations such as preprocessing, spectral unmixing, and depth calculation. The preprocessing stage includes steps such as parallax calculation, auxiliary imaging, image denoising, and distortion correction; spectral unmixing is to separate the multispectral information captured by the mosaic filter into independent spectral bands; and the depth calculation is based on the principle of binocular stereo vision, and the depth map is obtained through parallax calculation. Finally, the data processing unit 500 outputs the multispectral image, depth map, and fused three-dimensional information to the display device and storage medium.

[0049] The depth imaging module uses the binocular stereo vision principle to calculate the depth information of the target object. This principle is based on the fact that when two cameras (or two different perspectives of the same camera) observe the same scene from different positions, the position of the same object in the two camera images will be offset. This offset is called parallax. By measuring the parallax and combining it with the camera's geometric parameters (such as baseline distance, focal length, etc.), the distance from the object to the camera, i.e., the depth information, can be calculated. In a binocular stereo vision system, the relationship between depth information and parallax d can usually be derived through the principle of similar triangles (1):

[0050]

[0051] Among them, the baseline distance between the two lenses is B (that is, the distance between the optical centers of the two cameras), the focal length is f, for a point P in the scene, its projection points on the two lens images are p1 and p2 respectively, the parallax d is the pixel distance between p1 and p2 in the horizontal direction, and Z is the actual distance from point P to the optical center of the camera (that is, depth).

[0052] Considering that the image sensor is located near the focal plane, we can approximate the above equation (1) as equation (2):

[0053]

[0054] In summary, the depth imaging module uses the principle of binocular stereo vision, combined with the camera's geometric parameters and image matching technology, to achieve accurate measurement of the target object's depth information.

[0055] In the specific implementation process, the above components can be adjusted and optimized according to the needs. For example, different specifications of lenses, sensors and filters can be selected according to different application scenarios. In addition, the acquired images need to be optimized through software algorithms to improve image quality.

[0056] During operation, the incident light first enters the camera system through the lens, and is then divided into two paths by a dichroic mirror. One path of light is directly projected onto the main RGB sensor to capture a color image of the scene; the other path of light is decomposed into image information of multiple specific spectral bands through the spectroscopic effect of the mosaic filter, and is received by the detector to form multispectral image data. Since the main RGB sensor and the multispectral imaging module share the same optical path, the images captured by the two are highly consistent in space, which greatly facilitates subsequent image registration and fusion. At the same time, the auxiliary RGB sensor, as another important component of the stereo vision system, works in conjunction with the main RGB sensor. By calculating the parallax information between the two, the system can accurately measure the depth of each point in the scene, thereby constructing a high-precision depth image.

[0057] Through the above embodiments, the present invention provides the design and specific implementation of a multispectral depth camera. The camera can simultaneously acquire multispectral information and depth information, has high image acquisition accuracy and efficiency, and is suitable for various application scenarios requiring multispectral and depth information. For those skilled in the art, the present application may have various changes and variations. All modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of the claims of the present application.

Claims

1. A multispectral depth camera, characterized in that: include: Combined spectroscopic system, multi-spectral imaging module, depth imaging module, precision lens and data processing unit; The combined light splitting system includes a dichroic mirror, which is arranged in the light path and is used to split the incident light into two paths, which are respectively directed to the main RGB sensor and the multi-spectral imaging module; The multispectral imaging module includes a special mosaic filter and an image sensor; the special mosaic filter is arranged in the optical path and has a plurality of partitions, each partition allowing light of a specific spectral band to pass through; the image sensor is located behind the special mosaic filter and receives the multispectral image information decomposed by the filter; The depth imaging module includes a main RGB sensor and an auxiliary RGB sensor; The precision lenses correspond to the shared optical path of the main RGB sensor and the multi-spectral imaging module and the optical path of the auxiliary RGB sensor respectively; The data processing unit is arranged behind the multi-spectral imaging module and the depth imaging module, and receives the image electrical signals output by the multi-spectral imaging module and the depth imaging module.

2. The multispectral depth camera according to claim 1, characterized in that: The combined light splitting system enables the main RGB sensor and the multi-spectral imaging module to share the same optical path through a dichroic mirror, so as to ensure that the images captured by the two are highly consistent in space.

3. The multispectral depth camera according to claim 1, characterized in that: The optical path of the multi-spectral imaging module includes a special mosaic filter having preset spectral band and transmittance characteristics.

4. The multispectral depth camera according to claim 1, characterized in that: The precision lenses are respectively arranged at the shared optical path of the main RGB sensor and the multi-spectral imaging module and the optical path entrance of the auxiliary RGB sensor to optimize the clarity of image acquisition.

5. The multispectral depth camera according to claim 1, characterized in that: The data processing unit further includes disparity calculation, auxiliary imaging, image denoising, and spectral correction processing algorithms to complete the synthesis of depth images and improve the quality of multi-spectral images and depth images.

6. The multispectral depth camera according to claim 1 or 2, characterized in that: It also includes a synchronization control device for controlling the synchronous operation of the main RGB sensor, the auxiliary RGB sensor and the multi-spectral imaging module.

7. A depth measurement method, using the multispectral depth camera according to any one of claims 1 to 6, characterized in that: The depth value is calculated by the following formula, Where B is the baseline distance between the two cameras, f is the focal length, and X R With X T They represent the distances from the imaging points of the left and right cameras to the left imaging plane, and Z is the depth value from the object to the camera.

Images