Three-dimensional motion imaging sensing device integrating laser ranging and monocular vision

Abstract

The invention relates to a three-dimensional motion imaging sensing device integrating laser ranging and monocular vision, which comprises a laser, a camera and a visual axis calibration mirror, the visual axis calibration mirror is a single-sided reflector, the included angle between the reflecting surface of the visual axis calibration mirror and the visual axis of the camera is 45 degrees, and the center of the visual axis calibration mirror is provided with a through hole. The emission laser emitted by the laser and the reflection laser received by the laser pass through the through hole; the device further comprises a light beam controller which is arranged between the optical axis calibration mirror and the target object and used for adjusting the direction of the optical axis of the camera after laser emission and reflection of the optical axis calibration mirror, scanning of all positions of the surface of the target object is completed, and a three-dimensional imaging result containing depth information of the target object is obtained. Compared with the prior art, the method has the advantages that under the condition of not depending on a speedometer and complex software calculation, visible light pixel points with depth information can be quickly acquired in a mode of ensuring that the laser beams and the visual axis of the camera are absolutely and completely parallel and consistent.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of monocular vision-based three-dimensional imaging technology, in particular to a three-dimensional motion imaging perception device integrating laser ranging and monocular vision. BACKGROUND

[0002] In the automatic driving system, the perception system plays a crucial role, and the accuracy and processing speed of its target detection are vital to the performance of the entire system. Common sensors for automatic driving vehicles include cameras, laser radars, millimeter wave radars, infrared thermal imagers, and ultrasonic radars, etc. Each sensor has its unique performance characteristics. Cameras can obtain two-dimensional images of the surrounding environment, and these image information contains rich color and texture, but due to its passive nature, the imaging quality is easily affected by external factors such as light and weather. In addition, cameras cannot provide depth information, and there is a certain limitation in accurately locating targets. In contrast, laser radars collect point cloud information of the environment, which contains accurate three-dimensional coordinates and reflectivity of the object surface, which is very helpful for accurately locating the position of the target in three-dimensional space. However, point cloud data itself is usually sparse, arranged in disorder, and lacks color information, which is not advantageous in tasks such as target classification and signal light recognition.

[0003] In order to overcome the limitations of each sensor and make the perception system more comprehensive and reliable, the fusion of laser radar and visual imaging has become an important solution. This fusion can complement the perception characteristics of both and meet the stability and reliability requirements of the automatic driving system for the perception system. To realize the integration of laser radar and vision, several methods have been proposed in the prior art: Tao Binbin et al. in the document "Design and implementation of laser ranging system based on monocular guidance" proposed a method of imaging through a monocular camera and guiding a fixed single-point laser range finder on a mobile gimbal to construct a three-dimensional laser scanner. In this method, the camera provides color information of the target point, and the spatial position of the target is calculated by the angle of the laser range finder and the gimbal. Although this scheme realizes the integration of cameras and laser radars, it needs complex angle calculation, and the gimbal operation is relatively inconvenient. It is difficult to meet the three-dimensional imaging accuracy of moving targets.

[0004] Another patent document "A catadioptric laser ranging three-dimensional panoramic imaging integrated device" by Jiang Dawei et al. proposed a method of using a ring-shaped refractive prism to refract laser, so that image acquisition and laser ranging share the same mirror surface. This method ensures the strict unity of the optical axis, but still needs to rely on a rotating laser emission probe, on the one hand its structure is relatively complex, on the other hand the rotating laser emission probe inevitably introduces errors during the process, thereby reducing the accuracy of three-dimensional imaging. SUMMARY

[0005] The purpose of this invention is to provide a three-dimensional motion imaging and sensing device that integrates laser ranging and monocular vision. This device can rapidly acquire visible light pixels with depth information by ensuring that the laser beam and the camera's line of sight are perfectly parallel and consistent, without relying on odometers or complex software calculations.

[0006] The objective of this invention can be achieved through the following technical solutions: A three-dimensional motion imaging sensing device integrating laser ranging and monocular vision includes a laser, a camera, and a line-of-sight calibration mirror. The line-of-sight calibration mirror is a single-sided reflector with its reflective surface forming an angle of 45 degrees with the line of sight of the camera. A through hole is provided in the center of the line-of-sight calibration mirror, through which the emitted laser emitted by the laser and the reflected laser received by the laser pass. The device also includes a beam controller, which is located between the line-of-sight calibration mirror and the target object. The beam controller is used to adjust the direction of the emitted laser and the camera's line of sight after reflection by the line-of-sight calibration mirror, so as to complete the scanning of various positions on the surface of the target object and obtain a three-dimensional imaging result of the target object containing depth information.

[0007] The laser includes a laser emitter and a laser detector. The emitted laser light is emitted by the laser emitter and passes through a through-hole in the eye axis calibration mirror. The aperture of the through-hole is large enough to ensure that the laser light is not blocked.

[0008] The through-hole is located in the center of the sight axis calibration mirror.

[0009] The beam controller is a rotating deflection prism, a fast reflector, an eccentric lens, or a MEMS galvanometer. The camera's line of sight after reflection by the line-of-sight calibration mirror is parallel to the reflected laser.

[0010] The sight axis calibration mirror is circular in shape.

[0011] The camera is a monocular camera.

[0012] The device also includes a base and a main housing. The main housing is cylindrical. The camera is located inside the base, and the camera's line of sight coincides with the central axis of the main housing. The laser, the line-of-sight calibration mirror, and the beam controller are all placed inside the main housing. The main housing is located on the base and rotates relative to the base around its own central axis.

[0013] The main housing is driven by a motor.

[0014] The camera is a color camera.

[0015] The device also includes a processor, which is connected to a camera, a laser, and a beam controller. The processor changes the optical path through the beam controller to scan each point of the target object. While acquiring a continuous image sequence, it extracts the pixel motion vector field using optical flow information and fuses it synchronously with the laser TOF ranging results. By comparing the motion information between adjacent frames, it calculates the speed and direction of the target and combines it with laser depth data to reconstruct the three-dimensional motion trajectory.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. By using a single-sided reflector as the line-of-sight calibration mirror and setting a through hole on the line-of-sight calibration mirror for laser penetration, on the one hand, the laser and the camera line of sight can be completely overlapped, avoiding errors caused by adjustment accuracy issues during operation. On the other hand, without relying on odometers and complex software calculations, the rapid acquisition of visible light pixels with depth information can be achieved by ensuring that the laser beam and the camera line of sight are absolutely parallel and consistent.

[0017] 2. The laser emitted by the laser emitter passes through a through-hole in the line-of-sight calibration mirror. The aperture size of the through-hole ensures that the laser is not obstructed, thus preventing energy attenuation. This results in lower energy consumption and increased detection range. The laser emitted by the laser emitter passes unobstructed through the through-hole in the line-of-sight calibration mirror. The aperture size of the through-hole ensures that the laser is not obstructed. This design fundamentally avoids energy attenuation caused by obstruction, directly improving the system's signal-to-noise ratio and absolute ranging accuracy, ensuring reliability when detecting targets at long distances or with low reflectivity.

[0018] 3. By setting up a base and a main housing, the camera is located inside the base, and the camera's line of sight coincides with the central axis of the main housing. This allows the camera's line of sight to remain unchanged and the laser beam to remain overlapping during the rotation of the main housing, thus enabling three-dimensional imaging of targets at various azimuth angles. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a technical schematic diagram; Figure 3 Workflow diagram; Figure 4 The target space results obtained from different sensors and their fusion effect are shown. Among them, (a), (b), and (c) are laser ranging point cloud data, monocular imaging image, and fused and colored point cloud image, respectively; Figure 5 This is a schematic diagram of the imaging principle; The components are: 1. Laser, 2. Axis alignment mirror, 3. Camera, 4. Beam controller, 5. Target object, and 6. Main housing. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0021] A three-dimensional motion imaging sensing device integrating laser ranging and monocular vision, such as Figure 1 and Figure 2 As shown, it includes a laser 1, a camera 3, and a line-of-sight calibration mirror 2. The line-of-sight calibration mirror 2 is a single-sided reflector with its reflecting surface at an angle of 45 degrees to the line of sight of the camera 3. A through hole is provided in the center of the line-of-sight calibration mirror 2, through which the emitted laser emitted by the laser 1 and the reflected laser received by the laser 1 pass. The device also includes a beam controller 4, which is located between the line-of-sight calibration mirror 2 and the target object. The beam controller 4 is used to adjust the direction of the emitted laser and the line-of-sight of the camera 3 after reflection by the line-of-sight calibration mirror 2, so as to complete the scanning of various positions on the surface of the target object and obtain a three-dimensional imaging result of the target object containing depth information.

[0022] By using a single-sided reflector as the line-of-sight calibration mirror 2 and setting a through hole on the line-of-sight calibration mirror 2 for laser penetration, on the one hand, the laser and the camera line of sight can be completely overlapped, without errors caused by adjustment accuracy issues during operation. On the other hand, without relying on odometers and complex software calculations, the rapid acquisition of visible light pixels with depth information can be achieved by ensuring that the laser beam and the camera line of sight are absolutely parallel and consistent.

[0023] Laser 1 includes a laser emitter and a laser detector. The emitted laser light is emitted by the laser emitter and passes through the through hole of the line-of-sight calibration mirror 2. The size of the through hole ensures that the laser light is not blocked.

[0024] Generally, the through hole is located in the center of the sight axis calibration mirror 2, and the sight axis calibration mirror 2 is circular in shape.

[0025] In addition, the beam controller 4 is a rotating deflection prism, a fast reflector, an eccentric lens or a MEMS galvanometer, and the line of sight of the camera 3 after being reflected by the line-of-sight calibration mirror 2 is parallel to the reflected laser.

[0026] In this embodiment, camera 3 is a monocular camera. Generally, camera 3 can be a color camera. In this embodiment, camera 3 is a high frame rate CCD camera. The line of sight is reflected by the line-of-sight alignment mirror 2 and is coaxial with the horizontally emitted laser. The two are pointed to the target through a unified beam controller 4.Figure 5 As shown, based on the principle of optical path reversibility, the visible light reflected by the target object 5 in space will be imaged in the camera 3 through the line-of-sight calibration mirror. The beam controller 4 adjusts the line of sight to enable it to present monocular images at different line-of-sight angles. The beam controller 4 is directly driven by a motor in the control system, and the driving motion signal comprehensively considers the ranging density and the camera's field of view requirements.

[0027] In some other embodiments, the device further includes a base and a main housing. The main housing is cylindrical, with the camera 3 located inside the base and its line of sight coinciding with the central axis of the main housing. The laser 1, the line-of-sight calibration mirror 2, and the beam controller 4 are all housed within the main housing. The main housing is situated on the base and rotates relative to the base around its own central axis. By setting up the base and main housing, and placing the camera inside the base with its line of sight coinciding with the central axis of the main housing, the line of sight of the camera can be maintained and the laser beam overlap can remain unchanged during the rotation of the main housing. This allows for three-dimensional imaging of targets at various azimuth angles. However, this method leads to decreased stability, specifically a decrease in the consistency between the line of sight and the laser beam. Generally, the main housing is driven by a motor.

[0028] like Figure 2 As shown, the workflow of this device includes laser ranging, visual imaging, and motion information extraction. The industrial camera 3, laser 1 (also called a laser transceiver), reflector, and beam controller 4 require joint calibration before use to ensure coordinate system consistency. The parameters of the industrial camera include focal length, optical dimensions, field of view, pixel area, and pixel resolution. The parameters of the laser transceiver include laser wavelength, laser power, measurement accuracy, and angular resolution. The reflector adjusts the camera's line of sight; its parameters include reflectivity, surface roughness, shape, and central aperture size. The beam controller can utilize various optical elements, including but not limited to rotating / deflecting prisms, fast reflectors, eccentric lenses, and MEMS galvanometers. It is important to note that the camera's line of sight after passing through the reflector is parallel to the laser beam and is synchronously controlled by the beam control element to ensure alignment and matching of the ranging information and the resulting image.

[0029] The device also includes a processor, which is connected to the camera 3, the laser 1 and the beam controller 4 respectively. The processor changes the optical path through the beam controller 4 to complete the scanning of each point of the target object. While acquiring a continuous image sequence, it uses optical flow information to extract the pixel motion vector field and fuses it synchronously with the laser TOF ranging results. By comparing the motion information between adjacent frames, the velocity and direction of the target are calculated, and the three-dimensional motion trajectory is reconstructed by combining the laser depth data.

[0030] like Figure 4As shown, during motion imaging, the host computer records the azimuth-elevation angle of the laser beam and the laser transmission-reception time difference to accurately calculate the distance information of the target object, obtaining distance-angle-angle information. Figure 4 (a) The point cloud data generated by the ranging system represents a discrete point in the scene and provides precise 3D coordinates. Simultaneously, the camera obtains a 2D image coaxial with the laser beam through a mirror. Figure 4 (b) The image contains rich texture and color information. Through precise matching and optical flow motion calculation, the system can generate dynamic 3D scene descriptions, superimposing static point clouds with motion vectors to form richer and more accurate target representations. This helps the system to more accurately identify and understand target objects, thereby improving the application performance and effectiveness in fields such as autonomous driving, robot navigation, and environmental monitoring.

[0031] This embodiment achieves coaxial integration of the laser ranging unit and the monocular camera by setting a reflector with a small hole in the center. This ensures that the laser beam path for transmitting and receiving is aligned with the camera's line of sight, and deflection is uniformly controlled by a beam controller, thereby enabling 3D perception from any position and multiple perspectives within a large field of view. This device not only integrates 2D image imaging and depth ranging functions, but also, with the support of optical flow fusion calculations, can perform 3D motion perception of dynamic targets. Since there is no need to move or fix the camera and laser, the overall motion structure is greatly simplified, resulting in a compact and robust system. The design of optical axis alignment and synchronous control significantly reduces the complexity of 3D measurement and motion perception, giving the device greater flexibility and superior imaging performance.

[0032] In its typical operating mode, the device first performs dynamic scanning imaging to quickly acquire sparse point cloud and texture information of the scene, establishing a preliminary 3D scene model. Subsequently, the host computer performs optical flow analysis on the target feature points in the image sequence, extracting their motion trajectory and velocity information, and combines this with laser TOF ranging results for depth correction, achieving precise 3D positioning and dynamic prediction of the target. Based on this, the device can provide real-time obstacle avoidance information for autonomous driving and UAV navigation, and can also perform high-precision ranging and target acquisition for moving targets in security monitoring or military missions. The system has a compact overall structure, few moving parts, and strong robustness. The optical axis alignment and synchronous control design significantly reduces the complexity of 3D motion imaging, giving the device high flexibility and excellent application effects in both static modeling and dynamic tracking, providing technical support for dynamic obstacle avoidance in autonomous driving, UAV interception, and target acquisition missions.

[0033] This invention proposes for the first time an integrated solution for laser point cloud scanning and monocular vision based on a hollow pinhole mirror common optical path imaging structure.

[0034] By creating a through-hole in the center of the reflector, the laser ranging beam is directed directly at the target through the aperture, and the camera imaging beam is reflected by the mirror and is completely collinear with the laser emission axis in space, thus achieving a direct mapping relationship between image pixels and laser rays. This structure realizes a fusion mode of "coaxial, same-focus, and same-time" optical path, fundamentally eliminating the cumulative deviations caused by external parameter drift, time synchronization errors, and installation errors, providing a stable and real-time hardware foundation for dynamic 3D perception.

[0035] Conventional "coaxial designs" are mechanically coaxial (the camera and laser are mounted in the same direction), but they are still different optical paths; reflections or windows in front of the camera lens can cause slight parallax. "Hollow pinhole imaging" allows the laser to pass directly through the aperture, while the camera is reflected by a mirror, achieving true collinearity of light rays. Therefore, it is the only design that can structurally achieve "perfect alignment of pixel rays with laser rays."

[0036] Unlike methods that align images and point clouds through external parameter calibration or software registration, this invention utilizes the optical common-path imaging principle of a hollow pinhole mirror to ensure that the laser ranging ray and the camera imaging ray are physically completely overlapped. This establishes a natural correspondence between pixels and laser rays during the acquisition stage, enabling image-point cloud alignment without the need for registration.

[0037] Compared to traditional beam splitter solutions, the "hollow pinhole reflector" design avoids energy loss and ensures the signal-to-noise ratio of laser ranging and the clarity of camera imaging.

[0038] Compared to off-axis solutions, this method ensures the coaxiality of the laser optical axis and the camera's line of sight at a physical level, greatly reducing the difficulty and error of data fusion.

[0039] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An integrated laser ranging and monocular vision three-dimensional motion imaging perception device, comprising a laser (1), a camera (3) and a visual axis calibration mirror (2), characterized in that, The visual axis calibration mirror (2) is a single mirror, the included angle between the reflecting surface and the visual axis of the camera (3) is 45 degrees, the central part of the visual axis calibration mirror (2) is provided with a through hole, and the emitted laser and the reflected laser received by the laser are both passed through the through hole. The device further comprises a light beam controller (4) arranged between the visual axis calibration mirror (2) and the target object, which is used to adjust the direction of the visual axis of the camera (3) after the emitted laser is reflected by the visual axis calibration mirror (2), complete the scanning of each position on the surface of the target object, and obtain the three-dimensional imaging result of the target object containing depth information. 2.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The laser (1) comprises a laser emitter and a laser detector, the emitted laser emitted by the laser emitter is passed through the through hole of the visual axis calibration mirror (2), and the aperture size of the through hole ensures that the laser is not blocked. 3.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The through hole is located at the center of the visual axis calibration mirror (2). 4.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The light beam controller (4) is a rotating deflection prism, a fast mirror, an eccentric lens or a MEMS mirror, and the visual axis of the camera (3) after being reflected by the visual axis calibration mirror (2) is parallel to the reflected laser. 5.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The visual axis calibration mirror (2) is circular in shape. 6.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The camera (3) is a monocular camera. 7.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The device further comprises a base and a main shell, the main shell is cylindrical in shape, the camera (3) is located in the base, the visual axis of the camera (3) coincides with the central axis of the main shell, the laser (1), the visual axis calibration mirror (2) and the light beam controller (4) are all arranged in the main shell, and the main shell is arranged on the base and rotates around the central axis of the main shell relative to the base. 8.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The main shell is driven by a motor. 9.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The camera is a color camera. 10.The integrated laser ranging and monocular vision three-dimensional motion imaging perception device of claim 1, wherein, The device further comprises a processor connected with the camera (3), the laser (1) and the light beam controller (4), respectively, the processor changes the light path through the light beam controller (4) to complete the scanning of each point of the target object, extracts the pixel motion vector field by using the optical flow information while acquiring the continuous image sequence, synchronously fuses the laser TOF ranging result, compares the motion information between adjacent frames to calculate the speed and direction of the target, and combines the laser depth data to realize the reconstruction of the three-dimensional motion trajectory.

Images