Galvanometer type visual sensor capable of realizing dynamic imaging compensation and dynamic imaging compensation method
By employing a dynamic imaging compensation method using a coaxial optical system and a galvanometer-type vision sensor, and combining a two-dimensional galvanometer with a laser rangefinder, high-definition imaging in dynamic visual measurement is achieved, solving the problems of image distortion and blurring under dynamic conditions and improving measurement accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2026-01-27
- Publication Date
- 2026-05-05
AI Technical Summary
In dynamic visual measurement, existing technologies struggle to achieve high-definition imaging under dynamic conditions, especially when exposure time is long or the target is moving at high speed, which can easily lead to motion blur and image distortion, and existing methods are unable to effectively solve these problems.
By employing a coaxial optical system combined with a two-dimensional galvanometer and a laser rangefinder, and through mathematical models and Kalman filtering and PID control strategies, precise control of the viewing angle and dynamic imaging compensation are achieved. The high dynamic characteristics of the galvanometer are utilized to track the target and maintain synchronous motion, thereby eliminating image distortion and blurring.
It achieves high-definition imaging under dynamic conditions, strictly restores the correspondence between object points and image points, and improves the accuracy and clarity of dynamic measurements.
Smart Images
Figure CN121978703A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of sensor technology, specifically relating to a galvanometer-type visual sensor with dynamic imaging compensation and a dynamic imaging compensation method. Background Technology
[0002] Dynamic visual measurement refers to the observation process where there is relative motion between the observed target and the measurement system. When the exposure time is long or the target moves at high speed, such as in low-light scenes like at night or when the moving vehicle itself experiences significant vibration, motion blur is easily caused. If a rolling shutter camera is used, it will cause even more severe image distortion, making it difficult for existing algorithms to achieve the measurement. How to achieve high-definition imaging under dynamic measurement conditions and solve image distortion and blur is the key to realizing high-precision dynamic visual measurement.
[0003] There are two main approaches to addressing image distortion and blur: the first is software-based, which uses image deblurring algorithms to post-process the image after acquisition to eliminate distortion and blur; the second is hardware-based, which uses a visual stabilization platform to mitigate the effects of motion on measurement. Software methods typically rely on prior or estimated blur kernels for operations such as deconvolution to improve image sharpness and are widely used in perception tasks such as recognition and segmentation. However, because they cannot precisely restore the projection correspondence between spatial objects and image pixels, they are ill-suited for precision measurement tasks.
[0004] Hardware methods can resolve distortion and blurring issues during measurement, making them more suitable for measurement tasks compared to software methods. Existing visual stabilization platforms mainly include inertial platforms and PTZ (Pan-Tilt-Zoom) cameras. However, inertial platforms can only sense and compensate for their own motion, unable to actively capture the dynamics of the observed target, thus failing to compensate for distortion and blurring caused by target motion. In contrast, PTZ cameras can actively sense and follow target rotation, but due to the slow speed of their mechanical turntables, they struggle to track highly dynamic targets and cannot perform imaging compensation. Therefore, using a high-speed perspective transformation device and establishing a dynamic imaging compensation mechanism would effectively solve the distortion and blurring problems in dynamic imaging processes.
[0005] A galvanometer is an actuator that can rapidly deflect light, consisting of a high-speed motor and lightweight mirrors. It is often used in conjunction with lasers for applications such as laser marking and lidar. In recent years, some researchers have also combined galvanometers with cameras to create galvanometer-type vision sensors, changing the field of view by rotating the mirror. However, these solutions have failed to establish a precise viewpoint control model, and therefore are mostly used only for target tracking, unable to achieve dynamic imaging compensation for flexible and maneuvering targets. Summary of the Invention
[0006] To address the distortion and blurring issues that easily occur during dynamic visual measurement, this invention provides a galvanometer-type visual sensor with dynamic imaging compensation and a dynamic imaging compensation method. Utilizing the high dynamic characteristics of the galvanometer, it achieves target tracking, locking, and coordinated motion during dynamic imaging, enabling high-definition imaging under dynamic conditions. This invention mainly comprises three parts: the design of the galvanometer-type visual sensor system with dynamic imaging compensation, a mathematical model, and a tracking compensation method. These provide the hardware design and software framework, the modeling and computational foundation, and the dynamic imaging compensation method, respectively. This invention enables clear imaging during dynamic measurement, providing image assurance for improving the accuracy of dynamic measurements.
[0007] To achieve the above objectives, the present invention adopts the following technical solution:
[0008] A galvanometer-type vision sensor with dynamic imaging compensation includes a high-speed camera, a laser rangefinder, a beam splitter prism, and a two-dimensional galvanometer. The high-speed camera and the laser rangefinder are located on the transmission and reflection surfaces of the beam splitter prism, respectively, forming a coaxial optical system. The laser rangefinder solves the problem of scalelessness in monocular vision sensors and achieves precise control of the viewing angle by combining mathematical models. The two-dimensional galvanometer is set behind the beam splitter prism, reflects the coaxial field of view, and controls the rotation of the mirror to quickly adjust the field of view angle.
[0009] The present invention also provides a dynamic imaging compensation method for a galvanometer-type vision sensor, comprising the following steps:
[0010] Step 1: Use a tracker to track and detect the target position in the image obtained from the current viewpoint, and based on the extrinsic parameter matrix. Calculate its position in the current virtual camera coordinate system Three-dimensional coordinates in;
[0011] Step 2: Based on the mathematical model, unify the coordinates to the reference camera coordinate system through the coordinate transformation relationship between different viewpoints. middle;
[0012] Step 3, in the coordinate system Kalman filtering is used to smooth the tracking results while estimating the target's velocity. The results of Kalman filtering are combined with a model of image deviation to camera rotation angle to calculate the galvanometer rotation angle required to point to the latest target position.
[0013] Step 4: Use a PID control strategy to predict overshoot by introducing integral and differential components, thereby achieving accurate tracking.
[0014] Step 5: During the exposure process, the velocity estimate obtained by Kalman filtering is used to estimate the movement of the target during the exposure process. The galvanometer is then controlled to rotate continuously at the estimated velocity during the camera exposure process, thereby achieving synchronous movement with the target, completing dynamic imaging compensation, and obtaining a clear image.
[0015] Beneficial effects:
[0016] This invention proposes a coaxial optical path structure, establishes a high-precision galvanometer-based vision sensor model, and proposes a high-precision dynamic imaging compensation mechanism, achieving high-definition imaging in dynamic visual measurement processes. This invention utilizes the high dynamic response characteristics of the galvanometer to achieve dynamic imaging compensation at the hardware level, which can strictly restore the correspondence between object points and image points, solving the distortion and blurring problems in dynamic imaging processes, and providing an image foundation for high-precision dynamic visual measurement. Attached Figure Description
[0017] Figure 1 This is a schematic diagram of the system structure of a galvanometer-type vision sensor with dynamic imaging compensation according to the present invention.
[0018] Figure 2 This is a schematic diagram of the system workflow;
[0019] Figure 3 This is a schematic diagram of the system imaging process;
[0020] Figure 4 A schematic diagram for tracking the compensation process;
[0021] Figure 5 This is a flowchart of PID control. Detailed Implementation
[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.
[0023] like Figure 1As shown, the dynamically image-compensated galvanometer-type vision sensor of the present invention includes a high-speed camera (or an infrared camera or a combined infrared and high-speed camera vision sensor, etc.), a laser rangefinder, a beam splitter prism, a two-dimensional galvanometer, and related accessories. The high-speed camera and the laser rangefinder are respectively located on the transmission and reflection surfaces of the beam splitter prism, forming a coaxial optical system. The introduction of the laser rangefinder can effectively solve the scale-free problem of monocular vision sensors, and at the same time, it can achieve precise control of the viewing angle by combining with the mathematical model proposed in this invention. The two-dimensional galvanometer is set behind the beam splitter prism, which can reflect the coaxial field of view and control the rotation of the mirror to quickly adjust the field of view angle.
[0024] During operation, the system's viewpoint can follow the target's rotation and estimate the target's moving speed. It also controls the viewpoint to continuously follow the target's movement during exposure, ensuring that the target is always imaged at the same pixel position, thus fundamentally solving the problem of motion blur.
[0025] like Figure 2 As shown, the workflow of the galvanometer-type vision sensor is as follows: After acquiring an image, an image tracking and detection algorithm is first executed to obtain the target position. Then, based on the deviation between the target and the image center, the galvanometer rotation angle is calculated, and the galvanometer rotation is controlled to keep the target always in the center region of the image, thereby achieving target tracking. Simultaneously, the target's motion speed is estimated during this process, and the galvanometer is controlled to move synchronously with the target during the next camera exposure, ensuring the target is always imaged at the same position on the sensitive element, thus achieving dynamic imaging compensation at the hardware level.
[0026] The mathematical model construction process of the dynamically image-compensated galvanometer vision sensor is as follows:
[0027] The mathematical model proposed in this invention describes the relationship between the galvanometer rotation angle and the change in viewing angle, providing an important mathematical foundation for high-precision target tracking and dynamic imaging compensation. Unlike existing galvanometer camera models, this invention establishes a complete three-dimensional coordinate transformation model through reflection transformation analysis; simultaneously, by combining depth information provided by a laser rangefinder, the three-dimensional position of the target is reconstructed, effectively solving the system deviation problem caused by external rotation. Figure 3 As shown, the reflection from the galvanometer can be used to represent the actual optical system as a virtual optical system, and its imaging process can be described as follows:
[0028] (1)
[0029] in, Let be any non-zero scale factor. It is a distortion-free two-dimensional image point of an object. homogeneous coordinates This is the camera's internal parameter matrix. For mirror parameters The derived extrinsic parameter matrix, For object point In the coordinate system of the two-dimensional galvanometer camera The three-dimensional homogeneous coordinates below.
[0030] Mirror parameters These are a set of parameters describing the state of the galvanometer mirror surface, where , , and These represent the X-axis motor rotation angle, Y-axis motor rotation angle, distance from the camera to the X-axis mirror, and distance between the X-axis mirror and the Y-axis mirror, respectively. After reflection transformation, the galvanometer camera G can be mapped to a virtual camera C. For a single reflection transformation, the single reflection transformation matrix can be determined by knowing only the normal direction of the reflecting surface and the point through which it passes, as shown in equation (2).
[0031] (2)
[0032] in, Let be the three-dimensional homogeneous coordinates of the object point after reflection, and be the rotation matrix. , The mirror normal vector and the translation vector are... , To pinpoint the point on the mirror surface it passes through; Represent a The identity matrix, where the superscript T denotes the transpose of the matrix.
[0033] Based on the mirror parameters, the transformation relationship between the galvanometer camera and the first reflection virtual camera can be established sequentially. The transformation relationship between the first and second reflection virtual cameras. And the external parameter matrix M can be derived from equation (3):
[0034] (3)
[0035] Therefore, based on equation (3), the projection process at any viewing angle can be calculated under the condition that the mirror parameters are determined. Furthermore, since the mirror rotation angle needs to be calculated based on the image deviation during the tracking compensation process, a calculation model from image to rotation angle also needs to be established.
[0036] For the point to be pointed to Let its two-dimensional image homogeneous coordinates be... Then the point to be pointed to 3D coordinates in the current virtual camera coordinate system It can be calculated as:
[0037] (4)
[0038] Where z represents the result obtained from the laser rangefinder measurement. Let be the homogeneous coordinates of the object point in a distortion-free two-dimensional image.
[0039] The control objective during target tracking is to keep the z-axis of the virtual camera coordinate system pointing towards the target. Therefore, it can be determined according to... The coordinate system transformation relationship is accurately calculated, and then the required rotation angle of the galvanometer is calculated.
[0040] like Figure 4 As shown, the present invention also provides a dynamic imaging compensation method for a galvanometer-type vision sensor with dynamic imaging compensation, comprising the following steps:
[0041] Step 1: Use a tracker to track and detect the target position in the image obtained from the current viewpoint, and calculate its position in the current virtual camera coordinate system based on equation (3). Three-dimensional coordinates in;
[0042] Step 2: Based on the mathematical model, the coordinates can be unified to the reference camera coordinate system through the coordinate transformation relationship between different viewpoints. middle.
[0043] Step 3, in the coordinate system Kalman filtering can be used to smooth the tracking results while estimating the target's velocity. Using the Kalman filtering results and combining them with a model of image deviation to camera rotation, the required galvanometer rotation angle for pointing to the target's latest position can be calculated.
[0044] Step 4: Since the target is still in motion during image processing, an overshoot needs to be added to the angle calculated from the observations to meet the requirements for accurate pointing.
[0045] This invention uses, for example Figure 5 The PID control strategy shown achieves accurate tracking by introducing integral and derivative components to predict overshoot. In the PID controller, the proportional term is the deviation between the current galvanometer angle and the previous frame's galvanometer angle, the integral term is the cumulative sum of each proportional term, and the derivative term is the difference between the current and previous proportional terms. The current angle is subtracted from the angle calculated after tracking by the galvanometer camera, and the new angle is obtained through the proportional, integral, and derivative components, then fed into the galvanometer camera for tracking.
[0046] Step 5: Based on this, the velocity calculated using Kalman filtering can be used to estimate the target's movement during the exposure process. Since the camera's exposure time is short, the target can be assumed to be moving at a constant speed during this process. The galvanometer can be controlled to rotate continuously at the estimated speed during the camera's exposure, thereby achieving synchronous movement with the target, completing dynamic imaging compensation, and obtaining a clear image.
Claims
1. A galvanometer-type vision sensor with dynamic imaging compensation, characterized in that, It includes a high-speed camera, a laser rangefinder, a beam splitter prism, and a two-dimensional galvanometer. The high-speed camera and the laser rangefinder are located on the transmission and reflection surfaces of the beam splitter prism, respectively, forming a coaxial optical system. The laser rangefinder solves the problem of scalelessness in monocular vision sensors and achieves precise control of the viewing angle by combining mathematical models. The two-dimensional galvanometer is set behind the beam splitter prism, reflects the coaxial field of view, and controls the rotation of the mirror to quickly adjust the field of view angle.
2. The galvanometer-type visual sensor with dynamic imaging compensation according to claim 1, characterized in that, During operation, the system's viewpoint follows the target's rotation and can estimate the target's moving speed. It also controls the viewpoint to continuously follow the target's movement during exposure, keeping the target always imaged at the same pixel position to prevent motion blur.
3. The galvanometer-type visual sensor with dynamic imaging compensation according to claim 1, characterized in that, After acquiring the image, the image tracking and detection algorithm is first executed to obtain the target position. The galvanometer rotation angle is calculated based on the deviation between the target and the image center, and the galvanometer rotation is controlled to keep the target always in the center area of the image, thereby achieving target tracking.
4. The galvanometer-type visual sensor with dynamic imaging compensation according to claim 3, characterized in that, Estimate the target's speed and control the galvanometer to move synchronously with the target during the next camera exposure, keeping the target always imaged at the same position on the sensitive element, thereby achieving dynamic imaging compensation at the hardware level.
5. The galvanometer-type visual sensor with dynamic imaging compensation according to claim 1, characterized in that, The mathematical model of the dynamically image-compensated galvanometer vision sensor describes the relationship between the galvanometer rotation angle and the change in viewing angle; by analyzing and establishing reflection transformation, a complete three-dimensional coordinate transformation model is established; at the same time, combined with the depth information provided by the laser rangefinder, the three-dimensional position of the target is reconstructed.
6. The galvanometer-type visual sensor with dynamic imaging compensation according to claim 5, characterized in that, The imaging process of the galvanometer is described by equation (1): （1） in, Let be any non-zero scale factor. It is a distortion-free two-dimensional image point of an object. homogeneous coordinates This is the camera's internal parameter matrix. For mirror parameters The derived extrinsic parameter matrix, For object point In the coordinate system of the two-dimensional galvanometer camera The three-dimensional homogeneous coordinates are shown below; Mirror parameters These are a set of parameters describing the state of the galvanometer mirror surface, where , , and These are the X-axis motor rotation angle, Y-axis motor rotation angle, distance from the camera to the X-axis mirror, and distance between the X-axis mirror and the Y-axis mirror, respectively; after reflection transformation, the galvanometer camera G is mapped to a virtual camera C; For a single reflection transformation, the matrix of a single reflection transformation can be determined by knowing only the direction of the normal to the reflecting surface and the point through which it passes, as shown in equation (2). （2） in, Let be the three-dimensional homogeneous coordinates of the object point after reflection, and be the rotation matrix. , The mirror normal vector and the translation vector are... , To pinpoint the point on the mirror surface it passes through; Represent a The identity matrix, where the superscript T denotes the transpose of the matrix; Based on the mirror parameters, that is, the transformation relationship between the galvanometer camera and the first reflection virtual camera established sequentially. The transformation relationship between the first and second reflection virtual cameras. And the external parameter matrix is derived according to equation (3). : （3）。 7. The galvanometer-type visual sensor with dynamic imaging compensation according to claim 6, characterized in that, Based on equation (3), the projection process under any viewing angle is calculated under the condition that the mirror parameters are determined; Furthermore, a computational model for image-to-corner calculation is established, including: For the point to be pointed to Let its two-dimensional image homogeneous coordinates be... Then the point to be pointed to 3D coordinates in the current virtual camera coordinate system The calculation is as follows: （4） Where z represents the result obtained from the laser rangefinder measurement. The homogeneous coordinates of the object point in the undistorted two-dimensional image; according to The coordinate system transformation relationship is accurately calculated, and then the required rotation angle of the galvanometer is calculated.
8. A dynamic imaging compensation method for a galvanometer-type vision sensor with dynamic imaging compensation, characterized in that, Includes the following steps: Step 1: Use a tracker to track and detect the target position in the image obtained from the current viewpoint, and based on the extrinsic parameter matrix. Calculate its position in the current virtual camera coordinate system Three-dimensional coordinates in; Step 2: Based on the mathematical model, unify the coordinates to the reference camera coordinate system through the coordinate transformation relationship between different viewpoints. middle; Step 3, in the coordinate system Kalman filtering is used to smooth the tracking results while estimating the target's velocity. The results of Kalman filtering are combined with a model of image deviation to camera rotation angle to calculate the galvanometer rotation angle required to point to the latest target position. Step 4: Use a PID control strategy to predict overshoot by introducing integral and differential components, thereby achieving accurate tracking. Step 5: During the exposure process, the velocity estimate obtained by Kalman filtering is used to estimate the movement of the target during the exposure process. The galvanometer is then controlled to rotate continuously at the estimated velocity during the camera exposure process, thereby achieving synchronous movement with the target, completing dynamic imaging compensation, and obtaining a clear image.
9. The dynamic imaging compensation method for a galvanometer-type visual sensor with dynamic imaging compensation according to claim 8, characterized in that, In step 1, the transformation relationship between the galvanometer camera and the first reflection virtual camera is established sequentially. The transformation relationship between the first and second reflection virtual cameras. And based on equation (3), the external parameter matrix M is derived: （3）。 10. The dynamic imaging compensation method for a galvanometer-type visual sensor with dynamic imaging compensation according to claim 8, characterized in that, The mathematical model in step 2 describes the relationship between the galvanometer rotation angle and the change in viewing angle; by analyzing and establishing reflection transformation, a complete three-dimensional coordinate transformation model is established; at the same time, combined with the depth information provided by the laser rangefinder, the three-dimensional position of the target is reconstructed. The imaging process of the galvanometer is described as follows: （1） in, Let be any non-zero scale factor. It is a distortion-free two-dimensional image point of an object. homogeneous coordinates This is the camera's internal parameter matrix. For mirror parameters The derived extrinsic parameter matrix, For object point In the coordinate system of the two-dimensional galvanometer camera The three-dimensional homogeneous coordinates are shown below; Mirror parameters These are a set of parameters describing the state of the galvanometer mirror surface, where , , and These are the X-axis motor rotation angle, Y-axis motor rotation angle, distance from the camera to the X-axis mirror, and distance between the X-axis mirror and the Y-axis mirror, respectively. After reflection transformation, the galvanometer camera G is mapped to the virtual camera C. For a single reflection transformation, the single reflection transformation matrix can be determined by knowing only the normal direction of the reflecting surface and the point through which it passes, as shown in equation (2). （2） in, Let be the three-dimensional homogeneous coordinates of the object point after reflection, and be the rotation matrix. , The mirror normal vector and the translation vector are... , To pinpoint the point on the mirror surface it passes through; Represent a The identity matrix, where the superscript T denotes the transpose of the matrix; Based on the mirror parameters, that is, the transformation relationship between the galvanometer camera and the first reflection virtual camera established sequentially. The transformation relationship between the first and second reflection virtual cameras. And based on equation (3), the external parameter matrix M is derived: （3） Therefore, based on equation (3), the projection process under any viewing angle is calculated under the condition of determining the mirror parameters; further, a calculation model from image to corner is established, including: For the point to be pointed to Let its two-dimensional image homogeneous coordinates be... Then the point to be pointed to 3D coordinates in the current virtual camera coordinate system The calculation is as follows: （4） Where z represents the result obtained from the laser rangefinder measurement. The homogeneous coordinates of the object point in the undistorted two-dimensional image; according to The coordinate system transformation relationship is accurately calculated, and then the required rotation angle of the galvanometer is calculated.