Virtual stereovision imaging array system and method for three-dimensional reconstruction of moving objects
By using a virtual stereo vision imaging array system, and through the collaborative tracking and geometric calibration of multiple cameras and galvanometers, the problem of high-efficiency and high-precision 3D reconstruction of high-speed moving targets under large field-of-view conditions in traditional stereo vision systems has been solved. This achieves simultaneous optimization of high-speed tracking, large-scale observation, and precise detail analysis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INST OF AUTOMATION CHINESE ACAD OF SCI
- Filing Date
- 2026-02-11
- Publication Date
- 2026-06-05
AI Technical Summary
Existing stereo vision systems struggle to achieve efficient and high-precision 3D reconstruction of high-speed moving targets under wide field-of-view conditions. Traditional methods, when increasing frame rate, typically only reduce spatial resolution or shrink the field of view by pixel merging, failing to simultaneously meet the demands of high-speed tracking, wide observation, and high-precision reconstruction.
A virtual stereo vision imaging array system is adopted, which uses multiple cameras and multiple galvanometers for collaborative tracking and geometric calibration. The high-speed angle adjustment of the galvanometers enables a wide range of virtual viewpoint switching, and real-time 3D reconstruction is performed by combining synchronously acquired image information.
It achieves efficient and high-precision 3D reconstruction of high-speed moving targets under large field of view conditions, breaks through the inherent limitations of the field of view of traditional systems, and improves the accuracy and measurement range of 3D measurement.
Smart Images
Figure CN122160490A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision imaging technology, specifically relating to a virtual stereo vision imaging array system and a method for three-dimensional reconstruction of moving targets. Background Technology
[0002] With the rapid development of technologies such as robotics, autonomous driving, and intelligent manufacturing, the demand for high-precision, real-time 3D localization and reconstruction of moving targets is increasing. 3D reconstruction of moving targets has significant application value in target recognition and tracking, robot vision navigation, and industrial automation inspection. Especially in scenarios where the target's motion state is complex and constantly changing, the ability to acquire 3D information in real time and stably is indispensable. For example, bullet trajectory analysis, car crash experiments, robot autonomous navigation, flight trajectory tracking, and dynamic detection and control of industrial targets all place extremely high demands on the speed, accuracy, and robustness of 3D reconstruction.
[0003] Common 3D reconstruction methods include stereo vision, structured light, and time-of-flight (ToF) sensors. Among these, stereo vision has significant advantages due to its independence from active light sources, ability to detect distant targets, and lower hardware costs. It estimates disparity in stereo-corrected binocular images and uses triangulation to obtain 3D geometric information. However, when dealing with high-speed moving targets, traditional stereo vision is limited by insufficient field of view, matching accuracy, and slow response speed, making it difficult to meet the dual requirements of real-time performance and accuracy.
[0004] Existing stereo vision systems are broadly classified into two categories: static vision systems and dynamic vision systems. Static vision systems typically employ fixed camera arrays to expand the field of view, enabling 3D localization and reconstruction of moving targets. While structurally simple, the fixed field of view prevents active tracking of high-speed moving targets, and incomplete reconstruction may occur if the target extends beyond the field of view. In contrast, dynamic vision systems actively track targets by controlling camera posture, making them more suitable for high-speed motion scenarios. However, most dynamic systems rely on gimbals or servo motors for viewpoint adjustment, which have limited response frequency and control precision, making it difficult to achieve stable real-time tracking of high-speed moving objects under large field-of-view conditions.
[0005] In recent years, virtual camera systems based on galvanometers have gradually attracted attention. By adjusting the angle of two high-speed rotatable mirrors, the galvanometer can quickly change the observation direction without moving the camera, achieving wide-range and high-speed field-of-view switching. Such systems combine the structural stability of a fixed camera with the high mobility of the galvanometer mechanism, showing significant potential for tracking high-speed moving targets in a large field of view.
[0006] However, current research on active stereo vision based on galvanometers still struggles to achieve high-precision 3D reconstruction of moving targets. Some existing methods utilize only a single galvanometer and a single camera for near-stereo reconstruction, applicable only to static targets. Other methods employ dual galvanometer systems to track moving objects, but can only measure the depth of the target's center point, failing to reconstruct the complete 3D structure. Therefore, in summary, achieving efficient and high-precision 3D reconstruction of high-speed moving targets under large field-of-view conditions remains a critical unsolved problem. Summary of the Invention
[0007] The purpose of this invention is to address the problems in the prior art by providing a virtual stereo vision imaging array system and a method for 3D reconstruction of moving targets, which takes into account the requirements of a large field of view and high precision. It utilizes the high-speed angle adjustment of the galvanometer to achieve a wide range of virtual viewpoint switching, and completes high-efficiency and high-precision 3D reconstruction of high-speed moving targets under a large field of view through the collaborative tracking and geometric calibration of the dual galvanometer-camera system.
[0008] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a virtual stereoscopic vision imaging array system is provided, including multiple cameras and multiple sets of galvanometers, with one-to-one correspondence between the galvanometers and cameras to form a real-time controllable virtual camera array. By synchronously controlling the galvanometer, each virtual camera can simultaneously track and lock onto the moving target; based on the real-time pose relationship between the virtual cameras and combined with the synchronously acquired image information, real-time 3D reconstruction of the moving target is completed.
[0009] As a preferred embodiment, the camera and the matching galvanometer are jointly calibrated to determine the camera's intrinsic parameters, the galvanometer's inherent parameters, and the spatial positional relationships between the camera and the galvanometer, and between multiple sets of galvanometers.
[0010] As a preferred embodiment, each virtual camera simultaneously captures image sequences of dynamic scenes through real-time synchronous adjustment of its respective galvanometer.
[0011] As a preferred approach, the three-dimensional coordinates of the target are calculated using the triangulation principle, based on the feature matching results of each virtual camera and the relative pose information calculated in real time between virtual cameras.
[0012] As a preferred embodiment, two cameras are set up, and two sets of galvanometers are set up accordingly. After matching them one by one, a pair of binocular images for stereo matching can be generated. The master acquisition system consists of one set of matched galvanometers and cameras, and the slave acquisition system consists of another set of matched galvanometers and cameras. The control commands of the slave acquisition system are obtained based on the measurement results of the master acquisition system.
[0013] Secondly, a method for three-dimensional reconstruction of a moving target based on the aforementioned virtual stereo vision imaging array system is provided, comprising the following steps: The matched galvanometer and camera are jointly calibrated to determine the camera's intrinsic parameters and the virtual camera's extrinsic parameters; Each virtual camera tracks the moving target in real time through the synchronous adjustment of its own galvanometer, acquires images of the moving target, and updates the virtual camera's external parameters in real time. Using images acquired by each virtual camera, stereo matching is performed in each frame. Combined with the real-time updated virtual camera extrinsic parameters, the 3D point cloud of the moving target is calculated to achieve 3D reconstruction.
[0014] As a preferred embodiment, in the step of jointly calibrating the matched galvanometer and camera, one set of matched galvanometers and cameras forms a master acquisition system, and another set of matched galvanometers and cameras forms a slave acquisition system. The slave acquisition system receives control commands output by the master acquisition system through a set of linear synchronous mapping relationships, and simultaneously aligns the two perspectives with the same moving target. The voltage relationship between the galvanometers in the master acquisition system and the slave acquisition system is modeled as an affine transformation, expressed as follows:
[0015] in, , ; Using homogeneous coordinates, the above expression becomes:
[0016] in, ; Rotate both the main viewpoint and the secondary viewpoint sequentially to the same set of N known chessboard corner points. And record the corresponding galvanometer voltage: , ; After stacking all samples, the estimates of A and c can be written as a least squares problem, as shown in the following expression:
[0017] Solving the above least squares problem yields the synchronization mapping relationship between the master mirror and the slave mirror; During the tracking process, the control voltage of the galvanometer in the main acquisition system is known. The control voltage required from the acquisition system galvanometer can be directly calculated using the synchronization mapping matrix. .
[0018] As a preferred embodiment, in the step of determining the camera's intrinsic parameters and the virtual camera's extrinsic parameters, the expression for the camera's intrinsic parameter matrix is as follows:
[0019] In the formula, It is the focal length in the horizontal direction, measured in pixels. It is the focal length in the vertical direction, measured in pixels. Principal point coordinates; Based on the synchronization mapping matrix, the chessboard grid is scanned synchronously in a step-by-step manner. Different voltage combinations are applied to the two galvanometers of the master acquisition system and the slave acquisition system, and the corresponding virtual camera pose and voltage value are recorded during the process. The recorded virtual camera pose and voltage values are stored together in an "extrinsic parameter-voltage lookup table"; given the camera's intrinsic parameters, the pose of each virtual viewpoint is represented as a 4×4 homogeneous transformation matrix, as shown below:
[0020] in, It is a rotation matrix. It is a translation vector; The "external parameter-voltage lookup table" can be used to directly look up values during subsequent moving target tracking.
[0021] As a preferred embodiment, in the steps of each virtual camera collaboratively tracking the moving target, acquiring the image of the moving target, and updating the virtual camera extrinsic parameters in real time through the real-time synchronous adjustment of its respective galvanometer, the main acquisition system acquires the left eye image and detects the position of the moving target. The galvanometer of the acquisition system is switched to the angle corresponding to the position of the moving target to acquire the right eye image, so that the left and right perspectives observe the same moving target at the same time. The galvanometer control voltage of the acquisition system is calculated according to the synchronous mapping relationship.
[0022] As a preferred embodiment, in the step of using the images acquired by each virtual camera to perform stereo matching in each frame and combining the real-time updated virtual camera extrinsic parameters to calculate the three-dimensional point cloud of the moving target and realize three-dimensional reconstruction, the extrinsic parameter matrix between the left and right virtual cameras at any time is obtained according to the "extrinsic parameter-voltage lookup table" so that the left and right images have epipolar geometric consistency after epipolar correction. Let the corrected parallax be... Both cameras have the same focal length. f The principal point coordinates are Using a stereo matching algorithm, pixel-level matching point pairs are extracted from the left and right images, respectively. and Then the coordinates of the target point in three-dimensional space Calculate using the following expression:
[0023] In the formula, Z represents the baseline, and Z represents the depth.
[0024] Compared with the prior art, the present invention has at least the following beneficial effects: For stereo vision reconstruction of moving targets, traditional fixed camera structures struggle to simultaneously meet the multiple demands of high-speed target tracking, a large tracking range, and fine detail resolution. The fundamental reason lies in the inherent "Impossible Triangle" problem of stereo vision systems, where temporal resolution (high-speed tracking capability), spatial resolution (detail resolution capability), and reconstruction range (tracking range) are mutually restrictive. When attempting to increase the frame rate for real-time tracking of high-speed moving targets, traditional fixed camera systems typically can only reduce spatial resolution through pixel binning or narrow the field of view (FOV) to limit the observation range, making it difficult to simultaneously achieve accurate tracking of high-speed moving targets, a wide observation range, and high-precision detail reconstruction. To address this, the virtual stereo vision imaging array system of this invention introduces a combination of high-speed precision galvanometers and fixed cameras to dynamically generate a real-time controllable virtual camera array. This effectively overcomes the limitations of the fixed field of view in traditional systems, achieving simultaneous optimization of performance in high-speed tracking, wide-range observation, and precise detail resolution. The system consists of multiple high-speed cameras and multiple sets of high-speed galvanometers. Through synchronous high-speed control of the galvanometers, the virtual cameras can simultaneously and accurately track and lock onto high-speed moving targets. Based on the real-time pose relationship between virtual cameras and combined with synchronously acquired image information, high-precision real-time 3D reconstruction of the target is achieved, thereby significantly improving the accuracy and measurement range of 3D measurement. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 A schematic diagram of the virtual stereoscopic vision imaging array system according to an embodiment of the present invention; Figure 2 A schematic diagram illustrating the working principle of the virtual stereoscopic vision imaging array system according to an embodiment of the present invention; Figure 3 A schematic diagram illustrating the method for calibrating extrinsic parameters between multiple virtual cameras according to an embodiment of the present invention; Figure 4(a) Image frames of binocular observation of moving targets at different time points in the high-speed tracking experiment of the embodiment of the present invention; Figure 4(b) shows the control voltage variation curve of the dual galvanometers in the high-speed tracking experiment of the present invention within the range of 0 to 6 seconds. Figure 5 Example diagrams of the 3D reconstruction results of a moving doll in different poses are shown in the embodiments of the present invention. Detailed Implementation
[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0028] This invention proposes a virtual stereo vision imaging array system that balances a wide field of view and high precision. Utilizing high-speed angle adjustment of galvanometers, it achieves a wide range of virtual viewpoint switching by "trading time for space," overcoming the limitation of traditional stereo vision systems where field of view and precision are difficult to balance. Through collaborative tracking and geometric calibration of multiple galvanometer-camera systems, it achieves efficient and high-precision 3D reconstruction of high-speed moving targets under a wide field of view. Please refer to [link / reference]. Figure 1 This embodiment of the virtual stereoscopic vision imaging array system includes multiple cameras 1 and multiple sets of galvanometers 2. Each galvanometer 2 is matched one-to-one with a camera 1, forming a real-time controllable virtual camera array. Through synchronous control of the galvanometers 2, each virtual camera simultaneously tracks and locks onto a moving target. Based on the real-time pose relationship between the virtual cameras and combined with synchronously acquired image information, real-time 3D reconstruction of the moving target is completed. Multiple sets of galvanometers 2 are matched with an embedded controller 3, and the embedded controller 3 is matched with each of the multiple cameras 1 via an FPGA 4. An FPGA (Field-Programmable Gate Array) is an integrated circuit that can be configured by the user in the field. In this embodiment, camera 1 is a high-speed camera, and galvanometer 2 is a high-speed galvanometer. The speed range of a high-speed camera is typically from 1000 frames per second to 2 billion frames per second, while the speed of a high-speed galvanometer typically ranges from several meters per second to tens of thousands of points per second. A galvanometer is a special type of oscillating motor where an energized coil generates torque in a magnetic field, driving a rotor (i.e., a reflecting mirror) to deflect. Unlike rotating electric motors, galvanometer rotors have a restoring torque applied to their rotors via mechanical springs or electronic methods. The magnitude of this torque is proportional to the angle at which the rotor deviates from its equilibrium position. When a certain current flows through the coil, and the rotor deflects to a certain angle, the electromagnetic torque and the restoring torque become equal. Therefore, the rotor cannot rotate like a conventional motor; it can only deflect.
[0029] In one possible implementation, the camera 1 and the matched galvanometer 2 are jointly calibrated to determine the intrinsic parameters of the camera 1, the inherent parameters of the galvanometer 2, and the spatial positional relationships between the camera 1 and the galvanometer 2, and between multiple sets of galvanometers 2.
[0030] In one possible implementation, each virtual camera in this embodiment simultaneously captures image sequences of a dynamic scene through real-time synchronous adjustment of its respective galvanometer 2. Furthermore, this embodiment calculates the three-dimensional coordinates of the target based on the feature matching results of each virtual camera and the relative pose information calculated in real-time between the virtual cameras, using the principle of triangulation.
[0031] In one possible implementation, two cameras 1 are provided in this embodiment, and two sets of galvanometers 2 are also provided accordingly. After the galvanometers 2 and cameras 1 are matched one-to-one, binocular image pairs for stereo matching can be generated. The master acquisition system consists of one set of matched galvanometers 2 and camera 1, and the slave acquisition system consists of another set of matched galvanometers 2 and camera 1. The control commands of the slave acquisition system are derived from the measurement results of the master acquisition system.
[0032] Please see Figure 2 , Figure 2 The working principle of the virtual stereo vision imaging array system of this invention is illustrated in an embodiment. Based on this working principle, another embodiment of this invention proposes a method for three-dimensional reconstruction of moving targets, implemented using the virtual stereo vision imaging array system, which mainly includes the following steps: S1. Perform joint calibration on the matched galvanometer 2 and camera 1 to determine the intrinsic parameters of camera 1 and the extrinsic parameters of the virtual camera; S2. Each virtual camera tracks the moving target in real time through the real-time synchronous adjustment of its own galvanometer 2, acquires the image of the moving target, and updates the virtual camera's external parameters in real time. S3. Using the images acquired by each virtual camera, perform stereo matching in each frame of the image, and combine the real-time updated virtual camera extrinsic parameters to calculate the 3D point cloud of the moving target and realize 3D reconstruction.
[0033] In one possible implementation, during the joint calibration of the matched galvanometers 2 and cameras 1 in step S1, one set of matched galvanometers 2 and cameras 1 forms the master acquisition system, and the other set forms the slave acquisition system. To achieve synchronization between the master and slave galvanometers, it is necessary to establish the geometric relationship between the two galvanometers and their corresponding virtual cameras. This embodiment involves three key parts of offline calibration: ① Dual-galvanometer synchronization mapping matrix. (denoted as a 3×3 matrix); ② Intrinsic parameters between different virtual cameras; ③ Extrinsic parameters between different virtual cameras.
[0034] Please see Figure 3During the calibration process of extrinsic parameters between multiple virtual cameras, the calibration plate is scanned sequentially by rotating the galvanometer. O 1, O 2,…, O n The relative poses between each virtual camera are calculated, and these poses and their corresponding voltages are written into an "external parameter-voltage lookup table." This is done during the calibration of the dual-mirror synchronization mapping matrix. At this time, only the active system is responsible for performing target detection. The passive system does not run detection independently, but directly receives control commands from the master system through a set of linear synchronization mapping relationships, thereby aligning both perspectives with the same physical target simultaneously and avoiding redundant calculations from the passive perspective.
[0035] The voltage relationship between galvanometer 2 and the master acquisition system is modeled as an affine transformation, expressed as follows:
[0036] in, , ; Using homogeneous coordinates, the above expression becomes:
[0037] in, ; These parameters are estimated using a checkerboard calibration. Specifically, the main and secondary viewpoints are sequentially rotated to the same set of N known checkerboard corner points. And record the corresponding galvanometer voltage: , ; After stacking all samples, the estimates of A and c can be written as a least squares problem, as shown in the following expression:
[0038] Solving the above least squares problem yields the synchronization mapping relationship between the master mirror and the slave mirror; During the tracking process, the control voltage of the galvanometer 2 of the main acquisition system is known. The control voltage required from the acquisition system galvanometer 2 can be directly calculated using the synchronization mapping matrix. .
[0039] In step S1, when determining the intrinsic parameters of camera 1 and the extrinsic parameters of the virtual camera, since the rotation of the galvanometer does not change the intrinsic parameters of the camera itself, the intrinsic parameters of each camera are calibrated separately. The expression for the intrinsic parameter matrix of camera 1 is as follows:
[0040] In the formula, It is the focal length in the horizontal direction, measured in pixels. It is the focal length in the vertical direction, measured in pixels. Principal point coordinates; Next, the extrinsic parameters between different virtual cameras need to be estimated. As the galvanometer deflection angle changes, the relative poses between the virtual cameras will continuously change. To avoid online recalibration during runtime, an extrinsic parameter-voltage lookup table is pre-calculated. Specifically, based on the synchronization mapping matrix, the checkerboard pattern is scanned synchronously in a step-by-step manner, applying different voltage combinations to the two galvanometers 2 of the master and slave acquisition systems, and recording the corresponding virtual camera poses and voltage values during this process; the recorded virtual camera poses and voltage values are stored together in the extrinsic parameter-voltage lookup table; given the intrinsic parameters of camera 1, the pose of each virtual viewpoint is represented as a 4×4 homogeneous transformation matrix, as shown below:
[0041] in, It is a rotation matrix. It is a translation vector; The invention utilizes an "extrinsic parameter-voltage lookup table" for direct lookup during subsequent moving target tracking. As the system continuously switches virtual camera perspectives, the invention can quickly obtain the corresponding extrinsic parameters. Based on the calibrated intrinsic and extrinsic parameters, the invention uses OpenCV's (Open Source Computer Vision Library) stereoRectify() function (an important function in OpenCV for stereo vision, primarily used to correct images captured by the left and right cameras, ensuring their imaging planes are coplanar and aligned, thus simplifying stereo matching and depth calculation) to calculate the rotation matrix used for binocular correction. , and projection matrix , These results will be used for subsequent disparity estimation and 3D reconstruction.
[0042] In one possible implementation, in step S2, two virtual cameras simultaneously capture image sequences of the dynamic scene through real-time synchronous adjustment of high-speed galvanometers. This step ensures that the acquired multi-view images are strictly consistent in time, thereby providing the necessary temporal consistency for accurate measurement and 3D reconstruction of high-speed moving targets. To stably acquire image pairs of high-speed moving targets at a high frame rate within a large field of view, while reducing computational overhead, this invention designs a dual-galvanometer synchronous detection algorithm, instead of running complete target detection independently on two subsystems. The pseudocode form of this algorithm is as follows:
[0043] Step S2 involves acquiring the left-eye image and detecting the position of the moving target using the main acquisition system. The galvanometer 2 of the slave acquisition system is then switched to the angle corresponding to the moving target's position to acquire the right-eye image, ensuring that the left and right viewpoints observe the same moving target simultaneously. The control voltage of the galvanometer 2 in the slave acquisition system is automatically calculated based on the synchronization mapping relationship. Therefore, the system can capture effective binocular image pairs suitable for stereo matching simultaneously. Since the control commands of the slave system are entirely derived from the measurement results of the main system, the slave system does not need to run target detection or calculate the target centroid position again, thus avoiding redundant calculations and significantly improving efficiency. This scheme can simultaneously maintain image acquisition, image processing, and galvanometer driving actions at a control closed-loop frequency of approximately 100Hz, thereby achieving high frame rate and synchronous detection of high-speed moving targets.
[0044] In one possible implementation, step S3, based on the feature matching results and the relative pose information calculated in real time between the virtual cameras, accurately calculates the three-dimensional coordinates of the target using the principle of triangulation, thereby achieving real-time three-dimensional information recovery and high-resolution spatial structure reconstruction of high-speed moving targets. To achieve real-time three-dimensional reconstruction under a dual-galvanometer system, this embodiment performs stereo matching in each frame of the image and, combined with real-time updated camera extrinsic parameters, calculates a high-precision three-dimensional point cloud.
[0045] The extrinsic parameter matrix between the left and right virtual cameras at any given time is obtained using the "extrinsic parameter-voltage lookup table," ensuring that the left and right images possess epipolar geometric consistency after epipolar correction; let the corrected disparity be... Both cameras 1 have the same focal length f The principal point coordinates are Using a stereo matching algorithm (Monster model), pixel-level matching point pairs are extracted from the left and right images, respectively. and The coordinates P(X,Y,Z) of the target point in three-dimensional space are calculated using the following expression:
[0046] In the formula, Z represents the baseline, and Z represents the depth.
[0047] The following specific examples further illustrate the three-dimensional reconstruction method for moving targets of the present invention: 1. Calibration settings: This invention calibrates the dual-mirror synchronization mapping matrix and the intrinsic and extrinsic parameters between different virtual cameras.
[0048] (i) Synchronization mapping matrix of dual-mirror system : Detection is performed only from the main viewpoint; the drive voltage from the slave viewpoint is determined by the synchronization matrix. The calculation yielded:
[0049] (ii) Intrinsic and extrinsic parameters of the virtual camera: The example intrinsic parameters of the two virtual cameras obtained by this invention are as follows:
[0050]
[0051] To construct an "external parameter-voltage lookup table," this invention performs a serpentine scan on the master galvanometer, gradually changing its pitch direction in 0.1V steps to control the voltage. At the same time, the voltage driven from the galvanometer is adjusted according to the synchronization relationship. and collected approximately 1×10 4 Group-synchronized stereo image pairs.
[0052] Given the intrinsic parameters, this invention estimates the extrinsic parameters of the corresponding virtual camera for each voltage pair. And these external parameters, along with their corresponding voltages, are stored in a lookup table. For example, for and The rotation matrix and translation vector obtained from a certain estimation are as follows:
[0053]
[0054] The extrinsic parameters of different virtual cameras can be calculated in the same way and uniformly written into the "extrinsic parameter-voltage lookup table" to support quick lookup during subsequent tracking.
[0055] Subsequently, this invention combines OpenCV's stereoRectify() and initUndistortRectifyMap() (functions in OpenCV used to pre-compute image correction and distortion correction maps) to calculate the rotation and pixel-by-pixel mapping relationship required for epipolar correction, and performs stereo correction on the acquired image pairs accordingly.
[0056] Please refer to Figures 4(a) and 4(b). To evaluate the robustness of the dual-mirror synchronous detection algorithm, a high-speed tracking experiment was conducted. The test target was a toy with a complex curved surface geometry. Figure 4(a) shows the left and right camera images (from C1 and C2) of the target acquired at different time points. It can be seen that even when the target moves rapidly and irregularly, the system can still stably keep the target in the center of the field of view and continuously obtain clear binocular image pairs without significant motion blur. Figure 4(b) shows the control voltage variation curve of the dual mirrors in the range of 0 to 6 seconds; it can be observed that when the target undergoes sudden rapid movement at certain moments (e.g., around t=2.72s), the control voltage of the mirrors will immediately fluctuate rapidly in response. Nevertheless, the system can still maintain robust tracking performance and always keep the target near the center of the image frame. This shows that the synchronous detection algorithm remains reliable in high-speed, complex motion scenarios.
[0057] To further verify the 3D reconstruction capability of the method of this invention, this invention acquires binocular image pairs of a target with complex textures during motion, and performs stereo 3D reconstruction based on these images to restore its 3D shape. To suppress noise and improve structural continuity, this invention uses weighted least squares (WLS) filtering as post-processing after the reconstruction output. Figure 5 The results are shown under multiple poses: the first three columns sequentially display the left and right camera images, feature matching relationships, and the reconstructed dense point cloud, while the last column provides a magnified view of the point cloud's local details. It can be seen that the method of this invention can effectively recover the fine-grained geometry and clear boundary contours of objects, further demonstrating its advantages in high accuracy and robustness.
[0058] This invention, through the introduction of a combination of a high-speed precision galvanometer and a fixed binocular camera system, dynamically generates a real-time controllable virtual camera array. This effectively overcomes the limitations of traditional systems with fixed fields of view, achieving simultaneous optimization in three aspects: high-speed tracking, wide-range observation, and precise detail resolution. Through synchronous high-speed control of the galvanometer, the virtual cameras can simultaneously and accurately track and lock onto high-speed moving targets. Based on the real-time pose relationship between the virtual cameras and combined with synchronously acquired image information, high-precision real-time 3D reconstruction of the target is achieved, thereby significantly improving the accuracy and measurement range of 3D measurements.
[0059] Another embodiment of the present invention provides an electronic device including a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the three-dimensional reconstruction method for moving targets described in the present invention.
[0060] Another embodiment of the present invention provides a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the three-dimensional reconstruction method for moving targets described in the present invention.
[0061] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.
[0062] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0063] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0064] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0065] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0066] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A virtual stereoscopic vision imaging array system, characterized in that, It includes multiple cameras (1) and multiple galvanometers (2), with one-to-one correspondence between the galvanometers (2) and the cameras (1), forming a real-time controllable virtual camera array; Through the synchronous control of the galvanometer (2), each virtual camera simultaneously tracks and locks onto the moving target; based on the real-time pose relationship between the virtual cameras and combined with the synchronously acquired image information, the moving target is reconstructed in real time.
2. The virtual stereoscopic vision imaging array system according to claim 1, characterized in that, The camera (1) and the matching galvanometer (2) are jointly calibrated to determine the intrinsic parameters of the camera (1), the inherent parameters of the galvanometer (2), and the spatial positional relationships between the camera (1) and the galvanometer (2) and between multiple sets of galvanometers (2).
3. The virtual stereoscopic vision imaging array system according to claim 1, characterized in that, Each virtual camera captures a sequence of images of a dynamic scene simultaneously through real-time synchronous adjustment of its respective galvanometer (2).
4. The virtual stereoscopic vision imaging array system according to claim 1, characterized in that, Based on the feature matching results of each virtual camera and the relative pose information calculated in real time between virtual cameras, the three-dimensional coordinates of the target are calculated using the principle of triangulation.
5. The virtual stereoscopic vision imaging array system according to claim 1, characterized in that, Two cameras (1) are set up, and two sets of galvanometers (2) are set up accordingly. After matching them one by one, a pair of binocular images for stereo matching can be generated. The master acquisition system consists of one set of matched galvanometers (2) and cameras (1), and the slave acquisition system consists of another set of matched galvanometers (2) and cameras (1). The control commands of the slave acquisition system are obtained based on the measurement results of the master acquisition system.
6. A method for three-dimensional reconstruction of a moving target based on the virtual stereo vision imaging array system of claim 1, characterized in that, Includes the following steps: The matched galvanometer (2) and camera (1) are jointly calibrated to determine the intrinsic parameters of camera (1) and the extrinsic parameters of virtual camera; Each virtual camera tracks the moving target in real time through the real-time synchronous adjustment of its own galvanometer (2), acquires the image of the moving target, and updates the virtual camera's external parameters in real time; Using images acquired by each virtual camera, stereo matching is performed in each frame. Combined with the real-time updated virtual camera extrinsic parameters, the 3D point cloud of the moving target is calculated to achieve 3D reconstruction.
7. The method for three-dimensional reconstruction of a moving target according to claim 6, characterized in that, In the step of jointly calibrating the matched galvanometer (2) and camera (1), one set of matched galvanometer (2) and camera (1) forms the main acquisition system, and another set of matched galvanometer (2) and camera (1) forms the slave acquisition system. The slave acquisition system receives the control commands output by the main acquisition system through a set of linear synchronous mapping relationships, and simultaneously aligns the two perspectives with the same moving target. The voltage relationship between the galvanometer (2) between the master acquisition system and the slave acquisition system is modeled as an affine transformation, expressed as follows: in, , ; Using homogeneous coordinates, the above expression becomes: in, ; Rotate both the main viewpoint and the secondary viewpoint sequentially to the same set of N known chessboard corner points. And record the corresponding galvanometer voltage: , ; After stacking all samples, the estimates of A and c can be written as a least squares problem, as shown in the following expression: Solving the above least squares problem yields the synchronization mapping relationship between the master mirror and the slave mirror; During the tracking process, the control voltage of the galvanometer (2) of the main acquisition system is known. The control voltage required from the acquisition system galvanometer (2) can be directly calculated using the synchronization mapping matrix. .
8. The method for three-dimensional reconstruction of a moving target according to claim 7, characterized in that, In the step of determining the intrinsic parameters of camera (1) and the extrinsic parameters of the virtual camera, the expression for the intrinsic parameter matrix of camera (1) is as follows: In the formula, It is the focal length in the horizontal direction, measured in pixels. It is the focal length in the vertical direction, measured in pixels. Principal point coordinates; According to the synchronization mapping matrix, the chessboard is scanned synchronously in a step-by-step manner. Different voltage combinations are applied to the two galvanometers (2) of the master acquisition system and the slave acquisition system, and the corresponding virtual camera pose and voltage value are recorded in the process. The recorded virtual camera pose and voltage values are stored together in the "external parameter-voltage lookup table"; given the intrinsic parameters of the camera (1), the pose of each virtual viewpoint is represented as a 4×4 homogeneous transformation matrix, as shown in the following expression: in, It is a rotation matrix. It is a translation vector; The "external parameter-voltage lookup table" can be used to directly look up values during subsequent moving target tracking.
9. The method for three-dimensional reconstruction of a moving target according to claim 8, characterized in that, In the steps of each virtual camera tracking the moving target, acquiring the image of the moving target, and updating the virtual camera's external parameters in real time through the real-time synchronous adjustment of its respective galvanometer (2), the main acquisition system acquires the left eye image and detects the position of the moving target. The galvanometer (2) of the acquisition system is switched to the angle corresponding to the position of the moving target to acquire the right eye image, so that the left and right perspectives observe the same moving target at the same time. The control voltage of the galvanometer (2) of the acquisition system is calculated according to the synchronous mapping relationship.
10. The method for three-dimensional reconstruction of a moving target according to claim 9, characterized in that, In the step of using the images acquired by each virtual camera to perform stereo matching in each frame and combine the real-time updated virtual camera extrinsic parameters to calculate the three-dimensional point cloud of the moving target and realize three-dimensional reconstruction, the extrinsic parameter matrix between the left and right virtual cameras at any time is obtained according to the "extrinsic parameter-voltage lookup table" so that the left and right images have epipolar geometric consistency after epipolar correction. Let the corrected parallax be... Both cameras (1) have the same focal length. f The principal point coordinates are ; Using a stereo matching algorithm, pixel-level matching point pairs are extracted from the left and right images, respectively. and Then the coordinates of the target point in three-dimensional space Calculate using the following expression: In the formula, Z represents the baseline, and Z represents the depth.