A bionic binocular vision tracking device and method driven by an optical wedge group

Abstract

The present invention relates to a bionic binocular vision tracking device and method driven by an optical wedge assembly. The device comprises a host computer, a left camera, a right camera, a rotating optical wedge device, and a calibration assembly. Left and right rotating optical wedge devices are respectively disposed directly in front of the left and right cameras, and the imaging visual axes of the left and right cameras are coaxial with the optical axes of the left and right rotating optical wedge devices. The host computer is connected to the rotating optical wedge device, the camera, and the tracking target. The present invention places a rotating optical wedge device in front of the camera, and while the camera is fixed, the rotating optical wedge is used to expand the camera's field of view. Compared to the prior art, the present invention introduces an optical wedge device drive scheme into a bionic binocular vision tracking model and uses the rotating optical wedge device to establish an adaptive binocular imaging configuration. This device is capable of rapidly tracking, capturing, and flexibly switching targets with a large field of view, high resolution, and low aberration sensitivity. The device also features a compact structure, simple calibration, and high tracking and measurement accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of optical imaging and visual tracking, and in particular to a bionic binocular vision tracking device and method driven by an optical wedge group. Background Art

[0002] The rapid development of technologies such as visual bionics and image processing has given rise to the new field of machine vision, which has been widely applied in intelligent manufacturing systems, intelligent monitoring, medical imaging, aerospace and military applications, and other fields. Within bionic vision research, binocular vision, simulating the human eye, has been a hot topic in machine vision research due to the remarkable dynamic characteristics and comprehensive perceptual capabilities of the human eye. Currently, binocular stereo vision systems typically have fixed left and right cameras, providing capabilities such as depth perception and three-dimensional scene reconstruction. However, due to the limitations of the fixed camera axis, traditional binocular vision systems have a small field of view and lack flexibility and adaptability.

[0003] The following prior art proposes several typical bionic binocular vision stereo devices:

[0004] Prior art, Chinese patent application number CN200910045961.3 discloses a bionic binocular stereoscopic vision device in the field of biomimetic technology. The device's base and a second link, one of the links, are connected by two branches: a first branch's driving rocker arm, one end of which is connected to a motor via a revolute joint, and the other end is connected to one end of the first link via a revolute joint. The other end of the first link is connected to the second link, which is connected to the camera, via a revolute joint. A second branch's driving rocker arm, one end of which is connected to the motor via a revolute joint, and the other end is connected to one end of a third link. The other end of the third link is connected to one end of a U-shaped member, which is equipped with a camera, via a revolute joint. The base and a sixth link, another link equipped with a camera, are also connected by two branches. Using two motors to achieve synchronous movement of the two cameras, the device features a compact structure and a fixed relative position between the two cameras, which reduces the difficulty of system calibration. However, the device's left and right cameras have limited degrees of freedom, resulting in poor flexibility and difficulty in simulating the various functions of both eyes.

[0005] Chinese patent application number CN201611055676.6 discloses a nine-degree-of-freedom binocular bionic eye driven by an optical wedge assembly. The device and method comprise a binocular bionic eye consisting of a left eyeball mechanism, a right eyeball mechanism, and a neck mechanism. The left eyeball mechanism includes a rotatable camera mounted within the eyeball, a first motor for controlling left and right eye movement, and a second motor for controlling up and down eye movement. The left eyeball mechanism is mounted on a bracket. The right eyeball mechanism is identical to the left eyeball mechanism and is mounted on the bracket in a mirror-symmetrical manner. The neck mechanism includes a third neck motor for rotating the binocular bionic eye, a first neck motor for tilting the binocular eye, and a second neck motor for swinging the binocular eye. This device can capture visual information in all scenarios, but the presence of multiple motors also results in a bulky mechanical structure and complex control methods.

[0006] In summary, the existing research on bionic binocular vision has shortcomings such as fixed camera visual axis, lack of flexibility and adaptability, bulky mechanical structure and complex control method. Summary of the Invention

[0007] The present invention aims to overcome the shortcomings of the aforementioned prior art by providing a bionic binocular vision tracking device and method driven by an optical wedge assembly. While the left and right cameras are fixed, the present invention adds rotating optical wedges in front of each lens to enable real-time adjustment of the camera's imaging visual axis. By incorporating human eye movement mechanisms and using a control algorithm to adjust the corresponding rotation angles and rotation speeds of the left and right rotating optical wedges, the device achieves multiple dynamic visual modes, including conjugate motion, anisotropic motion, and steady-state tracking, similar to those of the human eye. This method offers excellent imaging flexibility, dynamic responsiveness, and environmental adaptability.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A bionic binocular vision tracking device driven by an optical wedge group, comprising a host computer, a camera, a rotating optical wedge device and a calibration component;

[0010] The camera includes a left camera and a right camera, and the optical axes of the left camera and the right camera are parallel; the camera is used to obtain a target image;

[0011] The rotating optical wedge device includes a left rotating optical wedge device and a right rotating optical wedge device, the left rotating optical wedge device and the right rotating optical wedge device are respectively arranged in front of the left camera and the right camera, and the rotating optical wedge device is used to adjust the imaging visual axis of the camera;

[0012] The imaging visual axes of the left camera and the right camera are coaxial with the optical axes of the left rotating optical wedge device and the right rotating optical wedge device respectively;

[0013] The calibration assembly includes a planar target and a calibration plate, wherein the crosshairs of the planar target coincide with the centerline of the camera image, and the calibration assembly is arranged in the common field of view of the left camera and the right camera; the calibration assembly is used to calibrate the internal and external parameters between the camera and the rotating optical wedge device;

[0014] The host computer is connected to the camera and the rotating optical wedge device respectively.

[0015] Preferably, the rotating optical wedge device includes an optical wedge assembly and a driving assembly;

[0016] The optical wedge assembly includes a plurality of optical wedge elements;

[0017] The driving assembly is used to drive the rotating optical wedge to expand the field of view of the camera.

[0018] A bionic binocular vision tracking method driven by an optical wedge group comprises the following steps:

[0019] S1. Establish an equivalent relationship between the human eyes and the left and right optical wedge components. Based on the object-image mapping and imaging feedback mechanism of the target of interest in the binocular overlap domain, and with the principle of continuous movement of the visual axis adjustment, the left and right optical wedge components are used to flexibly adjust the visual axis of the imaging perception unit to simulate binocular motion imaging.

[0020] S2. Based on the deflection characteristics of the left and right optical wedge components on the camera imaging axis and the principle of optical path reversibility, an equivalent dynamic virtual binocular camera model is established;

[0021] S3. Establish the coordinate systems of the left camera and the right camera, use the Zhang Zhengyou calibration method to obtain the internal parameters of the left camera and the right camera and the radial distortion coefficient of their lenses, and determine the relative pose parameters of the left camera and the right camera;

[0022] S4. Determine the zero position of the main section of each set of optical wedges using the self-collimation mutual calibration method, calibrate the alignment relationship between the camera and the optical wedge group, and establish the coordinate system of the left optical wedge assembly and the right optical wedge assembly;

[0023] S5. Calibrate the internal parameters of the virtual camera, the posture parameters between the virtual binocular cameras, and the imaging distortion of the dynamic virtual camera according to the dynamic virtual camera model;

[0024] S6. Place the tracking target in the common field of view of the left camera and the right camera. The host computer uses the line of sight adjustment algorithm to adjust the imaging line of sight of the left and right optical wedge components and their corresponding cameras, so that the tracking target is in the center of the common field of view of the binocular cameras.

[0025] S7. Utilizing the distributed perception principle and collaborative control strategy that mimics the physiological mechanism of the human eye, the host computer adjusts the left and right optical wedge components and their corresponding camera imaging axes through a visual tracking algorithm to achieve closed-loop feedback automatic image stabilization and rapid tracking of dynamic targets in the bionic binocular vision system.

[0026] Preferably, the process of the left optical wedge assembly and the right optical wedge assembly in S1 jointly controlling and flexibly adjusting the visual axis of the imaging perception unit to simulate binocular motion imaging includes:

[0027] S11, based on the deflection effect of the optical wedge assembly on the camera imaging visual axis, sequentially obtain the imaging perspectives generated by the optical wedge assembly at different rotation angles and orientations, and use the independent rotational motion of each optical wedge around the optical axis to change the visual axis direction of the imaging unit;

[0028] S12. Eye movements can be mainly divided into conjugate movements, anisotropic movements, and steady-state gaze. According to the movement pattern of the imaging target within the camera's field of view, different wedge elements in the optical wedge assembly are given a certain angle and speed for synchronous rotation, thereby changing the imaging visual axes of the left and right cameras in turn, so that the tracking target is always within the common field of view of the left and right cameras.

[0029] Preferably, the process of establishing an equivalent dynamic virtual binocular camera model in S2 includes:

[0030] S21. If the visual axis direction of the virtual camera is consistent with the visual axis deflection direction of the actual camera, the optical center of the virtual camera is located at the intersection of the visual axis of the virtual camera and the reverse extension line of the reverse tracing light;

[0031] S22. Constructing a pose transformation matrix between the virtual camera and the actual camera under different viewing axis directions according to the actual camera position and the optical wedge element parameters;

[0032] S23. Combining the optical center position and spatial pose parameters of the virtual camera, the visual axis pointing sequence of the two fixed cameras is equivalent to an infinite set of virtual binocular vision arrays, that is, an equivalent dynamic virtual binocular camera model.

[0033] Preferably, the process of determining the relative pose parameters of the left camera and the right camera in S3 includes:

[0034] S31, placing a calibration plate in the common field of view of the left camera and the right camera, and changing the position of the calibration plate so that the left camera and the right camera respectively capture several groups of left and right images;

[0035] S32: Taking the left camera coordinate system as the world coordinate system, calculate the internal parameters of the left camera and the right camera, the radial distortion coefficient of the lens, and the relative position of the left camera and the right camera according to the left and right images.

[0036] Preferably, the process of determining the zero position of the main cross section of each set of optical wedges and calibrating the alignment relationship between the camera and the optical wedge group in S4 includes:

[0037] S41. Determine the zero position of the main cross-sections of the left and right optical wedge assemblies using the autocollimation mutual calibration method, perform reference positioning, adjust the reticles of the two collimators so that their horizontal lines fall within the main cross-section of the reference optical wedge, and record this position.

[0038] S42, replacing the reference optical wedge with the optical wedge group to be calibrated for installation and calibration, using the collimator scribed lines as a reference to calibrate and obtain the initial position of the main cross section of each optical wedge in the optical wedge group;

[0039] S43, coaxially mounting the left and right optical wedge assemblies with calibrated main cross sections with the left and right cameras, removing the optical wedge assembly in front of the left camera, and arbitrarily placing a planar target within the common field of view of the left and right cameras to control direct imaging of the left camera and refraction imaging of the right camera through the double optical wedges;

[0040] S44, based on the posture relationship between the left camera and the right camera, extract the reference coordinates of the calibration points from the left camera's direct view image and transfer them to the right camera's coordinate system, and at the same time predict the three-dimensional coordinates of the calibration points from the right camera's refraction imaging results;

[0041] S45. Determine the alignment relationship between the right camera and its optical wedge group by minimizing the deviation between the reference datum and the model prediction. Similarly, use this method to calibrate the alignment relationship between the left camera and its optical wedge group.

[0042] Preferably, the process of calibrating the internal parameters of the virtual camera, the posture parameters between the virtual binocular cameras, and the imaging distortion of the dynamic virtual camera in S5 includes:

[0043] S51, based on the image mapping relationship between the virtual camera and the actual camera, any point on the virtual image plane is mapped to a point on the actual image plane that has been corrected for lens distortion, and a pinhole imaging model of the left and right virtual cameras is established according to the theory of geometric optics;

[0044] S52. Based on the relative posture relationship between the left camera and the right camera and the control rule of the optical wedge group on the imaging visual axis direction, the imaging visual axis direction and the projection center position of the left camera and the right camera corresponding to the virtual camera are derived respectively, and the coordinate transformation relationship and relative posture parameters between the left virtual camera and the right virtual camera are obtained;

[0045] S53, using a virtual camera projection ray reverse tracing method to derive a mathematical mapping relationship between the virtual image plane and the actual image plane, and establish a nonlinear imaging distortion fitting model;

[0046] S54, using the coincidence constraint between the virtual camera projection light and the initial incident light of the optical wedge group, establishing an optimization objective function for the virtual camera imaging distortion coefficient;

[0047] S55. Based on the distribution and evolution characteristics of imaging distortion under different prism rotation angle combinations, an optimization objective function is formed with multi-order distortion coefficients as variables and the minimum ray tracing deviation as the criterion.

[0048] Preferably, the process of fast tracking the dynamic target in S7 includes:

[0049] S71. Based on the motion trajectory description and projection transformation matrix of the virtual binocular camera, establish the spatiotemporal coupling constraint relationship of the binocular image sequence, and construct the visual axis adjustment algorithm for different motion modes such as bionic binocular conjugation, anisotropy, and gaze;

[0050] S72. Extract SURF feature motion flow field from binocular camera image sequences, and combine visual saliency model and spatial information guidance mechanism to perform foreground region segmentation and dynamic target recognition;

[0051] S73, using a recursive least squares algorithm to estimate target pose parameters and establish a motion trajectory prediction model, and obtaining image deviation information of the current target relative to the center of the field of view of the left camera and the right camera;

[0052] S74. Using the triangulation principle, the absolute positioning information of the target is estimated from the deviations of the left and right images. The visual tracking algorithm uses the optical wedge group inverse solution algorithm to solve the rotation angle parameters of the left and right optical wedge components to ensure that the centers of the two fields of view can synchronously lock onto the target.

[0053] S75. Substitute the angle parameters of the optical wedge group into the state equation of the control system. Under the energy constraint of dual-field collaborative control, generate the joint optimal estimate of multiple control quantities such as the angular velocity and angular acceleration of the left and right optical wedge components, establish a closed-loop process, and realize continuous, stable, and smooth target tracking.

[0054] Preferably, the inverse solution algorithm for the optical wedge group adopts one of a table lookup method, an approximation method or an iterative method.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] 1. The present invention provides a rotating optical wedge device in front of each left and right camera. By adjusting the imaging visual axis of the camera through the rotating optical wedge device, a bionic binocular motion mode can be achieved. This completely abandons the typical binocular motion mode that uses muscle simulation or complex mechanism combination, greatly simplifies the structure of the bionic eyeball, and achieves a significant improvement in the camera's field of view and imaging flexibility when the camera is fixed. It can achieve rapid tracking, capture, and flexible switching of targets with a large field of view, high resolution, and low aberration sensitivity, meeting the function of simulating the human eye's binocular dynamic visual information collection and transmission. At the same time, it has a compact structure, simple calibration, and high tracking and measurement accuracy.

[0057] 2. The two fixed cameras in the present invention can be equivalent to an infinite set of virtual binocular vision arrays, forming an adaptive binocular configuration mechanism, thereby providing a rich selection mode for bionic binocular movement.

[0058] 3. Since the two cameras are fixed, a single calibration of internal and external parameters can meet the binocular imaging needs of various eye movement patterns without the need for repeated calibration, which simplifies the calibration process of dynamic binocular vision and makes the control method simple.

[0059] 4. The present invention adopts a refractive axis adjustment mode, and there is a large reduction ratio between the prism rotation angle and the light beam refraction angle, which has high axis pointing accuracy and reduces the influence of mechanical error disturbance on the axis adjustment.

[0060] 5. The present invention adopts the forward and inverse solution theory of the optical wedge device, which can accurately control the optical wedge device to adjust the direction of the visual axis. Combined with an effective collaborative control strategy, it can track the visual target and thus obtain binocular stereo vision information. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a structural schematic diagram of the present invention;

[0062] Figure 2 Schematic diagram of the relationship between the camera and the virtual camera pose;

[0063] Figure 3 Flow chart for camera and optical wedge system calibration;

[0064] Figure 4 Schematic diagram of camera and virtual camera pose calibration;

[0065] Figure 5 Schematic diagram of the equivalent motion pattern of the human eye and double optical wedges;

[0066] Figure 6 It is a bionic dynamic binocular vision imaging and tracking solution;

[0067] Figure 7 This is the flow chart of the iterative algorithm for dual-optical wedge line-of-sight tracking.

[0068] Reference numerals:

[0069] 1-left camera; 2-left rotating double optical wedge; 3-right camera; 4-right rotating double optical wedge; 5-host computer; 6-calibration component; 61-calibration plate; 62-planar target; 7-tracking target; 8-left virtual camera; 9-right virtual camera; (a)-conjugate motion; (b)-anisotropic motion; (c)-steady-state gaze. DETAILED DESCRIPTION

[0070] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0071] Example 1

[0072] like Figure 1 As shown, this embodiment proposes an optical wedge group driven bionic binocular vision tracking device, including a host computer 5, a camera, a rotating optical wedge device and a calibration component 6. Specifically, it includes a left camera 1, a left rotating double optical wedge 2, a right camera 3, a right rotating double optical wedge 4, a host computer 5 and a calibration component 6;

[0073] The left camera 1 and the right camera 3 are arranged around the tracking target 7, and the optical axes of the two cameras are placed in parallel; the left rotating double optical wedge 2 and the right rotating double optical wedge 4 are respectively arranged in front of the left camera 1 and the right camera 3; and the visual axes of the left and right cameras and the optical axes of the left and right rotating optical wedge devices are kept coaxial.

[0074] The calibration assembly 6 includes a calibration plate 61 and a plane target 62, which are arranged in the common field of view of the left camera 1 and the right camera 3; the crosshairs of the plane target 62 coincide with the center line of the camera image. In this embodiment, the same plane target is used to calibrate the left camera and the right camera in sequence. In other embodiments, two plane targets can be used to calibrate the left camera and the right camera respectively.

[0075] The host computer 5 is in communication connection with the left camera 1 , the right camera 3 , the left-rotating double optical wedge 2 , the right-rotating double optical wedge 4 and the tracking target 7 . In this embodiment, the tracking target 7 is a robot as an example.

[0076] like Figure 2 As shown, the dual-optical wedge-driven binocular stereo vision system can be equated to a dynamic virtual binocular camera system, improving the simplicity and flexibility of dynamic visual tracking control and information processing. The imaging axis of left camera 1, passing through the left-rotating dual-optical wedge 2, can be equivalent to the left virtual camera 8 looking directly at the target. Similarly, the imaging axis of right camera 3, passing through the right-rotating dual-optical wedge 4, can be equivalent to the right virtual camera 9 looking directly at the target.

[0077] In this embodiment, the internal parameters of the left and right cameras, such as focal length and resolution, and the external position parameters of the corresponding rotating optical wedge device can be synchronously adjusted according to changes in specific application scenarios. The models and parameters of the left and right cameras, as well as the external position parameters of the corresponding rotating optical wedge device, can be completely identical, or they can be selected differently based on actual needs.

[0078] Preferably, the optical wedge parameters and arrangement of the left-rotating optical wedge device and the right-rotating optical wedge device are completely the same, but can also be selected differently according to actual needs.

[0079] The rotating optical wedge device includes an optical wedge assembly and a drive assembly. The optical wedge assembly includes multiple optical wedge elements of varying numbers, whose optical parameters and layout can be matched and adjusted according to the field of view requirements; the drive assembly can adopt a torque motor direct drive or gear drive, synchronous belt drive, worm gear drive, etc.

[0080] This embodiment places a rotating optical wedge device in front of the camera. With the camera fixed, the rotating optical wedge expands the camera's field of view. This constructs a three-degree-of-freedom monocular visual motion model based on the dual optical wedge visual axis orientation, motion dimension, and tracking mode, and introduces binocular stereo vision. Simultaneously, an equivalent virtual binocular camera is constructed based on imaging principles, forming an adaptive binocular configuration mechanism. This establishes a correspondence between the left and right eye models and the camera's field of view, improving the simplicity and flexibility of dynamic visual tracking control and information processing, achieving binocular vision that mimics the human eye. Compared to existing multi-degree-of-freedom bionic binocular vision systems, this embodiment's imaging system eliminates the need for any form of movement of the left and right cameras. Driven by two sets of optical wedges, the system achieves six-dimensional motion of the bionic binoculars. The overall device is compact and offers excellent imaging field of view, image resolution, imaging efficiency, flexibility, and environmental adaptability.

[0081] Example 2

[0082] like Figures 3 to 7 As shown, this embodiment proposes a bionic binocular vision tracking method driven by an optical wedge group, comprising the following steps:

[0083] S1. Establish an equivalent relationship between the human eyes and the left and right optical wedge components. Based on the object-image mapping and imaging feedback mechanism of the target of interest in the binocular overlap domain, and with the principle of continuous movement of the visual axis adjustment, the left and right optical wedge components are used to flexibly adjust the visual axis of the imaging perception unit to simulate binocular motion imaging.

[0084] S2. Based on the deflection characteristics of the left and right optical wedge components on the camera imaging axis and the principle of optical path reversibility, an equivalent dynamic virtual binocular camera model is established;

[0085] S3. Establish the coordinate systems of the left camera and the right camera, use the Zhang Zhengyou calibration method to obtain the internal parameters of the left camera and the right camera and the radial distortion coefficient of their lenses, and determine the relative pose parameters of the left camera and the right camera;

[0086] S4. Determine the zero position of the main section of each set of optical wedges using the self-collimation mutual calibration method, calibrate the alignment relationship between the camera and the optical wedge group, and establish the coordinate system of the left optical wedge assembly and the right optical wedge assembly;

[0087] S5. Calibrate the internal parameters of the virtual camera, the posture parameters between the virtual binocular cameras, and the imaging distortion of the dynamic virtual camera according to the dynamic virtual camera model;

[0088] S6. Place the tracking target in the common field of view of the left camera and the right camera. The host computer uses the line of sight adjustment algorithm to adjust the imaging line of sight of the left and right optical wedge components and their corresponding cameras, so that the tracking target is in the center of the common field of view of the binocular cameras.

[0089] S7. Utilizing the distributed perception principle and collaborative control strategy that mimics the physiological mechanism of the human eye, the host computer adjusts the left and right optical wedge components and their corresponding camera imaging axes through a visual tracking algorithm to achieve closed-loop feedback automatic image stabilization and rapid tracking of dynamic targets in the bionic binocular vision system.

[0090] The left and right optical wedge components in S1 are controlled together to flexibly adjust the visual axis of the imaging perception unit, simulating binocular motion imaging. The process includes:

[0091] S11, based on the deflection effect of the optical wedge assembly on the camera imaging visual axis, sequentially obtain the imaging perspectives generated by the optical wedge assembly at different rotation angles and orientations, and use the independent rotational motion of each optical wedge around the optical axis to change the visual axis direction of the imaging unit;

[0092] S12. Eye movements can be mainly divided into conjugate movements, anisotropic movements, and steady-state gaze. According to the movement pattern of the imaging target within the camera's field of view, different wedge elements in the optical wedge assembly are given a certain angle and speed for synchronous rotation, thereby changing the imaging visual axes of the left and right cameras in turn, so that the tracking target is always within the common field of view of the left and right cameras.

[0093] The process of establishing an equivalent dynamic virtual binocular camera model in S2 includes:

[0094] S21. If the visual axis direction of the virtual camera is consistent with the visual axis deflection direction of the actual camera, the optical center of the virtual camera is located at the intersection of the visual axis of the virtual camera and the reverse extension line of the reverse tracing light;

[0095] S22. Constructing a pose transformation matrix between the virtual camera and the actual camera under different viewing axis directions according to the actual camera position and the optical wedge element parameters;

[0096] S23. Combining the optical center position and spatial pose parameters of the virtual camera, the visual axis pointing sequence of the two fixed cameras is equivalent to an infinite set of virtual binocular vision arrays, that is, an equivalent dynamic virtual binocular camera model.

[0097] The process of determining the relative pose parameters of the left camera and the right camera in S3 includes:

[0098] S31, placing a calibration plate in the common field of view of the left camera and the right camera, and changing the position of the calibration plate so that the left camera and the right camera respectively capture several groups of left and right images;

[0099] S32. Taking the left camera coordinate system as the world coordinate system, calculate the internal parameters of the left camera and the right camera, the radial distortion coefficient of the lens, and the relative position of the left camera and the right camera according to the left and right images.

[0100] The process of determining the zero position of the main section of each set of optical wedges and calibrating the alignment relationship between the camera and the optical wedge group in S4 includes:

[0101] S41. Determine the zero position of the main cross-sections of the left and right optical wedge assemblies using the autocollimation mutual calibration method, perform reference positioning, adjust the reticles of the two collimators so that their horizontal lines fall within the main cross-section of the reference optical wedge, and record this position.

[0102] S42, replacing the reference optical wedge with the optical wedge group to be calibrated for installation and calibration, using the collimator scribed lines as a reference to calibrate and obtain the initial position of the main cross section of each optical wedge in the optical wedge group;

[0103] S43, coaxially mounting the left and right optical wedge assemblies with calibrated main cross sections with the left and right cameras, removing the optical wedge assembly in front of the left camera, and arbitrarily placing a planar target within the common field of view of the left and right cameras to control direct imaging of the left camera and refraction imaging of the right camera through the double optical wedges;

[0104] S44, based on the posture relationship between the left camera and the right camera, extract the reference coordinates of the calibration points from the left camera's direct view image and transfer them to the right camera's coordinate system, and at the same time predict the three-dimensional coordinates of the calibration points from the right camera's refraction imaging results;

[0105] S45. Determine the alignment relationship between the right camera and its optical wedge group by minimizing the deviation between the reference datum and the model prediction. Similarly, use this method to calibrate the alignment relationship between the left camera and its optical wedge group.

[0106] The process of calibrating the internal parameters of the virtual camera, the pose parameters between the virtual binocular cameras, and the imaging distortion of the dynamic virtual camera in S5 includes:

[0107] S51, based on the image mapping relationship between the virtual camera and the actual camera, any point on the virtual image plane is mapped to a point on the actual image plane that has been corrected for lens distortion, and a pinhole imaging model of the left and right virtual cameras is established according to the theory of geometric optics;

[0108] S52. Based on the relative posture relationship between the left camera and the right camera and the control rule of the optical wedge group on the imaging visual axis direction, the imaging visual axis direction and the projection center position of the left camera and the right camera corresponding to the virtual camera are derived respectively, and the coordinate transformation relationship and relative posture parameters between the left virtual camera and the right virtual camera are obtained;

[0109] S53, using a virtual camera projection ray reverse tracing method to derive a mathematical mapping relationship between the virtual image plane and the actual image plane, and establish a nonlinear imaging distortion fitting model;

[0110] S54, using the coincidence constraint between the virtual camera projection light and the initial incident light of the optical wedge group, establishing an optimization objective function for the virtual camera imaging distortion coefficient;

[0111] S55. Based on the distribution and evolution characteristics of imaging distortion under different prism rotation angle combinations, an optimization objective function is formed with multi-order distortion coefficients as variables and the minimum ray tracing deviation as the criterion.

[0112] The process of fast tracking of dynamic targets in S7 includes:

[0113] S71. Based on the motion trajectory description and projection transformation matrix of the virtual binocular camera, establish the spatiotemporal coupling constraint relationship of the binocular image sequence, and construct the visual axis adjustment algorithm for different motion modes such as bionic binocular conjugation, anisotropy, and gaze;

[0114] S72. Extract SURF feature motion flow field from binocular camera image sequences, and combine visual saliency model and spatial information guidance mechanism to perform foreground region segmentation and dynamic target recognition;

[0115] S73, using a recursive least squares algorithm to estimate target pose parameters and establish a motion trajectory prediction model, and obtaining image deviation information of the current target relative to the center of the field of view of the left camera and the right camera;

[0116] S74. Using the triangulation principle, the absolute positioning information of the target is estimated from the deviations of the left and right images. The visual tracking algorithm uses the optical wedge group inverse solution algorithm to solve the rotation angle parameters of the left and right optical wedge components to ensure that the centers of the two fields of view can synchronously lock onto the target.

[0117] S75. Substitute the angle parameters of the optical wedge group into the state equation of the control system. Under the energy constraint of dual-field collaborative control, generate the joint optimal estimate of multiple control quantities such as the angular velocity and angular acceleration of the left and right optical wedge components, establish a closed-loop process, and realize continuous, stable, and smooth target tracking.

[0118] The inverse solution algorithm of the optical wedge group adopts one of the table lookup method, approximation method or iteration method.

[0119] The specific implementation includes the following steps:

[0120] Step 1: Parameter matching and system construction:

[0121] Step 11. Based on the imaging properties, field of view, and image resolution requirements of the bionic eye system, the parameters of the left and right cameras should be exactly the same. The following parameters are selected for the left and right cameras: horizontal field of view angle of 23.39°, vertical field of view angle of 17.65°, imaging resolution of 1600 × 1200, pixel size of 4.4 μm × 4.4 μm, lens focal length f = 12 mm, 16 mm, and 35 mm, and frame rate greater than 120 fps. For different application scenarios, a trade-off between the viewing axis pointing range and stereo imaging quality is considered.

[0122] Step 12: The two wedge elements in the rotating double wedge device are identical, with a wedge angle α = 20.05°, a refractive index n = 1.517, and a diameter D p =80mm, the thickness of the thin end d0 = 5mm, the two optical wedges are installed outward with a distance D1 = 100mm, the camera is coaxial with the rotating double optical wedge and is D2 = 30mm away from the nearest optical wedge plane, and the stepping motor drives the synchronous belt transmission mechanism to drive the optical wedge to achieve full circumferential rotation;

[0123] Step 13: Establish the left camera coordinate system O with the left camera optical center as the origin CL -X CL Y CL Z CL and the left image coordinate system x l o l y l , take the center of the incident plane of the left-rotating double wedge as the origin and establish the left wedge coordinate system O PL -X PL Y PL Z PL ; Similarly, establish the right camera coordinate system O CR -X CR Y CR Z CR , right image coordinate system x r o r y r And the right wedge coordinate system O PR -X PR Y PR Z PR ; Establish world coordinate system O W -X W Y W Z W The tracking target coordinate system O is established with the robot base as the origin R -X R Y R Z R .

[0124] Step 2: Calibrate the internal and external parameters of the imaging system, such as Figure 3As shown in the figure, the calibration of the bionic binocular vision system adopts an inside-out strategy. First, the internal characteristic parameters of the camera and the double optical wedge are obtained respectively, namely the internal and external parameters of the camera and the zero position of the main section of the optical wedge; then the relative positions of the left and right cameras and the double optical wedge are determined in turn, and the alignment of the camera and the double optical wedge is achieved by minimizing the deviation. The specific steps are:

[0125] Step 21: Select a checkerboard calibration plate with a minimum square side length of 12.5 mm. Use the Zhang Zhengyou calibration method. The left and right cameras capture multiple images of the calibration plate at different poses to obtain the internal parameters of the left and right cameras and the radial distortion coefficients of their lenses.

[0126] Step 22: Determine the relative pose parameters of the left and right cameras, namely the rotation matrix and translation vector, by combining the perspective projection model with the rigid body transformation relationship.

[0127] Step 23: Use the autocollimation calibration method to determine the zero position of the main cross-section of each set of dual optical wedges, perform reference positioning, adjust the reticles of the two collimators so that their horizontal lines fall within the main cross-section of the reference optical wedge, and record this position;

[0128] Step 24: Replace the reference double wedge with the double wedge to be calibrated for installation and calibration. Using the collimator scribed lines as a reference, calibrate the initial position of the double wedge's main cross section. Then, install the calibrated double wedge directly in front of the camera with its flat side facing outward.

[0129] Step 25: Use the auxiliary reference transfer principle to calibrate the axial alignment relationship between each camera group and the double wedge. Remove the double wedge device in front of the left camera. Arbitrary planar targets are placed within the common field of view of the left and right cameras. For planar targets at different positions, control the left camera to directly image and the right camera to refract through the double wedge.

[0130] Step 26: Extract the reference coordinates of the calibration points from the left camera's direct view image and transfer them to the right camera's coordinate system. Simultaneously, predict the 3D coordinates of the calibration points from the right camera's refraction imaging results. Determine the alignment relationship between the right camera and its dual wedge by minimizing the deviation between the reference coordinates and the model prediction.

[0131] Step 27: Similarly, use this method to calibrate the alignment relationship between the left camera and its double optical wedge, and calibrate the geometric position parameters between the left and right cameras and the left and right double optical wedges.

[0132] Step 3: Virtual camera pose calibration and distortion correction:

[0133] Step 31: Based on the camera and rotating dual-optical wedge imaging model, the dual-optical wedge driven binocular stereo vision system is equivalent to a dynamic virtual binocular camera system, and the left and right virtual camera coordinate systems are established, such as Figure 4 As shown;

[0134] Step 32: Any point on the virtual camera image plane can be mapped to a point on the actual image plane that has been corrected for lens distortion. Based on the image mapping relationship between the virtual camera and the actual camera, a pinhole imaging model of the left and right virtual cameras is established to obtain their equivalent focal lengths, f and f respectively. v1 and f v2 ;

[0135] Step 33: Combine the relative position relationship of the left and right cameras and the control law of the double optical wedge on the imaging axis direction, and use the vector refraction law and the reverse ray tracing method to determine the imaging axis direction s of the left and right cameras. r1 、s r2 Derived the imaging axis direction s of the corresponding virtual camera v1 、s v2 , combined with geometric optics theory to determine the projection center position of the left and right virtual cameras o r1 、o r2 . r1 With s v1 、s r2 With s v2 Substituting the Rodrigues transformation Rot, we can get the rotation matrix R of the left and right virtual cameras relative to their respective actual cameras. v1 、R v2 .

[0136]

[0137] The projection center position o of the actual left and right cameras r1 、o r2 For reference, the translation vector t of the left and right virtual cameras v1 , t v2 Respectively expressed as:

[0138]

[0139] Since the relative rotation matrix R of the left and right actual cameras r and the relative translation vector t r It has been pre-calibrated, and the coordinate transformation relationship of the left and right virtual cameras is determined by the rotation matrix R v and the translation vector t v Description, expressed as:

[0140]

[0141] Step 34: Using a virtual camera projection ray reverse tracing method, derive a mathematical mapping relationship between the virtual image plane and the actual image plane, and establish a nonlinear imaging distortion fitting model;

[0142] Step 35: Using the coincidence constraint between the virtual camera projection light and the initial incident light of the optical wedge group, an optimization objective function for the virtual camera imaging distortion coefficient is established;

[0143] Step 36: Based on the imaging distortion distribution and evolution characteristics under different prism rotation angle combinations, an optimization objective function is formed with multi-order distortion coefficients as variables and the minimum ray tracing deviation as the criterion. The objective function is solved by a numerical iterative optimization algorithm and is expressed as:

[0144]

[0145] The superscript m is used to distinguish the left and right virtual cameras, b r Based on b o1 or b o2 The actual camera projection light traced, h r is the exit position of the light on the double wedge plane side, {m v} represents the set of all image points on the virtual image plane, and c represents the vector composed of the distortion coefficients to be optimized.

[0146] Step 4: Bionic binocular stereo imaging and visual axis adjustment:

[0147] Step 41: Tracking the target Taking the robot end as an example, make the tracking target fall into the common field of view of the left and right cameras. The exit point of the target light corresponding to the left double wedge is (X fL ,Y fL ,Z fL ), the target pointing pitch angle is ρ L , the azimuth is The exit point of the right double wedge is (X fR ,Y fR ,Z fR ), the pitch angle is ρ R , the azimuth is According to the coordinate transformation relationship, the exit points of the left and right double wedges satisfy:

[0148]

[0149] Where, T LR (R LR ,t LR ) is the transformation matrix between the left and right cameras, T PR (R PR ,t PR ) and T PL (R PL ,t PL ) are the transformation matrices of the right camera and the left camera relative to the right double wedge and the left double wedge respectively. The coordinates of the spatial target point satisfy:

[0150]

[0151] Step 42: Based on the calibrated left and right cameras and the left and right rotating dual wedges, the position of the tracking target in the public field of view and the change in the visual axis of the virtual binocular camera are combined to perform reverse tracking. The left and right rotating dual wedge angles are adjusted by the inverse solution of the rotating dual wedge, that is, the pitch angle ρ and azimuth angle of the left and right light rays relative to the system optical axis are adjusted. Make the tracking target be located at the center of the common field of view of the left and right cameras.

[0152] Step 5: Bionic binocular imaging visual axis tracking, such as Figure 6 The figure shows a bionic dynamic binocular vision imaging and tracking solution. The left and right bionic eyes transmit target images to the high-bionic binocular vision host system through the left and right cameras and the dual optical wedge communication control module respectively. The eyeball drive control system realizes the control of the angle of each optical wedge, the stereo imaging system realizes the generation and modeling of three-dimensional point clouds, and the dynamic tracking system realizes the coordinated tracking control of the dual-field imaging visual axis. The specific steps are as follows:

[0153] Step 51: Set the target moving path, extract the URF feature motion flow field from the binocular camera image sequence, and perform foreground region segmentation and dynamic target recognition by combining the visual saliency model and spatial information guidance mechanism;

[0154] Step 52: Use a recursive least squares algorithm to estimate the target pose parameters and establish its motion trajectory prediction model, obtain the image deviation information of the current target relative to the center of the left and right camera fields of view, and use the triangulation principle to estimate the target absolute positioning information from the left and right image deviations;

[0155] Step 53, using different visual axis adjustment strategies of bionic binocular conjugation, anisotropy, gaze and other motion modes, such as Figure 5 As shown in the figure, (a) represents conjugate motion, (b) represents anisotropic motion, and (c) represents steady-state gaze. The rotation angle parameters of the left and right dual wedge groups are solved and adjusted by the inverse solution algorithm of the rotating dual wedge to ensure that the centers of the two fields of view can synchronously lock onto the target, thus realizing bionic binocular vision joint feedback collaborative control;

[0156] Step 54: Set the comprehensive image error threshold, calculate the distance between the image point of the tracking target and the center of the field of view in the left and right cameras, and calculate the comprehensive tracking error of the left and right cameras. If the comprehensive error is greater than the error threshold, iterate the above adjustment process; if it is less than the error threshold, end the task and complete the coordinated control of the left and right dual optical wedges on the visual axis. The iterative process is as follows: Figure 7 shown.

[0157]

[0158] In the formula, (x C ,y C ) is the normalized image coordinate of the camera target focal length, v inis the back-projection light vector of the target in the camera, v out is the target’s outgoing light vector in the prism, N is the prism plane normal vector, n is the prism refractive index, g(x) is the outgoing light response function with the double wedge incident light as the variable, f(x) is the angle response function with the double wedge exit light as the variable, (θ C1 ,θ C2 ) is the current rotation angle of the double wedge, (θ F1 ,θ F2 ) is the double wedge adjustment angle, (x FL ,y FL ) is the coordinate of the image point after adjustment of the target in the left camera, (x FR ,y FR ) is the coordinate of the image point after the target is adjusted in the right camera, (x L ,y L ) is the center coordinate of the left camera’s field of view, (x R ,y R ) is the coordinate of the center of the right camera's field of view.

[0159] like Figure 7 The figure shows the flow chart of the dual-optical wedge axis tracking iterative algorithm, which includes the following steps:

[0160] 1) Set the upper limit of the number of iterations CountMax, the initial iteration step Step and the tracking accuracy BiasThresh, and set the number of iterations Count = 0;

[0161] 2) Adjust the camera's viewing axis azimuth;

[0162] 3) If the current deviation Bias > BiasThresh, adjust the camera's visual axis pitch angle, execute Count++, and proceed to the next step; otherwise, the tracking is successful and all steps are completed.

[0163] 4) Determine whether Bias>BiasThresh and Count≤CountMax are satisfied. If so, set Step=Step / 2 and return to step (2); otherwise, execute the next step.

[0164] 5) Determine whether Bias>BiasThresh is satisfied. If so, the number of iterations is exceeded and all steps are terminated; otherwise, the tracking is successful and all steps are terminated.

[0165] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A bionic binocular vision tracking method driven by an optical wedge group, comprising the following steps: The equivalent relationship between the human eyes and the left and right optical wedge components is established. Based on the object-image mapping and imaging feedback mechanism of the target of interest in the binocular overlap domain, the left and right optical wedge components are used to flexibly adjust the visual axis of the imaging perception unit by joint control, simulating binocular motion imaging. According to the deflection characteristics of the left and right optical wedge components on the camera imaging axis, an equivalent dynamic virtual binocular camera model is established based on the principle of optical path reversibility. Establish the left and right camera coordinate systems, use the Zhang Zhengyou calibration method to obtain the internal parameters of the left and right cameras and the radial distortion coefficients of their lenses, and determine the relative pose parameters of the left and right cameras; The self-collimation mutual calibration method is used to determine the zero position of the main section of each set of optical wedges, and the alignment relationship between the camera and the optical wedge group is calibrated to establish the coordinate system of the left and right optical wedge components; According to the dynamic virtual camera model, the internal parameters of the virtual camera, the posture parameters between the virtual binocular cameras, and the imaging distortion of the dynamic virtual camera are calibrated; The tracking target is placed in the common field of view of the left and right cameras. The host computer uses the line of sight adjustment algorithm to adjust the imaging line of sight of the left and right optical wedge components and their corresponding cameras, so that the tracking target is in the center of the common field of view of the binocular cameras. Utilizing the distributed perception principle and collaborative control strategy that mimics the physiological mechanism of the human eye, the host computer uses a visual tracking algorithm to adjust the left and right optical wedge components and their corresponding camera imaging axes, achieving closed-loop feedback automatic image stabilization and rapid tracking of dynamic targets in the bionic binocular vision system. The process of flexibly adjusting the visual axis of the imaging perception unit by jointly controlling the left optical wedge assembly and the right optical wedge assembly to simulate binocular motion imaging includes: Based on the deflection effect of the optical wedge assembly on the camera imaging axis, the imaging angles generated by the optical wedge assembly at different rotation angles are obtained in sequence, and the independent rotational motion of each optical wedge around the optical axis is used to change the direction of the imaging unit's axis. According to the motion pattern of the imaging target in the camera field of view, the different optical wedge elements in the optical wedge assembly are given a certain angle and speed to rotate synchronously, thereby changing the imaging axis of the left and right cameras in turn, so that the tracking target is always within the common field of view of the left and right cameras.

2. The optical wedge group driven bionic binocular vision tracking method according to claim 1, characterized in that: The process of establishing an equivalent dynamic virtual binocular camera model includes: The visual axis direction of the virtual camera is consistent with the visual axis deflection direction of the actual camera, and the optical center position of the virtual camera is located at the intersection of the visual axis of the virtual camera and the reverse extension line of the reverse tracing light; According to the actual camera position and the optical wedge element parameters, the pose transformation matrix between the virtual camera and the actual camera under different viewing axis directions is constructed; Combining the optical center position and spatial pose parameters of the virtual camera, the visual axis pointing sequence of the two fixed cameras is equivalent to an infinite set of virtual binocular vision arrays, that is, an equivalent dynamic virtual binocular camera model.

3. The optical wedge-driven bionic binocular vision tracking method according to claim 1, characterized in that: The process of determining the relative pose parameters of the left camera and the right camera includes: Place a calibration plate in the common field of view of the left and right cameras, and change the position of the calibration plate so that the left and right cameras can capture several sets of left and right images respectively. The left camera coordinate system is used as the world coordinate system, and the internal parameters of the left camera and the right camera, the radial distortion coefficient of the lens, and the relative position of the left camera and the right camera are calculated according to the left and right images.

4. The optical wedge-driven bionic binocular vision tracking method according to claim 1, characterized in that: The process of determining the zero position of the main cross section of each set of optical wedges and calibrating the alignment relationship between the camera and the optical wedge group includes: Use the self-collimation mutual calibration method to determine the zero position of the main cross-section of the left and right optical wedge components, perform reference positioning, adjust the reticles of the two parallel light tubes so that their horizontal lines fall within the main cross-section of the reference optical wedge, and record this position; Replace the reference optical wedge with the optical wedge group to be calibrated and calibrate it. Using the parallel light tube scribed line as the reference, calibrate the initial position of the main section of each optical wedge in the optical wedge group. The left and right optical wedge assemblies with calibrated main cross sections are coaxially mounted with the left and right cameras. The optical wedge assembly in front of the left camera is removed. A planar target is arbitrarily placed in the common field of view of the left and right cameras to control direct imaging of the left camera and refraction imaging of the right camera through the double optical wedges. Based on the pose relationship between the left and right cameras, the reference coordinates of the calibration points are extracted from the left camera's direct view image and transferred to the right camera's coordinate system. At the same time, the three-dimensional coordinates of the calibration points are predicted from the right camera's refraction imaging results. The alignment between the right camera and its optical wedge group is determined by minimizing the deviation between the reference datum and the model prediction. Similarly, the alignment between the left camera and its optical wedge group is calibrated in the same way.

5. The optical wedge group driven bionic binocular vision tracking method according to claim 1, characterized in that: The process of calibrating the internal parameters of the virtual camera, the posture parameters between the virtual binocular cameras, and the imaging distortion of the dynamic virtual camera includes: According to the image mapping relationship between the virtual camera and the actual camera, any point on the virtual image plane is mapped to a point on the actual image plane that has been corrected for lens distortion. The pinhole imaging model of the left and right virtual cameras is established based on the theory of geometric optics. Combining the relative pose relationship between the left and right cameras and the control law of the optical wedge group on the imaging axis direction, the imaging axis direction and projection center position of the left and right cameras corresponding to the virtual cameras are derived respectively, and the coordinate transformation relationship and relative pose parameters between the left and right virtual cameras are obtained; The virtual camera projection ray reverse tracing method is used to derive the mathematical mapping relationship between the virtual image plane and the actual image plane, and establish a nonlinear imaging distortion fitting model; By using the coincidence constraint between the projection ray of the virtual camera and the initial incident ray of the optical wedge group, an optimization objective function for the imaging distortion coefficient of the virtual camera is established. According to the distribution and evolution characteristics of imaging distortion under different prism rotation angle combinations, an optimization objective function is formed with multi-order distortion coefficients as variables and the minimum ray tracing deviation as the criterion.

6. The optical wedge group driven bionic binocular vision tracking method according to claim 1, characterized in that: The process of fast tracking of dynamic targets includes: Based on the motion trajectory description and projection transformation matrix of the virtual binocular camera, the spatiotemporal coupling constraint relationship of the binocular image sequence is established, and the visual axis adjustment algorithm for different motion modes of the bionic binocular is constructed; Extract SURF feature motion flow field from binocular camera image sequences, and combine visual saliency model and spatial information guidance mechanism to perform foreground region segmentation and dynamic target recognition; The recursive least squares algorithm is used to estimate the target pose parameters and establish its motion trajectory prediction model to obtain the image deviation information of the current target relative to the center of the field of view of the left and right cameras; The absolute positioning information of the target is estimated from the deviation of the left and right images using the principle of triangulation. The visual tracking algorithm uses the inverse solution algorithm of the optical wedge group to solve the angle parameters of the left and right optical wedge components to ensure that the centers of the two fields of view can synchronously lock onto the target. The angular parameters of the optical wedge group are substituted into the state equation of the control system. Under the energy constraint of dual-field-of-view collaborative control, the joint optimal estimation of the angular velocity and angular acceleration of the left and right optical wedge components is generated, and a closed-loop process is established to realize the target tracking function.

7. The optical wedge group driven bionic binocular vision tracking method according to claim 6, characterized in that: The inverse solution algorithm of the optical wedge group adopts one of a table lookup method, an approximation method or an iterative method.

8. A device for the tracking method according to any one of claims 1 to 7, characterized in that: It includes a host computer (5), a camera, a rotating optical wedge device and a calibration component (6); The camera comprises a left camera (1) and a right camera (3), wherein the optical axes of the left camera (1) and the right camera (3) are parallel; the camera is used to acquire a target image; The rotating optical wedge device comprises a left rotating optical wedge (2) device and a right rotating optical wedge (4) device, the left rotating optical wedge (2) device and the right rotating optical wedge (4) device are respectively arranged in front of the left camera (1) and the right camera (3), and the rotating optical wedge device is used to adjust the imaging visual axis of the camera; The imaging visual axes of the left camera (1) and the right camera (3) are respectively coaxial with the optical axes of the left rotating optical wedge (2) device and the right rotating optical wedge (4) device; The calibration component (6) includes a plane target (62) and a calibration plate (61); the calibration component (6) is arranged in a common field of view of the left camera (1) and the right camera (3); the calibration component (6) is used to calibrate internal and external parameters between the camera and the rotating optical wedge device; The host computer (5) is connected to the camera and the rotating optical wedge device respectively.

9. The device according to claim 8, characterized in that The rotating optical wedge device includes an optical wedge assembly and a driving assembly; The optical wedge assembly includes a plurality of optical wedge elements; The driving assembly is used to drive the rotating optical wedge to expand the field of view of the camera.

Images