Imaging and human eye mapping method and system based on multi-degree-of-freedom bionic eye platform

Abstract

The invention relates to the technical field of robot vision and advanced man-machine interaction, in particular to an imaging and human eye mapping method and system based on a multi-degree-of-freedom bionic eye platform, which can realize high-precision and real-time matching of a bionic eye vision system and a human eye vision system. Based on the multi-degree-of-freedom bionic eye platform, compared with a binocular camera commonly used in the prior art, the multi-degree-of-freedom bionic eye platform has more degrees of freedom and is more flexible, and the bionic eye joint angle and the human eye motion parameters are directly associated through the trigonometric function relation for the first time. And meanwhile, the eyeball rotation angle is calculated by using the optical axis direction vector of the rotation matrix, so that the accuracy of biokinematics is ensured.

Description

Technical Field

[0001] This invention relates to the field of robot vision and advanced human-computer interaction technology, specifically to an imaging and human eye mapping method and system based on a multi-degree-of-freedom bionic eye platform. Background Technology

[0002] Multi-degree-of-freedom bionic eye first-person perspective stereoscopic teleoperation technology involves technologies such as stereoscopic vision and machine interaction. With the development of technologies such as remote operation and virtual reality (VR) live streaming, bionic eye systems capable of simulating and extending human visual capabilities are receiving increasing attention. An ideal bionic eye system should provide users with an immersive first-person visual experience, meaning that what the bionic eye "sees" is what the user "sees." However, the core challenge in achieving this goal lies in establishing a high-precision, natural mapping relationship between the bionic eye's mechanical vision system and the human eye's biological vision system. Currently, common technical approaches to achieving visual feedback in bionic eyes include: Direct video feed display involves directly displaying the image signals captured by the binocular camera on the bionic eye, after basic distortion correction, in a split-screen format on the user's head-mounted display. This method completely ignores the fundamental differences between the multi-degree-of-freedom mechanical structure of the bionic eye and the human eye's head-eye coordinated movement model. Users perceive a rigid visual feedback disconnected from their natural movement habits, resulting in poor immersion and extremely unintuitive operation. Simplified kinematic mapping: Some systems attempt to roughly map the overall orientation of the bionic eye (such as yaw and pitch angles) to the virtual camera's perspective. This method fails to deeply analyze and map the precise impact of the movements of each joint of the multi-degree-of-freedom bionic eye on human visual perception. Due to the lack of a quantitative model from joint space to human visual parameter space, the generated visual scene is geometrically distorted and cannot support the user's accurate judgment of spatial distance and depth. Fixed field-of-view processing: For the ultra-wide-angle field of view often found in bionic eye cameras, existing solutions typically use fixed image cropping or global distortion correction algorithms. This approach lacks an adaptive field-of-view matching mechanism that conforms to the physiological characteristics of the human eye. It cannot intelligently and dynamically adapt the wide field of view of the bionic eye to the most comfortable natural field of view of the human eye, resulting in either a waste of visual information or a distorted visual experience.

[0003] Furthermore, the aforementioned existing technologies generally treat system parameters (such as calibration parameters) as fixed and lack a closed-loop framework that can quantify the mapping accuracy and perform online or offline self-optimization accordingly. This results in a significant decrease in the system's performance and robustness when faced with hardware wear, temperature drift, or physiological differences among different users.

[0004] Currently, there is a VR holographic live streaming system based on AI and 3D scanning. It uses 3D scanning stereoscopic imaging technology (such as the Azure Kinect depth camera) to generate real-time color 3D models, combined with VR devices (such as the HTC Vive) to achieve holographic live streaming. Viewers can interact through VR glasses. Application scenarios include live e-commerce and virtual fitting rooms. Another type is a VR / AR-based robot teleoperation system that uses binocular cameras (such as the ZED Mini) to collect environmental depth information, combined with VR devices (such as the Oculus Quest 2) to achieve remote robot control. Unity3D is used to implement coordinate transformation and augmented reality functions, assisting operators in completing precision tasks (such as painting). However, the 3D live streaming technologies mentioned above still have many shortcomings and cannot provide users with an immersive, true 3D live streaming experience. The main problems are as follows: Existing methods typically display camera images directly after simple processing, lacking a quantitative mapping model from the pixel coordinates of the bionic eye camera to the physical coordinates of the human retina. This lack of geometric mapping results in geometrically inaccurate stereoscopic vision presented to the user, leading to deviations in judgments of distance, depth, and three-dimensional spatial relationships of target objects, failing to meet the needs of precise observation or operation. Bionic eye cameras often possess ultra-wide-angle fields of view (e.g., 170°), while the human eye's natural comfortable field of view is limited. Existing technologies employ simple image cropping or distortion correction, failing to establish a dynamic matching mechanism for the overlapping areas of the two fields of view. This results in either wasting the bionic eye's wide field of view or imposing distorted, ultra-wide-angle images onto the human eye, severely compromising the realism and comfort of visual immersion. Existing vision systems are mostly based on simple binocular camera mounts with limited motion modes. In contrast, the multi-degree-of-freedom bionic eye platform upon which this invention is based generates complex kinematic relationships through its multi-joint linkage structure. Existing methods cannot accurately and biomechanically map these complex joint angle vectors to head posture and eye movement parameters that are intuitively understandable to the human eye, resulting in unnatural motion perception and inconsistency between control intention and visual feedback.

[0005] Existing mapping parameters typically rely on one-time calibration. Once the hardware experiences minor wear or drifts due to factors such as ambient temperature, the mapping accuracy will continuously decrease. The system lacks a quantitative accuracy evaluation system and an automated parameter optimization closed loop, making it unable to adaptively compensate for hardware fluctuations and environmental interference, resulting in poor long-term system stability and robustness.

[0006] In summary, current bionic eye visual feedback technologies suffer from insufficient visual mapping accuracy, leading to spatial distortion, mismatched field of view resulting in poor immersion and comfort, lack of dedicated motion posture mapping for multi-degree-of-freedom bionic eyes, fixed and rigid mapping relationships, and a lack of adaptive optimization capabilities. Summary of the Invention

[0007] In view of this, the present invention provides an imaging and human eye mapping method and system based on a multi-degree-of-freedom bionic eye platform, which can achieve high-precision, real-time matching between the bionic eye visual system and the human eye visual system.

[0008] To achieve the objectives of this invention, the following technical solutions are provided.

[0009] An imaging and human eye mapping method based on a multi-degree-of-freedom bionic eye platform, the method comprising the following steps: A mapping model is established to quantitatively convert the mechanical motion parameters and imaging data of the bionic eye into the biological visual parameters of the human eye. The mapping model includes three dimensions: motion posture mapping, imaging coordinate mapping, and field of view mapping. By combining the visual servoing mechanism of the bionic eye, real-time mapping control is performed to achieve dynamic synchronization between bionic eye imaging and human eye imaging. The motion posture mapping specifically involves mapping the multi-degree-of-freedom bionic eye joint angle vector into a human eye motion parameter vector acquired through VR glasses. This human eye motion parameter vector includes the yaw angle, pitch angle, roll angle of the human head, and the horizontal and vertical rotation angles of the left and right eyes. Among them, the motion parameters of the human eye head are directly related to the joint angle of the bionic eye neck; the rotation angle of the human eyeball is obtained by calculating the trigonometric function relationship corresponding to the component of the optical axis direction vector of the bionic eye camera in the neck coordinate system; the imaging coordinate mapping is specifically: converting the pixel coordinates of the target point in the bionic eye camera into the human eye retina coordinates with the pupil as the origin; The conversion process includes: first converting the pixel coordinates into the corresponding target angle, and then converting the target angle into human retinal coordinates based on the equivalent focal length and small angle approximation of the human eye. The field of view mapping specifically involves establishing a matching mechanism for the overlapping area between the extended field of view of the bionic eye and the natural field of view of the human eye. By defining the pixel boundaries of the bionic eye and the angular boundaries of the human eye, and using inverse mapping to convert the human eye's field of view boundary into the pixel boundaries of the bionic eye, imaging coordinate mapping calculation is performed only when the pixel coordinates of the target point fall within this overlapping pixel area.

[0010] Specifically, the real-time mapping control includes: Images are acquired in real time using left and right cameras, and the pixel coordinates of target points are extracted. Determine whether the pixel coordinates fall within the overlapping area of ​​the field of view: if they are within the overlapping area, then directly perform subsequent mapping calculations; if they are outside the overlapping area, then generate a neck joint adjustment command to move the target point into the overlapping area. Based on the proportional control law and the visual servo relation matrix, the speed commands of the left and right eye joints are calculated and output according to the pixel error to achieve dynamic tracking.

[0011] Specifically, the optimization step based on maximizing the accuracy function includes: Construct a comprehensive evaluation function, which is a weighted sum of the imaging coordinate mapping accuracy and the motion attitude mapping accuracy; The optimization variable is defined as the diagonal gain matrix in servo control, whose elements are the error adjustment gains in the horizontal and vertical directions, and constraints are imposed on the gains. The optimization variables are iteratively updated using the gradient ascent method to maximize the comprehensive evaluation function; The optimized gain matrix is ​​updated to the servo control loop in real time, forming a closed-loop optimization.

[0012] The present invention also provides an imaging and human eye mapping system based on a multi-degree-of-freedom bionic eye platform for performing the method described herein, the system comprising: A multi-degree-of-freedom bionic eye platform, comprising a neck module, a left eye module, a right eye module, and a binocular camera fixedly mounted at its end; Head-mounted display and tracking devices are used to display visual images and collect human eye movement parameters; The computing and control unit has built-in software modules for executing algorithms, mapping calculations, and generating control instructions; Servo drive unit, used to drive the movement of each joint of the bionic eye platform; The data acquisition and interface module is used to read joint angles, image streams, and human eye movement parameters. The multi-dimensional mapping model calculation module is used to perform motion pose mapping, imaging coordinate mapping, and field of view mapping; The visual servo and control module is used for target detection, field of view determination, and servo control. The image rendering and output module is used to generate and output stereoscopic images to the head-mounted display.

[0013] The multi-dimensional mapping model calculation module includes: The motion posture mapping submodule is configured to calculate the expected bionic eye motion posture based on the current joint angle; The imaging coordinate mapping submodule is configured to convert the pixel coordinates of the target point into predicted human eye retinal coordinates. The field-of-view mapping submodule is configured to dynamically determine the effective pixel mapping area.

[0014] The visual servoing and control module includes: The target detection and tracking submodule is used to extract the pixel coordinates of target points from the image in real time; The field of view determination and decision-making submodule is used to determine the control strategy; The servo control law calculation submodule is used to calculate and output the speed command of the eye joint.

[0015] The system further includes an adaptive optimization module, which comprises: The accuracy evaluation submodule is used to calculate the accuracy of imaging coordinate mapping and motion attitude mapping. An optimization submodule is used for iterative optimization of the servo gain matrix; The parameter update submodule is used to update the optimized gain to the servo control law.

[0016] Beneficial effects First, this invention is based on a multi-degree-of-freedom bionic eye platform, which has more degrees of freedom and is more flexible than the binocular cameras commonly used in existing technologies. It is the first to directly link the joint angles of the bionic eye with human eye motion parameters through trigonometric functions. Simultaneously, it uses the optical axis direction vector of the rotation matrix to calculate the eyeball rotation angle, ensuring the accuracy of biokinematics.

[0017] Second, this invention is the first to quantify mapping accuracy as an optimizable function, combining weighting coefficients to highlight imaging precision. Automated parameter adjustment based on gradient ascent adapts to hardware fluctuations and environmental changes. The proposed imaging coordinate mapping model combines camera imaging models and human eye physiological parameters (such as equivalent focal length) to achieve a quantitative conversion from pixel coordinates to retinal coordinates. Furthermore, a small-angle approximation simplifies calculations, ensuring real-time performance while maintaining accuracy. Detailed Implementation

[0018] The present invention will be further described below with reference to specific embodiments.

[0019] This invention proposes an imaging and human eye mapping method based on a multi-degree-of-freedom bionic eye platform. By building upon an existing multi-degree-of-freedom bionic eye platform, it models the mapping relationship and optimizes it based on maximizing a precision function, achieving visual imaging and binocular mapping of the human eye. This method, through establishing a multi-dimensional mapping model and optimization algorithms, achieves high-precision, real-time matching between the bionic eye visual system and the human eye visual system.

[0020] The bionic eye platform upon which this invention is based has a total of 7 degrees of freedom (DOF) and adopts a "neck-eyeball serial open chain" configuration. The specific joint allocation and functional correspondence are as follows: Neck module (3DOF): includes joint 1 ( Horizontal yaw), Joint 2 ( Vertical pitch), joint 3 ( (x, y), which correspond to the yaw, pitch, and roll movements of the human head, respectively; Left eye module (2DOF): includes joint 4 ( Tilt movement, tilt), joint 5 ( Rotational motion (pan) corresponds to the vertical and horizontal rotation of the left eye, respectively. Right eye module (2DOF): includes joint 7 ( Tilt movement, tilt), joint 8 ( Rotational motion (pan) corresponds to the vertical and horizontal rotation of the right eye, respectively. Fixed mounting frame: Joints 6 and 9 are fixed mounting frames for the camera, with no active degrees of freedom of movement, and are only used for the mechanical connection between the camera and the eyeball linkage.

[0021] The linkage and joint parameters of this system are modeled using the Denavit-Hartenberg (DH) standard, with the core linkage length parameter being: , , (Left eye side linkage) (Right eye side linkage) , The complete DH parameters are shown in Table 1.

[0022] Table 1. DH Parameter Table of the 7DOF Bionic Eye System

[0023] The imaging and human eye mapping method based on a multi-degree-of-freedom bionic eye platform of this invention specifically includes the following steps: Step 1: Based on the mapping model, the mechanical motion parameters and imaging data of the bionic eye are quantitatively converted into the biological visual parameters of the human eye, including three dimensions: motion posture mapping, imaging coordinate mapping, and field of view mapping.

[0024] The specific process of dynamic attitude mapping is as follows: Multi-degree-of-freedom bionic eye joint angle vector (multi-degree-of-freedom bionic eye platform) Mapped to a vector of human eye motion parameters acquired through VR glasses. ,in: The yaw, pitch, and roll angles of the human eye and head. These represent the horizontal and vertical rotation angles of the left and right eyes, respectively.

[0025] In head motion mapping, the neck coordinate system The posture is determined by the angles of joints 1 to 3. The parameters of human eye-head movement are directly determined, and therefore correspond directly to the angles of the bionic eye-neck joints, i.e.:

[0026] In eye movement mapping, the camera optical axis direction is determined by a rotation matrix. of Axial direction vector ( for The Line number The column elements describe the direction vector, which has a trigonometric function relationship with the angle of rotation of the human eyeball: For the left eye, horizontal rotation angle With vertical rotation angle for:

[0027] For the right eye, horizontal rotation angle With vertical rotation angle for:

[0028] in, Left eye rotation matrix ( This is a 3×3 rotation matrix describing the left eye camera coordinate system relative to the neck coordinate system. The posture, right eye Similarly, the corresponding elements, Right eye rotation matrix The corresponding element.

[0029] In summary, the motion attitude mapping function can be expressed as:

[0030] This invention, based on a motion posture mapping function, is the first to directly correlate the angle of a bionic eye joint with human eye motion parameters through trigonometric functions. Simultaneously, it utilizes the optical axis direction vector of the rotation matrix to calculate the eyeball rotation angle, ensuring the accuracy of biokinematics.

[0031] The specific process of imaging coordinate mapping is as follows: The pixel coordinates of the target point P in the bionic eye camera and Convert to human retinal coordinates with the pupil as the origin and ， The specific mapping is as follows: Pixel coordinates to target angle conversion:

[0032] in, The physical size of a pixel. This refers to the camera's focal length.

[0033] Retinal coordinate calculation (based on small angle approximation):

[0034] in, , which is the equivalent focal length for the human eye.

[0035] The complete mapping function can be obtained (taking the left eye as an example, and the right eye as an example). Similarly):

[0036] The specific process of field-of-view mapping is as follows: Establish a matching mechanism for the overlapping area between the extended field of view (170°) of the bionic eye and the natural field of view of the human eye (150° horizontally and 120° vertically, with differences in the field of view among different individuals).

[0037] First, the field of view boundary is defined. For the bionic eye, the pixel boundary is as follows:

[0038] The angular boundary of the human eye is:

[0039] Mapping is performed on overlapping regions only if the target point pixel coordinates are... Mapping calculations are only performed when the pixel falls within the pixel boundary corresponding to the human eye's field of view.

[0040] in, These are the upper and lower limits of the horizontal pixel boundary. These are the upper and lower limits of the vertical pixel boundary.

[0041] Therefore, the field of view boundary can be calculated using inverse mapping:

[0042] The dynamic field-of-view matching mechanism in the field-of-view mapping of this embodiment can ensure that the mapping result conforms to the natural perception range of the human eye, and convert the human eye field-of-view boundary into the bionic eye pixel boundary through inverse mapping to achieve cross-domain adaptation.

[0043] Step 2: Combining the visual servoing mechanism of the bionic eye, real-time mapping control is performed to achieve dynamic synchronization between bionic eye imaging and human eye imaging. The specific steps are as follows: Step 21, target detection and field of view determination, specifically involves acquiring target scene images in real time using left and right cameras, and extracting target points using a deep learning-based target detection algorithm (or AprilTag recognition algorithm). pixel coordinates Determine if the pixel coordinates fall within the overlapping area of ​​the field of view: if within the overlapping area, directly perform subsequent mapping calculations; if outside the overlapping area, generate a neck joint adjustment command to control... Movement, towards the target point The image is moved into the overlapping area.

[0044] Step 22, mapping result calculation and dynamic servo control, specifically based on the motion posture and imaging coordinate mapping model, inputting the current joint angle. with pixel coordinates Calculate human eye motion parameters With retinal coordinates When the target point Calculate the rate of change of pixel coordinates during motion. Error convergence is achieved using a proportional control law, which is defined as follows:

[0045] in, For pixel error ( (The coordinates of the desired pixels within the overlapping area are usually taken near the principal point). The diagonal gain matrix ( (Adjust gain for errors in the horizontal and vertical directions).

[0046] because For a fixed value ( ), we can get Substituting into the control law, we get Combining the relationship matrix of visual servoing ( For camera interaction matrix, (Joint Jacobian matrix), joint velocity satisfy:

[0047] The joint velocity command is obtained by solving:

[0048] Generate joint velocity commands separately for the left and right eyes: The left eye is for control Movement, speed command is:

[0049] The right eye is for control Movement, speed command is:

[0050] in, These are the relationship matrices for the left and right eyes, respectively.

[0051] Furthermore, to address hardware parameter fluctuations and environmental interference, this invention provides optimization steps based on maximizing the accuracy function, thereby improving the accuracy and robustness of the mapping model. The specific steps are as follows: Step 31, Construct the accuracy function: Define the comprehensive evaluation function The value range is [0,1], and the expression is:

[0052] in, For imaging accuracy weights, The weights are for motion posture accuracy.

[0053] Imaging coordinate mapping accuracy :

[0054] in, N is the number of samples; For the first Left eye retinal coordinate mapping results for the dataset; For the first True values ​​of the left eye retinal coordinates for the set of data (obtainable from VR glasses calibration); Motion attitude mapping accuracy :

[0055] in, . For the first Head motion parameter mapping results for the group data; For the first True values ​​of head motion parameters in the dataset (measured using VR glasses eye-tracking equipment). Step 32, Optimization Variables and Constraints: The optimization function includes optimization variables. ,in, It is a diagonal gain matrix. The gain is used to adjust the error in the horizontal and vertical directions; the formula has the following constraints: servo gain : .

[0056] Step 33, Gradient Ascent Solution: Maximize using the gradient ascent method. The specific steps are as follows: Iterative updates using formulas:

[0057] in, The step size.

[0058] Convergence condition: .

[0059] The results of each iteration are updated in real time until convergence.

[0060] Furthermore, this embodiment performs adaptive servo gain adjustment:

[0061] The result after optimization This will be incorporated into the motion control section to generate new speed commands, which will be changed after each iteration to achieve closed-loop control.

[0062] The present invention also provides an imaging and human eye mapping system based on a multi-degree-of-freedom bionic eye platform, which can execute the methods described in the present invention. The system of the present invention includes a multi-degree-of-freedom bionic eye platform, a head-mounted display and tracking device, a computing and control unit, a servo drive unit, a data acquisition and interface module, a multi-dimensional mapping model calculation module, a visual servo and control module, and an image rendering and output module.

[0063] The multi-degree-of-freedom bionic eye platform serves as the system's actuator, comprising a neck module, a left eye module, a right eye module, and a binocular camera. The neck module includes three active rotational joints (joints 1, 2, and 3) to simulate the yaw, pitch, and roll movements of the human head. The left eye module includes two active rotational joints (joints 4 and 5) to drive the left eye camera in vertical and horizontal rotation. The right eye module includes two active rotational joints (joints 7 and 8) to drive the right eye camera in vertical and horizontal rotation. The binocular camera is fixedly mounted at the ends of the left and right eye modules (i.e., the connecting rod containing joints 6 and 9) to acquire stereoscopic image information. Preferably, the camera has an ultra-wide-angle field of view.

[0064] Head-mounted display and tracking devices are used for display and tracking, such as VR glasses. The display presents the user with a mapped stereoscopic visual image, and the tracking acquires and outputs the user's eye motion parameter vector H in real time, including the head's yaw, pitch, roll angles, and the horizontal and vertical rotation angles of the eyes.

[0065] The computing and control unit is the core processing component of the system, such as an embedded computer, industrial computer, or server. It has built-in software modules used to execute all the algorithms, mapping calculations, and control instruction generation described below.

[0066] The servo drive unit connects to the computing and control unit, receives joint speed or position commands, and drives the joint motors of the bionic eye platform. The data acquisition and interface module is responsible for reading the real-time joint angle vector θ from the joint encoder of the bionic eye platform; receiving real-time left and right eye image streams from the binocular camera; and receiving real-time human eye motion parameter vector H from the head-mounted display and tracking device.

[0067] The multi-dimensional mapping model calculation module includes a motion posture mapping submodule, an imaging coordinate mapping submodule, and a field of view mapping submodule. The motion posture mapping submodule is configured to execute the aforementioned motion posture mapping function H = f(θ) to calculate the expected bionic eye motion posture based on the current joint angle θ. The imaging coordinate mapping submodule is configured to execute the aforementioned imaging coordinate mapping function to convert the pixel coordinates of the target point in the camera into the predicted human eye retinal coordinates. The field of view mapping submodule is configured to execute the aforementioned field of view boundary definition and overlapping area judgment logic to dynamically determine the effective pixel mapping area.

[0068] The visual servoing and control module includes a target detection and tracking submodule, a field-of-view determination and decision-making submodule, and a servo control law calculation submodule. The target detection and tracking submodule integrates algorithms based on deep learning or AprilTag to extract the pixel coordinates of the target point P in real time from the left and right camera images. The field-of-view determination and decision-making submodule determines whether the target point is within the overlapping area of ​​the field of view and decides accordingly whether to perform direct mapping or generate adjustment commands for the neck joint. The servo control law calculation submodule is based on a proportional control law. = -K·e and the relation matrix, calculate and output the velocity commands of the left and right eye joints to achieve dynamic tracking.

[0069] Image rendering and output module: Receives retinal coordinate data from the imaging coordinate mapping module, combines it with the parameters of the head-mounted display device, generates the final stereoscopic image that conforms to the visual characteristics of the human eye, and outputs it to the head-mounted display for display.

[0070] Furthermore, it also includes an adaptive optimization module, which comprises an accuracy evaluation submodule, an optimization solution submodule, and a parameter update submodule. The accuracy evaluation submodule is configured to calculate the imaging coordinate mapping accuracy F1(X) and motion posture mapping accuracy F2(X) based on the collected N sets of sample data (including mapping results and true values ​​measured by VR devices), thereby obtaining the comprehensive accuracy function F(X). The optimization solution submodule uses the gradient ascent method to iteratively optimize the parameters k1 and k2 in the servo gain matrix K with the goal of maximizing F(X). The parameter update submodule updates the new gain obtained after optimization to the servo control law in the visual servo and control module in real time, forming a closed-loop optimization and improving the robustness of the system.

[0071] The workflow of the system of this invention is as follows: Initialization: The system starts up, each unit is powered on, and initial parameters (such as DH parameters, camera intrinsic parameters, initial servo gain K, etc.) are loaded.

[0072] Data stream synchronization: The data acquisition module synchronously acquires joint angle θ, binocular image stream, and human eye motion parameters H.

[0073] Target perception and mapping: The target detection submodule identifies target point P from the image; the field of view determination submodule determines the control strategy; and the mapping model calculation module calculates the motion posture H and retinal coordinates in the current state in parallel.

[0074] Servo control: The visual servo control module calculates and issues joint speed commands based on the pixel error e and the current gain K, driving the bionic eye to move and keep the target in the center of the field of vision.

[0075] Visual feedback: The image rendering module sends the processed image to the VR glasses to provide users with an immersive visual experience.

[0076] Furthermore, background optimization can be performed: the adaptive optimization module periodically collects data, calculates the mapping accuracy, and runs optimization algorithms to update the servo gain K in order to continuously maintain the system's best performance.

[0077] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made using the present invention specification, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for imaging and human eye mapping based on a multi-degree of freedom bionic eye platform, characterized in that, The method comprises the following steps: establishing a mapping model to quantitatively convert the mechanical motion parameters and imaging data of the bionic eye into the biological vision parameters of the human eye, the mapping model comprising three dimensions of motion posture mapping, imaging coordinate mapping and field of view range mapping; combining the visual servo mechanism of the bionic eye, performing real-time mapping control to realize dynamic synchronization of the bionic eye imaging and the human eye imaging; the motion posture mapping is specifically: mapping the multi-degree-of-freedom bionic eye joint angle vector into the human eye motion parameter vector collected by the VR glasses, the human eye motion parameter vector comprising the yaw angle, the pitch angle, the roll angle of the human eye head and the horizontal rotation angle and the vertical rotation angle of the left and right eyes; wherein the motion parameters of the human eye head directly correspond to the joint angle of the bionic eye neck; the rotation angle of the human eye eyeball is obtained by calculating the trigonometric function relationship corresponding to the component of the bionic eye camera optical axis direction vector in the neck coordinate system; the imaging coordinate mapping is specifically: converting the pixel coordinates of the target point in the bionic eye camera into the human eye retina coordinates with the pupil as the origin; the conversion process comprises: first converting the pixel coordinates into corresponding target angles, and then converting the target angles into human eye retina coordinates based on the human eye equivalent focal length and small-angle approximation; the field of view range mapping is specifically: establishing an overlapping area matching mechanism of the bionic eye extended field of view and the natural field of view of the human eye; by defining the bionic eye pixel boundary and the human eye angle boundary, and using inverse mapping to convert the human eye field of view boundary into the bionic eye pixel boundary, only when the target point pixel coordinates fall within the overlapping pixel area, the imaging coordinate mapping calculation is performed.

2. The method of claim 1, wherein, The real-time mapping control specifically comprises: real-time image acquisition and target point pixel coordinate extraction by left and right cameras; judging whether the pixel coordinates fall within the field of view overlapping area: if within the overlapping area, directly performing subsequent mapping calculation; if beyond the overlapping area, generating a neck joint adjustment instruction to move the target point into the overlapping area; based on the proportional control law and the visual servo relationship matrix, calculating and outputting the speed instruction of the left and right eyeball joints according to the pixel error to realize dynamic tracking.

3. The method according to claim 1 or 2, characterized in that, The optimization step based on the maximization of the accuracy function specifically comprises: constructing a comprehensive evaluation function, which is the weighted sum of the imaging coordinate mapping accuracy and the motion posture mapping accuracy; defining the optimization variable as the diagonal gain matrix in the servo control, the elements of which are the error adjustment gains in the horizontal and vertical directions, and imposing constraints on the gains; using the gradient ascent method to iteratively update the optimization variable to maximize the comprehensive evaluation function; updating the optimized gain matrix to the servo control loop in real time to form a closed-loop optimization.

4. An imaging and human eye mapping system based on a multi-degree of freedom bionic eye platform for performing the method of any one of claims 1 to 3, characterized in that, The system comprises: a multi-degree-of-freedom bionic eye platform comprising a neck module, a left eye module, a right eye module and a binocular camera fixedly installed at the end thereof; a head-mounted display and tracking device for displaying visual images and collecting human eye motion parameters; a computing and control unit with a software module built-in for executing algorithms, mapping calculations and control instruction generation; a servo drive unit for driving the joint motion of the bionic eye platform; a data acquisition and interface module for reading joint angles, image streams and human eye motion parameters; The multi-dimensional mapping model calculation module is configured to perform motion posture mapping, imaging coordinate mapping, and field of view range mapping. The visual servoing and control module is configured to perform target detection, field of view judgment, and servo control. The image rendering and output module is configured to generate and output a stereoscopic image to a head-mounted display.

5. The system of claim 4, wherein, The multi-dimensional mapping model calculation module includes: A motion posture mapping submodule configured to calculate an expected bionic eye motion posture according to a current joint angle; An imaging coordinate mapping submodule configured to convert pixel coordinates of a target point into predicted human eye retina coordinates; A field of view range mapping submodule configured to dynamically determine an effective pixel mapping region.

6. The system of claim 5, wherein, The visual servoing and control module includes: A target detection and tracking submodule configured to extract pixel coordinates of a target point from an image in real time; A field of view judgment and decision submodule configured to determine a control strategy; A servo control law calculation submodule configured to calculate and output a speed instruction of an eyeball joint.

7. The system of any of claims 4-6, wherein, The system further includes an adaptive optimization module, which includes: An accuracy evaluation submodule configured to calculate imaging coordinate mapping accuracy and motion posture mapping accuracy; An optimization solution submodule configured to iteratively optimize a servo gain matrix; A parameter update submodule configured to update the optimized gain to a servo control law.

Images