Virtual and actual calibration verification system and verification method based on mechanical arm

Abstract

The invention discloses a virtual and real calibration verification system and verification method based on a mechanical arm, and the system comprises the mechanical arm, a rotating platform, an observation camera, a detection module and a control module, and the front end of the mechanical arm is provided with a clamping jaw and a clamp which are detachably connected; a marker capable of synchronously rotating along with the rotating platform is fixedly arranged on the rotating platform; the detection module is provided with a device for shooting an actual marker image and generating a marker pose under a coordinate system of the device, the observation camera is used for shooting a virtual model image and generating a pose under the coordinate system based on the virtual model image, and the control module can drive the marker on the rotating platform to move synchronously. The mechanical arm is controlled to stretch out and draw back, a clamping jaw and a clamp on the mechanical arm are controlled to grab a marker located on the rotating platform to move, a transformation matrix from a real marker to an actual virtual model and then to an ideal virtual model is obtained, the transformation matrix is solved, and the attitude error between the virtual model and the marker is obtained; and determining the pose of the marker in the real world coordinate system.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a virtual-reality calibration verification system and verification method based on a mechanical arm. Background Art

[0002] With the development of mixed reality (MR) and augmented reality (AR) technologies in various fields, virtual-reality calibration technology, which plays an important role in them, has also received widespread attention. Virtual-reality calibration refers to the process of accurately matching objects in the virtual world with objects in the real world to achieve a more realistic and accurate interaction and fusion experience. Specifically, in AR and MR, the purpose of virtual-reality calibration is to ensure the consistency between virtual content and real scenes. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by related interactive work. A good calibration solution is the key to accurately rendering and aligning virtual objects and improving user experience.

[0003] At present, the common verification methods for virtual-real calibration include calibration test method and double cube matching method. Both schemes are to let users wear detection modules to observe the virtual cube corresponding to the real cube marker on the virtual screen, and to collect calibration data and calculate and evaluate by moving the handheld cube marker to overlap and align with the virtual cube. The problems are obvious. First, the movement and hand tremor of the user after wearing the detection module will make the alignment inaccurate and cause serious errors, and lose the objectivity of the evaluation; second, the user holding the marker will limit the reachable range of the marker, which will cause the calibration result to be unbalanced in the calibration space; third, the user cannot directly give the position of the real cube marker in the world coordinate system. Summary of the invention

[0004] In order to solve some or all of the technical problems existing in the above-mentioned prior art, the present invention provides a virtual-reality calibration verification system and verification method based on a robotic arm, which utilizes a programmable high-precision robotic arm with an attached observation camera to replace human eyes, and adds a clamp to fix the HMDs thereon; a rotating translation stage is used to take fixed markers, so that the position and posture of the markers in the real-world coordinate system can be directly determined; by combining the robotic arm and the rotating translation stage, the effect of high-precision repetition of multiple tracks can also be achieved, ultimately achieving sufficiently objective and accurate evaluation results, and being able to ensure the repeatability and high-precision repetition of the trajectory.

[0005] The technical solution of the present invention is as follows:

[0006] In a first aspect, the present invention provides a virtual-real calibration verification system based on a robotic arm, comprising:

[0007] A mechanical arm, wherein a detachably connected claw and a clamp are provided at the front end of the mechanical arm;

[0008] A rotating platform, the rotating platform is arranged at any position in the space that the mechanical arm can grasp by extension and retraction, and a marker is fixedly arranged on the rotating platform so that the marker can rotate synchronously with the rotating platform when the rotating platform rotates;

[0009] A detection module, on which a device for photographing an actual image of the marker is disposed, and the position and posture of the marker in the device coordinate system is obtained by using the photographed actual image of the marker;

[0010] An observation camera, which is arranged on the detection module and is used to capture an image of the virtual model and obtain the position and posture of the virtual model in the observation camera coordinate system by using the captured virtual model image; the detection module and the observation camera are fixedly arranged at the end of the robotic arm through the claw and the clamp;

[0011] A control module, wherein the control module is electrically or wirelessly connected to the robotic arm, the rotating platform, the camera and the detection module, the control module can control the rotation and movement of the rotating platform, and drive the marker on the rotating platform to rotate or move synchronously, and the control module stores a plurality of verification models and a plurality of control programs corresponding to the verification models, controls the extension and retraction of the robotic arm and the gripping of the marker on the rotating platform by the claw and the fixture on the robotic arm, so that the plurality of control programs can control the robotic arm and the rotating platform to move along a predetermined route, projects the marker in the real world into the virtual world through the control program in the control module, forms a corresponding virtual model, makes the marker in the virtual world accurately correspond to the marker in the real world, calculates the posture transformation of the marker before and after movement, obtains the transformation matrix from the real marker to the actual virtual model and then to the ideal virtual model, solves the transformation matrix, obtains the posture error between the virtual model and the marker, and determines the posture of the marker in the real world coordinate system.

[0012] Furthermore, in the above-mentioned robot-based virtual-reality calibration verification system, the marker includes a cube model or a calibration plate.

[0013] Furthermore, in the above-mentioned robot-based virtual-real calibration verification system, the device for capturing the actual image of the marker includes a fisheye camera.

[0014] Furthermore, in the above-mentioned robot-based virtual-reality calibration and verification system, the detection module includes a head display.

[0015] In a second aspect, the present invention further provides a virtual-real calibration verification method based on a robotic arm using the virtual-real calibration verification system based on a robotic arm, comprising:

[0016] Calling the stored calibration model to obtain the calibrated transformation matrix corresponding to the stored calibration model;

[0017] Acquire the position and posture information of the marker in real time, move the marker so that it is aligned with the calibration model, record the aligned data, and calculate the posture error between the virtual model and the marker;

[0018] The pose error between the calculated virtual model and the marker is evaluated and verified.

[0019] Furthermore, in the above-mentioned virtual-reality calibration verification method based on a robotic arm, it also includes collecting a data set for comparative testing of different calibration methods.

[0020] Furthermore, in the above-mentioned virtual-real calibration verification method based on a robotic arm, the posture error between the calculated virtual model and the marker is evaluated and verified, including displacement error and rotation error.

[0021] Furthermore, in the above-mentioned virtual-real calibration verification method based on the robotic arm, the displacement error is calculated by the following formula:

[0022]

[0023] in: represents the displacement error, and Δx, Δy, and Δz are the offsets of the virtual model before and after alignment with the marker.

[0024] Furthermore, in the above-mentioned virtual-real calibration verification method based on a robotic arm, the rotation error is calculated by the following formula:

[0025]

[0026] Where: L g represents the rotation error, R g and R 2 are the rotation matrices of the cube markers before and after realignment, respectively. tr() represents the trace of the matrix, which is numerically equal to the sum of the elements on the main diagonal of the matrix.

[0027] The main advantages of the technical solution of the present invention are as follows:

[0028] The virtual-real calibration verification system based on a mechanical arm of the present invention, through the structure of the mechanical arm and the rotating platform, and the claws and clamps are arranged on the mechanical arm, and the camera and the detection module are arranged at the end of the mechanical arm, so that under the control of the control module, the rotating platform can be controlled to rotate and move, and the marker on the rotating platform can be driven to rotate or move synchronously. By storing a plurality of verification models and a plurality of control programs corresponding to the plurality of verification models in the control module, the extension and retraction of the mechanical arm and the claws and clamps on the mechanical arm are controlled to grab the marker on the rotating platform, so that the plurality of control programs can control the mechanical arm and the rotating platform to move along a predetermined path. The markers in the real world are projected into the virtual world through the control program in the control module to form a corresponding virtual model, so that the markers in the virtual world correspond accurately to the markers in the real world. The position and posture transformation before and after the marker moves is calculated, and the transformation matrix from the real marker to the actual virtual model and then to the ideal virtual model is obtained. The posture error between the virtual model and the marker is obtained by solving the transformation matrix, and the position and posture of the marker in the real-world coordinate system is determined. This can achieve the effect of high-precision repetition of multiple tracks, and finally achieve a sufficiently objective and accurate evaluation result, and can ensure high-precision repetition of the trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The drawings described herein are used to provide a further understanding of the embodiments of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0030] Figure 1 It is a schematic diagram of the overall structure of a virtual-real calibration verification system based on a mechanical arm according to an embodiment of the present invention;

[0031] Figure 2 A schematic flow chart of a virtual-real calibration verification method based on a robotic arm provided in one embodiment of the present invention.

[0032] Description of reference numerals:

[0033] 1. Robotic arm; 2. Rotating platform; 3. Observation camera; 4. Detection module; 5. Marker. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0035] The following is combined with Figure 1-2 , describes in detail the technical solution provided by the embodiments of the present invention.

[0036] As attached Figure 1 As shown, an embodiment of the present invention provides a virtual-real calibration verification system based on a robotic arm, the virtual calibration verification system comprising: a robotic arm 1, a rotating platform 2, an observation camera 3, a detection module 4 and a control module (not shown in the figure), wherein:

[0037] A detachably connected claw and clamp are provided at the front end of the robot arm 1; the rotating platform 2 is arranged at any position in the space that the robot arm 1 can grasp by extension and retraction, and a marker 5 is fixedly provided on the rotating platform 2 so that the marker 5 can rotate synchronously with the rotating platform 2 when the rotating platform 2 rotates; a device for photographing an image of an actual marker is provided on the detection module 4, and the position and posture of the marker in the device coordinate system is obtained by the image of the actual marker photographed; the observation camera 3 is arranged on the detection module, and is used to photograph an image of a virtual model, and the position and posture of the virtual model in the coordinate system of the observation camera 3 is obtained by the image of the virtual model photographed; the detection module and the observation camera 3 are fixedly provided at the end of the robot arm 1 by means of claws and clamps; the control module is electrically or wirelessly connected to the robot arm 1, the rotating platform 2, the observation camera 3 and the detection module 4, and the control module can control the rotation of the rotating platform 2 and move, and drive the marker 5 on the rotating platform 2 to rotate or move synchronously. The control module stores a number of verification models and a number of control programs corresponding to the verification models, which control the extension and retraction of the robotic arm 1 and the claws and fixtures on the robotic arm 1 to grab the marker 5 on the rotating platform 2, so that the control programs can control the robotic arm 1 and the rotating platform 2 to move along a predetermined route, and project the marker 5 in the real world to the virtual world through the control program in the control module to form a corresponding virtual model, so that the marker in the virtual world accurately corresponds to the marker 5 in the real world, calculate the posture transformation of the marker 5 before and after the movement, and obtain the transformation matrix from the real marker to the actual virtual model and then to the ideal virtual model. The transformation matrix is ​​solved to obtain the posture error between the virtual model and the marker 5, and the posture of the marker 5 in the real world coordinate system is determined.

[0038] Specifically, in order to improve the efficiency of the robot arm 1 in grasping the marker for calibration and alignment during the execution of a pre-stored program, reduce the time and unnecessary movements of the robot arm 1 during the movement, and at the same time avoid collision between the robot arm and other surrounding objects, the rotating platform 2 is preferably set at the front end of the robot arm 1, so that the robot arm 1 can reduce rotation and other movements during the calibration process, thereby improving the calibration accuracy and efficiency.

[0039] In this way, the virtual-reality calibration verification system based on the robot arm of the present invention, through the structure of the robot arm 1 and the rotating platform 2, and the claws and clamps are arranged on the robot arm 1, and the observation camera 3 and the detection module 4 are arranged at the end of the robot arm 1, so that under the control of the control module, the rotating platform 2 can be controlled to rotate and move, and the marker 5 on the rotating platform 2 can be driven to rotate and / or move synchronously. By storing a plurality of verification models and a plurality of control programs corresponding to the plurality of verification models in the control module, the extension and retraction of the robot arm 1 and the claws and clamps on the robot arm 1 are controlled to grab the marker 5 located on the rotating platform 2, so that the plurality of control programs can control the robot arm 1 and the rotating platform 2 to move along a predetermined route, and the marker 5 in the virtual world can be aligned with the marker 5 in the real world. The device for capturing the image of the actual marker 5 and the observation camera 3 are used as the two capturing elements to estimate the pose of the actual marker 5 under the two cameras, thereby calculating the transformation matrix from the real world to the observation camera 3; the pose of the virtual model and the observation camera 3 are estimated respectively to obtain the transformation matrix from the actual virtual model to the virtual world. The two matrices are combined to obtain the transformation matrix from the real world to the virtual world, and the transformation matrix from the actual virtual model to the ideal virtual model. The posture error between the virtual model and the marker 5 is obtained by solving the transformation matrix, and the pose of the marker 5 in the real-world coordinate system is determined. This can achieve the effect of high-precision repetition of multiple tracks, and ultimately achieve sufficiently objective and accurate evaluation results, and can ensure high-precision repetition of the trajectory.

[0040] It should be noted that the marker 5 in the real world is projected to the virtual world through the projection program in the control module to form a corresponding virtual model. The virtual model projected by the uncalibrated detection module 4 and the real marker 5 often do not overlap, and only in an ideal situation do the two completely overlap. The essence of combining virtual and real calibration is to ensure that the model of the real marker 5 projected to the virtual world coincides with it through a series of operations. Due to the different specific operations of different calibration methods, in the present invention, the marker or the virtual model is moved to overlap the two, and then the posture transformation before and after the movement is calculated, so as to calculate the transformation matrix from the real marker to the actual virtual model and then to the ideal virtual model. In the process of virtual and real calibration, it is not ruled out that some calibration methods do not need to perform any movement and alignment to perform calculations directly. Therefore, the virtual and real calibration verification system based on the robot arm of the present invention can be applied to the above-mentioned different situations, so that the objectivity and accuracy of virtual and real calibration verification can be guaranteed.

[0041] Specifically, the rotating platform 2 in the present invention can not only realize rotation at any angle, but also can realize lifting and moving in any direction such as front, back, left, right, up and down, so that the markers in the virtual world can be accurately matched with the markers 5 in the real world.

[0042] In some optional implementations of this embodiment, according to the actual calibration needs and different calibration scenarios, a multi-joint multi-degree-of-freedom robotic arm 1 can be selected to implement complex calibration actions and calibration postures. By selecting a suitable robotic arm 1 and equipping different sensors and tools, the calibration verification system of the present invention can have stronger scalability and flexibility. And according to specific needs, suitable hardware equipment can be selected for system integration and optimization to meet the needs of different scenarios and applications, so that the virtual-real calibration verification system based on the robotic arm of the present invention has higher scalability and flexibility.

[0043] At the same time, the robotic arm 1 of the present invention is a programmable high-precision robotic arm. It can be imagined that no matter which robotic arm is selected and in which scenario the virtual-real calibration is performed, the robotic arm 1 in the present invention can be called and executed under the corresponding control module through the written program, so that the robotic arm 1 can realize the predetermined actions and predetermined postures in the written program, and realize the predetermined calibration actions and calibration tasks.

[0044] At the same time, in order to ensure the flexibility of the robot arm 1 while ensuring the strength of the robot arm 1 and facilitating manipulation, the robot arm 1 is preferably configured as a six-axis high-precision robot arm.

[0045] In the robot-based virtual-real calibration verification system of the present invention, the marker 5 is preferably a cube model.

[0046] As some optional implementations of this embodiment, a cylindrical fixing member may be provided at the bottom of the cube model to fix it on the rotating platform 2 .

[0047] In some optional implementations of this embodiment, the above-mentioned marker 5 may also be set as a calibration plate.

[0048] In the virtual-real calibration verification system based on a robotic arm of the present invention, the device for photographing the actual image of the marker is preferably a fisheye camera.

[0049] In some optional implementations of this embodiment, optional models of the observation camera 3 are: D435i depth camera or realsense observation camera.

[0050] In the virtual-real calibration verification system based on a robotic arm of the present invention, the detection module 4 is preferably a head-mounted display, such as a light-transmitting AR head-mounted display.

[0051] Specifically, in the present invention, in addition to the function of fixing the adding and detecting module 4, the fixture can also realize the function of moving the marker or calibration plate for virtual calibration under the control of the control module.

[0052] Specifically, the principles of the virtual-real calibration verification system based on a robotic arm of the present invention include:

[0053] Through the high-precision programmable robotic arm 1, the marker 5 set on the rotating platform 2 can be driven to realize the movement of any posture and any position in the three-dimensional space, and through the rotation and movement of the rotating platform 2 itself, the virtual-real calibration of different calibration models stored in the control module can be realized. The position and posture of the marker 5 is detected by the camera and detection module set on the robotic arm, which can replace manual virtual-real calibration, and can realize the calibration action of the marker 5 in the virtual world at any angle and any posture, and at the same time maintain the accuracy of the action. Combined with the rotating platform 2 to move the marker in the real world, the combination of the robotic arm 1 and the rotating translation stage can achieve the effect of multi-track high-precision repetition, by shooting the actual marker The two shooting elements of the device that captures the image of object 5 and the observation camera 3 respectively estimate the position and posture of the actual marker 5 under the two cameras, thereby calculating the transformation matrix from the real world to the observation camera 3; then estimate the position and posture of the virtual model and the observation camera 3 respectively, and obtain the transformation matrix from the actual virtual model to the virtual world. The two matrices are combined to obtain the transformation matrix from the real world to the virtual world, and the transformation matrix from the actual virtual model to the ideal virtual model. By solving the transformation matrix, the posture error between the virtual model and the marker 5 is obtained, and the position and posture of the marker 5 in the real world coordinate system is determined, finally achieving a sufficiently objective and accurate evaluation result, and being able to ensure the repeatability and high-precision repetition of the trajectory.

[0054] In summary, the virtual-reality calibration verification system based on the robot arm of the present invention can obtain a higher-precision virtual-reality calibration result by accurately controlling the position and posture of the robot arm 1 and combining accurate markers, so that the correspondence between the virtual world and the real world is more accurate, and the calibration accuracy is effectively improved, so that the virtual-reality calibration verification system based on the robot arm of the present invention has a higher calibration accuracy; by using the automatic control capability of the robot arm 1 and the corresponding control module, the automation and repeatability of the calibration process can be realized, thereby ensuring the consistency and stability of the calibration process, further enhancing the reliability of the results, and making the virtual-reality calibration verification system based on the robot arm of the present invention have higher repeatability and stability; at the same time, because the robot arm 1 has the ability to move in multiple degrees of freedom, the robot arm 1 can align objects between the virtual world and the real world in multiple dimensions, including position, posture, shape, etc., providing a more comprehensive alignment capability, making the virtual-reality calibration more accurate and comprehensive, so that the virtual-reality calibration verification system based on the robot arm of the present invention has a multi-dimensional alignment capability. Through the control of the control module and the automated calibration process, work efficiency can be improved and errors caused by manual intervention can be reduced. For example, the robot arm 1 can move according to a pre-written program, and processes such as data collection, calibration algorithm execution and result evaluation can be completed automatically, thereby improving the efficiency and reliability of the entire calibration process, so that the robot arm-based virtual-real calibration verification system of the present invention has a higher automation process.

[0055] Therefore, the present invention provides a virtual-reality calibration and verification system based on a robotic arm, which utilizes a programmable high-precision robotic arm 1 to attach a fisheye observation camera 3 to replace human eyes, and adds a clamp to fix the HMDs thereon; a rotating translation stage is used to take a fixed marker, so that the position and posture of the marker in the real-world coordinate system can be directly determined; and the combination of the robotic arm 1 and the rotating translation stage can also achieve the effect of high-precision repetition of multiple tracks, ultimately achieving a sufficiently objective and accurate evaluation result, and being able to ensure the repeatability and high-precision repetition of the trajectory.

[0056] Second, as Figure 2 As shown, the present invention also provides a virtual-real calibration verification method based on a robotic arm using the virtual-real calibration verification system based on a robotic arm, comprising:

[0057] Call the stored calibration model to obtain the calibrated transformation matrix corresponding to the stored calibration model; obtain the position and posture information of the marker 5 in real time, move the marker 5, realign the calibrated virtual model so that the marker 5 is aligned with the calibration model, record the aligned data, and calculate the posture error between the virtual model and the marker 5; evaluate and verify the calculated posture error between the virtual model and the marker 5.

[0058] With such a configuration, the virtual-reality calibration verification method based on a robotic arm in the present invention can accurately evaluate the accuracy and reliability of the calibration results of the proposed virtual-reality calibration method. Specifically, the virtual-reality calibration verification method proposed in the present invention defines appropriate error metrics to evaluate the calibration results produced by different calibration methods to determine whether the calibration has achieved the expected goal, which helps to improve the virtual-reality calibration algorithm and technology and enhance the interactive experience and application effect.

[0059] Therefore, in order to make the virtual-real calibration structure more accurate and reliable, the accuracy is within the preset range, meet the expected goals, and further improve the interactive experience and application effect, and considering that many devices used for virtual-real calibration in the prior art cannot easily access the pixel coordinate system on the virtual screen, the present invention designs the above-mentioned feasible and more reasonable evaluation and verification indicators to analyze the experimental results, and quantify them through the following formulas. Specifically, the displacement error, that is, the distance error between the marker 5 in the virtual world and the marker 5 in the real world is determined by the Euclidean distance in the three-dimensional space, and the rotation error, that is, the shortest path between two three-dimensional rotations, that is, the error of the shortest path between the marker 5 in the virtual world and the marker 5 in the real world is comprehensively verified to verify the result of virtual-real calibration.

[0060] It should be noted that in some virtual-real calibration processes, after obtaining the position and posture information of the marker 5 in real time, it is necessary to move the marker 5, realign the calibrated virtual model, and record the aligned data. Since some calibration methods require the above operation, realignment is required, but there are also some calibration methods that do not require alignment. The common point of these two methods is that they both need to record the position and posture of the actual cube and the virtual model, and the virtual-real calibration verification system based on the robot arm and the virtual-real calibration verification method based on the robot arm of the present invention can realize the position and posture estimation of the actual marker 5 and the virtual model. In addition, since the different principles of different calibration methods are reflected in the execution logic of different control programs, that is, the execution mode of different projection programs, therefore, in order to make the virtual-real calibration verification system based on the robot arm corresponding to the virtual-real calibration verification method based on the robot arm of the present invention better perform virtual-real calibration verification, the above control module can also be set as a host computer or equipped with a host computer and a control module for use, which plays a role in adapting different projection programs and optimizing and adjusting virtual-real calibration according to the actual needs of different calibration methods. That is, the robot-based virtual-reality calibration verification method of the present invention can not only be run on a robot-based virtual-reality calibration verification system, and further compare and verify its calibration effects according to unified standards under the same experimental conditions, but also objectively compare which projection program corresponds to a more accurate and effective calibration method.

[0061] Specifically, the above displacement error in the present invention is calculated by the following formula:

[0062]

[0063] in: represents the displacement error, Δx, Δy and Δz are the offsets of the virtual model before and after alignment with the marker 5 .

[0064] Specifically, the rotation error in the present invention is calculated by the following formula:

[0065]

[0066] Where: L g represents the rotation error, R g and R 2 are the rotation matrices of the cube markers before and after realignment, respectively. tr() represents the trace of the matrix, which is numerically equal to the sum of the elements on the main diagonal of the matrix.

[0067] Therefore, through the verification results of the above-mentioned displacement error and rotation error of the present invention, the calibrated structure is quantified from the perspective of three-dimensional space, with high verification accuracy, which is helpful to improve the virtual-reality calibration algorithm and technology, and provide a quantifiable basis for improving the virtual-reality calibration method, which can more effectively improve the interactive experience and application effect.

[0068] Specifically, in order to be able to use the above-mentioned robot-based virtual-reality calibration and verification system to better realize the implementation of the robot-based virtual-reality calibration and verification method of the present invention, the initial posture of the robot arm 1 is pre-set in the calibration method of the present invention, and the AR headset and D435i depth observation camera 3 are connected together using electric mechanical grippers and homemade fixtures and fixed at the end of the robot arm 1. Then, the homemade cube marker is fixed on the rotating translation stage to form the entire verification system for verification.

[0069] Specifically, in order to enable the above-mentioned virtual-reality calibration verification system of the present invention to evaluate the accuracy of the calibration system and the calibration method from different angles by establishing calibration effect comparison experiments of different calibration methods under the same trajectory and calibration effect comparison experiments of the same calibration method under different trajectories, and at the same time make the virtual-reality calibration system have high-precision repeatability for the trajectory, the virtual-reality calibration verification method based on a robotic arm of the present invention also includes collecting a data set for comparative testing of different calibration methods.

[0070] Specifically, the calibration effect comparison experiment of different calibration methods under the same trajectory mentioned above and the different trajectories include five situations: regular forward and backward movement in space; regular left and right movement in space; random movement; user-defined trajectory and system-based scene-based trajectory generation.

[0071] Therefore, in the virtual-reality calibration and verification system and verification method based on the robot arm of the present invention, compared with manual operation, the robot arm 1 can provide high-precision motion and position control, and thus can provide accurate and reliable data and results in the calibration and verification process; at the same time, automated processing can be achieved in the process of virtual-reality calibration without manual intervention or multiple manual data collection; through the automatic movement and data collection of the machine detection module 4 of the robot arm 1 and the observation camera 3, time and labor costs are saved; and the accuracy of the calibration method can be evaluated from different angles by establishing a calibration effect comparison experiment of different calibration methods under the same trajectory and a calibration effect comparison experiment of the same calibration method under different trajectories, which can ensure the repeatability and high-precision repetition of the trajectory, making the comparison of the calibration method more objective and accurate.

[0072] In summary, the virtual-reality calibration verification system and verification method based on a robotic arm of the present invention adopts a high-precision programmable and high-degree-of-freedom robotic arm 1: Using a high-precision programmable and high-degree-of-freedom robotic arm 1 to replace manual operation can greatly improve the objectivity of the calibration results and avoid the uncertainty caused by human errors. At the same time, according to different calibration needs, motion planning and path generation can be performed, and the virtual-reality calibration verification process can be divided into appropriate steps and paths through appropriate motion planning algorithms and path generation technologies; by optimizing path generation, the efficiency and accuracy of the movement can be improved, while avoiding collisions and unnecessary movements.

[0073] At the same time, in order to achieve more accurate calibration results, visual perception and sensors can be integrated to achieve more accurate correspondence between objects in the virtual world and objects in the real world, to achieve a more realistic and accurate interactive and integrated experience, and to accurately align the position, size, posture and other information of virtual objects and real objects output by the computer vision system to ensure the consistency of virtual content with the real scene. Specifically, visual perception technology and other sensors are combined to achieve perception and data collection of the calibration and verification process, and the robot arm 1 is interfaced with the observation camera 3, laser scanner or other sensors to obtain the required data for calibration and verification.

[0074] In summary, in order to solve the problems in the prior art that the evaluation of virtual-reality calibration is not objective, handheld markers limit the reachable range of the markers, resulting in an unbalanced performance of the calibration space, and the inability to directly give the position and posture of the real cube marker in the world coordinate system, in order to solve the above-mentioned problems existing in the prior art, the present application integrates the experimental conditions and devices required for virtual-reality calibration to form a virtual-reality calibration verification system based on a robotic arm of the present invention. At the same time, compared with the manual verification of the traditional method, in the present invention, a plurality of high-precision experimental devices and equipment are adopted, such as a high-precision six-axis robotic arm, a high-precision rotating translation stage, markers with different position and posture identification codes, and a self-developed optical head display adapted to different programs, etc., through the structure of the robotic arm 1 and the rotating platform 2, and claws and clamps are set on the robotic arm 1, and the observation camera 3 and the detection module 4 are set at the end of the robotic arm 1, so that under the control of the control module, the rotating platform 2 can be controlled to rotate and move, and the marker 5 on the rotating platform 2 can be driven to rotate and / or move synchronously, and a plurality of verification modules are stored in the control module. The invention relates to a robot 1 and a control program corresponding to the verification model, which controls the extension and retraction of the robot arm 1 and the claws and fixtures on the robot arm 1 to grasp the marker 5 located on the rotating platform 2, so that the control programs can control the robot arm 1 and the rotating platform 2 to move along the predetermined route, accurately correspond the marker 5 in the virtual world to the marker 5 in the real world, and estimate the pose of the actual marker 5 under the two cameras through two shooting elements, namely, the device for shooting the image of the actual marker 5 and the observation camera 3, so as to calculate the transformation matrix from the real world to the observation camera 3; then estimate the pose of the virtual model and the observation camera 3 respectively, and obtain the transformation matrix from the actual virtual model to the virtual world; the two matrices are combined to obtain the transformation matrix from the real world to the virtual world, and the transformation matrix from the actual virtual model to the ideal virtual model; and by solving the transformation matrix, the posture error between the virtual model and the marker 5 is obtained, and the pose of the marker 5 in the real world coordinate system is determined, so as to achieve the effect of high-precision repetition of multiple tracks, and finally achieve a sufficiently objective and accurate evaluation result, and ensure high-precision repetition of the trajectory. At the same time, it can also carry different calibration methods and execute different projection programs corresponding to different calibration methods, so as to select the optimal calibration method and the calibration path corresponding to the calibration method, etc., making the virtual-real calibration more efficient and accurate.

[0075] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In addition, "front", "back", "left", "right", "upper" and "lower" in this article are all referenced to the placement state shown in the accompanying drawings.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A virtual-real calibration verification system based on a robotic arm, characterized in that: include: A mechanical arm, wherein a detachably connected claw and a clamp are provided at the front end of the mechanical arm; A rotating platform, the rotating platform is arranged at any position in the space that the mechanical arm can grasp by extension and retraction, and a marker is fixedly arranged on the rotating platform so that the marker can rotate synchronously with the rotating platform when the rotating platform rotates; A detection module, on which a device for photographing an actual image of the marker is disposed, and the position and posture of the marker in the device coordinate system is obtained by using the photographed actual image of the marker; An observation camera, which is arranged on the detection module and is used to capture an image of the virtual model and obtain the position and posture of the virtual model in the observation camera coordinate system by using the captured virtual model image; the detection module and the observation camera are fixedly arranged at the end of the robotic arm through the claw and the clamp; A control module, wherein the control module is electrically or wirelessly connected to the robotic arm, the rotating platform, the camera and the detection module, the control module can control the rotation and movement of the rotating platform, and drive the marker on the rotating platform to rotate or move synchronously, and the control module stores a plurality of verification models and a plurality of control programs corresponding to the verification models, controls the extension and retraction of the robotic arm and the gripping of the marker on the rotating platform by the claw and the fixture on the robotic arm, so that the plurality of control programs can control the robotic arm and the rotating platform to move along a predetermined route, projects the marker in the real world into the virtual world through the control program in the control module, forms a corresponding virtual model, makes the marker in the virtual world accurately correspond to the marker in the real world, calculates the posture transformation of the marker before and after movement, obtains the transformation matrix from the real marker to the actual virtual model and then to the ideal virtual model, solves the transformation matrix, obtains the posture error between the virtual model and the marker, and determines the posture of the marker in the real world coordinate system.

2. The virtual-real calibration verification system based on a robotic arm according to claim 1, characterized in that: The marker includes a cubic model or a calibration plate.

3. The virtual-real calibration verification system based on a robotic arm according to claim 1, characterized in that: The device for taking the actual image of the marker includes a fisheye camera.

4. The virtual-real calibration verification system based on a robotic arm according to claim 1, characterized in that: The detection module includes a head display.

5. A virtual-real calibration verification method based on a robotic arm using the virtual-real calibration verification system based on a robotic arm according to any one of claims 1 to 4, characterized in that: include: Calling the stored calibration model to obtain the calibrated transformation matrix corresponding to the stored calibration model; Acquire the position and posture information of the marker in real time, move the marker so that it is aligned with the calibration model, record the aligned data, and calculate the posture error between the virtual model and the marker; The pose error between the calculated virtual model and the marker is evaluated and verified.

6. The virtual-real calibration verification method based on a robotic arm according to claim 5, characterized in that: It also includes the acquisition of data sets for comparative testing of different calibration methods.

7. The virtual-real calibration verification method based on a robotic arm according to claim 5, characterized in that: The posture error between the calculated virtual model and the marker is evaluated and verified, including displacement error and rotation error.

8. The virtual-real calibration verification method based on a robotic arm according to claim 7, characterized in that: The displacement error is calculated by the following formula: in: represents the displacement error, and Δx, Δy, and Δz are the offsets of the virtual model before and after alignment with the marker.

9. The virtual-real calibration verification method based on a robotic arm according to claim 7, characterized in that: The rotation error is calculated by the following formula: Where: L g represents the rotation error, R g R2 and R3 are the rotation matrices of the cube markers before and after realignment, respectively, and tr() represents the trace used to calculate the matrix, which is numerically equal to the sum of the elements on the main diagonal of the matrix.

Images