Calibration method based on multiple hand tracking cameras and head-mounted display camera and application

Abstract

The invention relates to a calibration method and application based on a multi-hand tracking camera and a head-mounted display camera, and the method comprises the steps: setting identification pieces which are respectively captured by the multi-hand tracking camera and the head-mounted display camera, and calibrating the multi-hand tracking camera and the head-mounted display camera through the relation between the corresponding coordinate systems of the multi-hand tracking camera and the head-mounted display camera and a world coordinate system; the method is applied to calibration and tracking of hand movement in a fixed space. According to the method, high-precision calibration is realized by customizing the identification piece and combining data acquisition and coordinate conversion of a multi-camera system, accurate projection of the hand articulation points under the coordinates of the head-mounted display camera is ensured, the quality and reliability of truth value data used for model training can be ensured through accurate coordinate conversion, the accuracy and generalization ability of the model can be improved, and the method is suitable for popularization and application. And the application process of the hand tracking technology in the fields of virtual reality and augmented reality is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of general image data processing or generation, and in particular to a calibration method and application based on multiple hand tracking cameras and head-mounted display cameras. Background Art

[0002] With the rapid development of the leisure and entertainment industry, numerous terminal simulators for virtual reality environments and augmented reality environments have emerged to meet users' needs for a stronger sense of control and immersion.

[0003] With the development of virtual reality (VR) and augmented reality (AR) technologies, hand tracking technology has become increasingly important in these applications. Reasonable hand tracking can enable users to interact naturally with the virtual environment and improve the user experience. An important condition for achieving high-precision hand tracking is to accurately calibrate between multiple cameras and sensors to ensure that hand movements can be correctly represented in different coordinate systems.

[0004] The training model of traditional hand tracking systems usually requires setting up a large amount of labeled data, but it is usually impossible to directly obtain data in the head-mounted display coordinate system through the hand tracking system. There is a lack of effective calibration methods in multi-camera systems. This lack of calibration may lead to inaccurate projection of hand joint coordinates, thus affecting subsequent model training and application effects.

[0005] In the prior art, when calibrating the coordinate conversion between the multi-camera system and the head-mounted display camera, problems such as complex equipment layout, insufficient calibration accuracy and cumbersome operation are often encountered, which affects the application of the hand tracking system in the virtual environment and leads to the problem of being unable to complete smooth and accurate hand tracking. Summary of the invention

[0006] The present invention solves the problems existing in the prior art and provides a calibration method and application based on multiple hand tracking cameras and head-mounted display cameras.

[0007] The technical solution adopted by the present invention is a calibration method based on multiple hand tracking cameras and head-mounted display cameras. The method sets identification parts, which are captured by the multiple hand tracking cameras and head-mounted display cameras respectively, and calibrates the multiple hand tracking cameras and head-mounted display cameras through the relationship between their corresponding coordinate systems and the world coordinate system.

[0008] Preferably, the identification element includes an identification code and corresponding luminous points arranged around the identification code.

[0009] Preferably, a plurality of hand tracking cameras are arranged in the space to be calibrated, and the identification piece is set in the space; the information of the light-emitting point is obtained by the plurality of hand tracking cameras, and the information of the identification code is obtained by the head-mounted display camera; in the process of setting the identification piece, it can be set on the inner wall of the space, or it can be operated manually or non-humanly and suspended at any position in the space.

[0010] Preferably, the position of the hand tracking camera is deployed and fixed, the camera intrinsic parameters are calibrated using a standard calibration process, the external parameters of each hand tracking camera are calibrated, and a conversion relationship between any hand tracking camera and the world coordinate system is established; the internal and external parameters of the head-mounted display camera are calibrated, and the camera position is fixed; when measuring a set of data, the hand tracking camera, the head-mounted display camera and the identification component are fixed relative to each other.

[0011] Preferably, the coordinate data of the light-emitting point on the hand tracking camera is obtained, the coordinate data of the identification code on the head-mounted display camera is obtained, and the coordinates of the light-emitting point are transformed into the head-mounted display coordinate system.

[0012] Preferably, let p head is the coordinate data of the light-emitting point on the hand tracking camera, p c is the coordinate data of the identification code in the head-mounted display camera. The rotation matrix R and the translation vector t describe the relationship between the corresponding coordinate systems. The coordinate transformation formula is p c =R·p head +t.

[0013] Preferably, n sets of two-dimensional code position points and corresponding fluorescent spot coordinate pairs are collected. Let the transformation error and E satisfy,

[0014]

[0015] Minimize E and update the rotation matrix R and translation vector t.

[0016] Preferably, after the calibration is completed, the coordinate data of the converted hand joint points are output, and the hand joint points include palm joint points, finger joint points and wrist joint points.

[0017] Preferably, the identification members are one or more groups; in fact, under the premise of setting a group of identification members, multiple groups of data can be obtained by moving the identification members, so that the final calibration result is more credible.

[0018] An application of the calibration method based on multiple hand tracking cameras and head-mounted display cameras is applied to the calibration and tracking of hand movements in a fixed space.

[0019] The present invention relates to a calibration method and application based on multiple hand tracking cameras and head-mounted display cameras. Identification pieces are set and captured by the multiple hand tracking cameras and head-mounted display cameras respectively. The multiple hand tracking cameras and head-mounted display cameras are calibrated through the relationship between their corresponding coordinate systems and the world coordinate system. The method is applied to the calibration and tracking of hand movements in a fixed space.

[0020] The beneficial effect of the present invention is that customized identification parts are combined with data collection and coordinate transformation of a multi-camera system to achieve high-precision calibration, ensuring the accurate projection of hand joints under the coordinates of the head-mounted display camera. Accurate coordinate transformation can ensure the quality and reliability of the true value data used in model training, help improve the accuracy and generalization ability of the model, and promote the application of hand tracking technology in the fields of virtual reality and augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of the method of the present invention;

[0022] Figure 2 This is an application scenario diagram of the present invention;

[0023] Figure 3 The figure is an application flow chart of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below in conjunction with embodiments, but the protection scope of the present invention is not limited thereto.

[0025] The present invention relates to a calibration method based on multiple hand tracking cameras and head-mounted display cameras. The method sets identification pieces, which are captured by the multiple hand tracking cameras and the head-mounted display cameras respectively, and calibrates the multiple hand tracking cameras and the head-mounted display cameras through the relationship between their corresponding coordinate systems and the world coordinate system.

[0026] In the present invention, an integrated identification component is provided, and the corresponding data signals are captured by the multi-hand tracking camera and the head-mounted display camera respectively. By combining the relationship between the corresponding coordinate systems of the multi-hand tracking camera and the head-mounted display camera and the world coordinate system, the calibration of the multi-hand tracking camera and the head-mounted display camera is completed, and the spatial mapping relationship between the two is obtained.

[0027] In the present invention, multiple hand tracking cameras are arranged in the space to be calibrated. They are generally arranged on the wall at the upper part of the space to avoid obstruction as much as possible to ensure that the hand information can be fully obtained. Generally speaking, hand tracking cameras are arranged at equal heights on each wall at fixed intervals.

[0028] Obviously, the identification element here needs to be recognized by the multi-hand tracking camera and the head-mounted display camera simultaneously. For the convenience of setting, it is generally set on any inner wall of the space and set as a combination of the identification code and the corresponding light points set around it. What can be collected from the shooting of the multi-hand tracking camera at the far end is a complete and sufficient light point to be captured, and the identification code in it is sufficient to be captured by the head-mounted display camera;

[0029] In the specific implementation process, the identification code is set to include but not limited to QR code, barcode, etc., generally a QR code, which not only facilitates rapid positioning, but also provides a benchmark for subsequent spatial calculations through the known size and arrangement relationship of the QR code.

[0030] In the present invention, the relationship between the corresponding coordinate systems of the multi-hand tracking cameras and the head-mounted display camera and the world coordinate system is combined to complete the calibration of the multi-hand tracking cameras and the head-mounted display camera, including the following steps:

[0031] S1 deploys and fixes the position of the hand tracking camera, uses the standard calibration process to calibrate the camera's intrinsic parameters, calibrates the extrinsic parameters of each hand tracking camera, and establishes the transformation relationship between any hand tracking camera and the world coordinate system; calibrates the intrinsic and extrinsic parameters of the head-mounted display camera, and fixes the camera position;

[0032] S2 obtains the coordinate data of the light-emitting point on the hand tracking camera, obtains the coordinate data of the identification code on the head-mounted display camera, and transforms the coordinates of the light-emitting point into the head-mounted display coordinate system.

[0033] Let p head is the coordinate data of the light-emitting point on the hand tracking camera, p c is the coordinate data of the identification code in the head-mounted display camera. The rotation matrix R and the translation vector t describe the relationship between the corresponding coordinate systems. The coordinate transformation formula is p c =R·p head +t.

[0034] It should be noted that during the data collection process, the QR code may introduce certain noise due to the error of the motion trajectory. The motion trajectory refers to the trajectory of the moving identification part. In order to eliminate these errors, the present invention uses the least squares method to optimize the rotation matrix and translation vector in the coordinate transformation formula, that is:

[0035] Collect n sets of QR code position points and corresponding fluorescent spot coordinate pairs Let the transformation error and E satisfy,

[0036]

[0037] Minimize E and update the rotation matrix R and translation vector t.

[0038] In the present invention, after the calibration is completed, the coordinate data of the converted hand joints are output. The hand joints include palm joints, finger joints and wrist joints. In order to ensure the accuracy of the training data, the local coordinate system here is established based on the wrist, and the coordinate data of the head display camera coordinate system is accurately mapped to each hand joint. The joints here are different from the joints in the sense of bones, and generally include:

[0039] Palm joint point: usually the reference point in the center of the hand;

[0040] Finger joints: the base, middle and tip of each finger (including thumb, index finger, middle finger, ring finger and little finger);

[0041] Wrist joint: The point where the hand connects to the forearm.

[0042] In practical applications, considering that the application of a group of identification parts may bring certain limitations, the identification parts can be set into one or more groups. By judging the confidence of each group of identification parts, the optimal rotation matrix and translation vector can be collected. This confidence is related to the indoor ambient light, angle, whether there are obstructions, etc.

[0043] The present invention also relates to an application of the calibration method based on multiple hand tracking cameras and head-mounted display cameras, which is applied to the calibration and tracking of hand movements in a fixed space.

[0044] A specific application example is given below:

[0045] (1) Design and produce a 2x2 QR code calibration plate, where each QR code has a unique code for easy identification and distinction;

[0046] (2) Paste high-brightness, easily identifiable marker fluorescent dots on the four corners of each QR code as feature points;

[0047] (3) Deploy and fix the positions of multiple hand tracking cameras, use the standard calibration process to calibrate the camera intrinsic parameters, perform extrinsic calibration on each camera, determine their positions relative to the predefined world coordinate system, and establish a unified world coordinate system as the basis for subsequent calculations;

[0048] (4) Build the head-mounted display data acquisition system, calibrate the camera's internal and external parameters, and fix the camera position;

[0049] (5) Obtaining coordinate data of the fluorescent point on the multi-hand tracking camera, obtaining coordinate data of the QR code on the head-mounted display system, and transforming the coordinates of the fluorescent point into the head-mounted display coordinate system;

[0050] The intrinsic parameter matrix K of each camera is used to describe the internal geometric characteristics of the camera. It is usually a 3x3 matrix that satisfies

[0051] The coordinate transformation formula is p c =R·p head +t, where R is the rotation matrix of the hand tracking camera coordinate system relative to the head display camera, and t is the translation vector of the hand tracking camera coordinate system relative to the head display camera;

[0052] During the data collection process, the movement trajectories of the QR code and the fluorescent spot in different directions are optimized using the least squares method; n QR code position points and corresponding fluorescent spot coordinate pairs are collected. Minimize the error function E of the transformation error sum of these points, Get the optimal rotation matrix and translation vector

[0053] (6) Verification and adjustment: verify whether the converted coordinate data is accurately aligned. If not, repeat the above steps;

[0054] (7) Output the converted coordinate data as the true value data for model training.

[0055] The present invention also relates to a computer-readable storage medium in its application, on which a calibration program based on multiple hand tracking cameras and head-mounted display cameras is stored. When the program is executed by a processor, the above-mentioned calibration method based on multiple hand tracking cameras and head-mounted display cameras is implemented.

[0056] The present invention also relates to a computer device in its application, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned calibration method based on multiple hand tracking cameras and head-mounted display cameras is implemented.

[0057] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0058] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0059] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0060] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0061] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0062] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A calibration method based on multiple hand tracking cameras and head-mounted display cameras, characterized in that: The method sets identification pieces, which are captured by a multi-hand tracking camera and a head-mounted display camera respectively, and calibrates the multi-hand tracking camera and the head-mounted display camera through the relationship between the corresponding coordinate systems of the multi-hand tracking camera and the head-mounted display camera and the world coordinate system.

2. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 1, characterized in that: The identification element comprises an identification code and luminous points correspondingly arranged around the identification code.

3. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 2, characterized in that: A plurality of hand tracking cameras are arranged in the space to be calibrated, and the identification element is set in the space; the information of the light-emitting point is obtained by the plurality of hand tracking cameras, and the information of the identification code is obtained by the head display camera.

4. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 3, characterized in that: Deploy and fix the position of the hand tracking camera, use the standard calibration process to calibrate the camera's intrinsic parameters, calibrate the extrinsic parameters of each hand tracking camera, and establish the transformation relationship between any hand tracking camera and the world coordinate system; calibrate the intrinsic and extrinsic parameters of the head-mounted display camera and fix the camera position.

5. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 3, characterized in that: Obtain the coordinate data of the light-emitting point on the hand tracking camera, obtain the coordinate data of the identification code on the head-mounted display camera, and transform the coordinates of the light-emitting point to the head-mounted display coordinate system.

6. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 5, characterized in that: Let p head is the coordinate data of the light point on the hand tracking camera, p c is the coordinate data of the identification code in the head-mounted display camera. The rotation matrix R and the translation vector t describe the relationship between the corresponding coordinate systems. The coordinate transformation formula is p c =R·p head +t.

7. The calibration method based on multiple hand tracking cameras and head mounted display cameras according to claim 6, characterized in that: Collect n sets of QR code position points and corresponding fluorescent spot coordinate pairs Let the transformation error and E satisfy, Minimize E and update the rotation matrix R and translation vector t.

8. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 1, characterized in that: After the calibration is completed, the coordinate data of the converted hand joint points are output, and the hand joint points include palm joint points, finger joint points and wrist joint points.

9. The calibration method based on multiple hand tracking cameras and head-mounted display cameras according to claim 1, characterized in that: The identification elements are in one group or multiple groups.

10. An application of the calibration method based on multiple hand tracking cameras and head-mounted display cameras according to any one of claims 1 to 9, characterized in that: Applied to calibration and tracking of hand movements in fixed space.

Images