Non-contact interaction method and apparatus
By acquiring human image data and calculating the absolute pose of the human body, the problem of recognition dead zone and expensive and bulky equipment in existing non-contact interaction methods has been solved, realizing accurate interaction without the need for specific equipment and improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INSTITUTE OF PETROCHEMICAL TECHNOLOGY
- Filing Date
- 2022-07-13
- Publication Date
- 2026-05-19
AI Technical Summary
Existing contactless interaction methods, such as gesture interaction and head movement interaction, have problems such as recognition dead zones and expensive and bulky devices, which affect the user experience.
By acquiring human image data and identifying key joints, and using depth cameras and preset matrix transformation technology, the absolute pose of the human body is calculated to achieve non-contact interaction.
It improves the user experience, solves the problems of recognition dead zones and expensive, bulky devices in traditional interaction methods, and achieves accurate interaction without the need for specific devices.
Smart Images
Figure CN115494938B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of touch technology, specifically relating to a non-contact interaction method and device. Background Technology
[0002] With the advancement of technology, contactless interaction methods have been further developed and applied. Currently, the mainstream contactless interaction methods are divided into gesture interaction and head movement interaction. However, in practical applications, they have the following drawbacks:
[0003] Based on gesture interaction and employing motion-sensing technology, people can directly interact with devices such as mobile phones and computers using gestures, achieving a natural and comfortable user experience without the need for any control devices such as mice, keyboards, or touchscreens. Currently, the most commonly used gesture interaction is the motion-sensing controller, where users can browse web pages, read articles, view photos, and play music simply by waving a finger. Even without using any pens or brushes, users can draw, doodle, and design with their fingertips. However, when users make hand gestures, sometimes the palm may obscure the fingers, causing the motion-sensing controller to fail to collect information from the fingers obscured by the palm, thus creating a recognition dead zone and causing recognition defects.
[0004] Currently, head-motion interaction is used in devices such as VR (Virtual Reality) glasses and sensor-based 3D glasses. VR technology creates and allows users to experience a virtual world through computer simulation systems. It utilizes computer-generated simulated environments, employing multi-source information fusion to create interactive, three-dimensional dynamic visuals and simulate physical behaviors, immersing the user in the environment. For example, using VR glasses, the user's head movements change the perspective of the scene within the glasses. However, this method suffers from poor user experience due to the high cost and generally bulky nature of VR glasses. Another approach is to control the scene using sensor-based 3D glasses. Using cameras, sensors, 3D glasses, and a stylus, users can grasp, rotate, and move virtual objects in 3D space. However, head-motion interaction requires specific 3D glasses with sensors, lacking universality and also being expensive.
[0005] In conclusion, both gesture-based and head-based interaction methods have inherent flaws, which to some extent affect the user experience. Summary of the Invention
[0006] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a non-contact interaction method and device to solve the problem that the non-contact interaction in the prior art has defects and affects the user experience.
[0007] To achieve the above objectives, the present invention adopts the following technical solution: a non-contact interaction method, comprising:
[0008] Acquire human body image data, identify key joint points in the human body image data, determine the three-dimensional coordinates of each key joint point, and obtain the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint point.
[0009] The three-dimensional coordinates are transformed using a preset first rotation matrix and a preset translation vector, converting the camera coordinate system three-dimensional coordinates into the world coordinate system three-dimensional coordinates;
[0010] The absolute pose of the human body is calculated based on the two-dimensional coordinates of key joints, the three-dimensional coordinates of the world coordinate system, and the intrinsic parameter matrix of the camera.
[0011] The absolute pose is used to control the device to perform interaction.
[0012] Furthermore, a depth camera is used to photograph the human body to obtain human image data; the depth camera is positioned above the smart terminal.
[0013] Furthermore, the corresponding two-dimensional coordinates of each key joint are obtained from the three-dimensional coordinates of each joint in the following manner:
[0014]
[0015] Where P is a key node, (x, y) are the two-dimensional coordinates of P, (X, Y, Z) are the three-dimensional coordinates of P, and f x f y The focal lengths in the x and y directions are respectively, (c x c y ) is the center of light, Let be the intrinsic parameter matrix of the camera, and s be a scaling factor.
[0016] Furthermore, the three-dimensional coordinates are transformed using a preset first rotation matrix and a preset translation vector in the following manner, converting the camera coordinate system three-dimensional coordinates into the world coordinate system three-dimensional coordinates.
[0017]
[0018] Where (U, V, W) are the three-dimensional coordinates of joint P in the world coordinate system, and R is the first rotation matrix of the world coordinates relative to the camera coordinates. t is the translation vector of world coordinates relative to camera coordinates.
[0019] Furthermore, the conversion of the camera coordinate system three-dimensional coordinates to the world coordinate system three-dimensional coordinates includes:
[0020] Transforming the 3D coordinates in the camera coordinate system with the camera center as the origin into the 3D coordinates in the world coordinate system with the nose tip as the origin includes:
[0021] With the tip of the nose as the origin, the plane formed by the tip of the nose, the left eye, and the right eye is defined as the first plane;
[0022] When the posture of the human head rotates, the new plane formed by the tip of the nose, the left eye, and the right eye is defined as the second plane;
[0023] Determine the first normal vector of the first plane and the second normal vector of the second plane, and calculate the rotation angle based on the first normal vector and the second normal vector;
[0024] Determine the first vector formed by the nose tip, left eye, and right eye before rotation and the second vector formed by the nose tip, left eye, and right eye after rotation, and calculate the rotation axis based on the first vector and the second vector;
[0025] The second rotation matrix is calculated based on the rotation angle and rotation axis;
[0026] The rotated coordinates are calculated based on the second vector and the second rotation matrix, and the rotated coordinates are determined as the three-dimensional coordinates of the world coordinate system.
[0027] Furthermore, the rotation angle is calculated based on the first and second normal vectors in the following manner:
[0028]
[0029] Where vector n1 is the normal vector of β, vector n2 is the normal vector of α, and θ∈[0°,90°];
[0030] The determination of the first vector formed by the nose tip, left eye, and right eye before rotation and the second vector formed by the nose tip, left eye, and right eye after rotation includes:
[0031] Before rotation, the vectors for the nose, left eye, and right eye are fixed as a(a1,a2,a3), and after rotation, the vectors for the nose, left eye, and right eye are b(b1,b2,b3).
[0032] set up The formula is derived from the definition of cross product.
[0033]
[0034] The rotation axis is calculated based on the first and second vectors in the following manner.
[0035]
[0036] Wherein, N1(n x ny n z () is the axis of rotation;
[0037] The second rotation matrix is calculated based on the rotation angle and rotation axis in the following manner.
[0038]
[0039] The rotated coordinates are calculated using the second vector and the second rotation matrix in the following manner.
[0040]
[0041] C(c1, c2, c3) is a new matrix C = R1 * b obtained by multiplying the rotation matrix R1 with the transformed coordinates b. C is the rotated coordinates.
[0042] Furthermore, based on the two-dimensional coordinates of key joints, the three-dimensional coordinates in the world coordinate system, and the camera's intrinsic parameter matrix, the absolute pose of the human body is calculated using the following method:
[0043]
[0044] Where R2 is the third rotation matrix, t2 is the translation vector, and (R2, t2) is the absolute pose.
[0045] Furthermore, based on the absolute pose, the device to be controlled is controlled and interacted with in the following manner.
[0046] The head orientation is calculated based on the absolute pose and the position of the human head, assuming...
[0047]
[0048] It can be calculated
[0049] pitch = atan2(r) 32 ,r 33 )
[0050]
[0051] yaw = atan2(r 21 ,r 11 )
[0052] Where, r 11 ~r 33 All values are known values. Pitch is the pitch angle, roll is the roll angle, yaw is the yaw angle, and atan2 is the azimuth function.
[0053] Furthermore, the key joint points include the tip of the nose, the left eye, the right eye, the left clavicle, the right clavicle, and the neck.
[0054] This application provides a contactless interactive device, including:
[0055] The acquisition module is used to acquire human image data, identify key joint points in the human image data, determine the three-dimensional coordinates of each key joint point, and obtain the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint point.
[0056] The transformation module is used to transform the three-dimensional coordinates using a preset first rotation matrix and a preset translation vector, converting the three-dimensional coordinates of the camera coordinate system into three-dimensional coordinates of the world coordinate system.
[0057] The calculation module is used to calculate the absolute pose based on the two-dimensional coordinates of the camera coordinate system and the three-dimensional coordinates of the world coordinate system;
[0058] The interaction module is used to control the device to be controlled and interact with it by means of absolute pose.
[0059] The beneficial effects that can be achieved by adopting the above technical solution in this invention include:
[0060] This invention provides a non-contact interaction method and device. The invention acquires human image data, performs coordinate system transformation, and calculates the absolute human pose using the two-dimensional coordinates of key joints, the three-dimensional coordinates of the world coordinate system, and the intrinsic parameter matrix of the camera. This application uses human pose estimation to accurately capture human pose through a camera, thereby realizing non-contact interaction with the device and improving the user experience. Attached Figure Description
[0061] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0062] Figure 1 This is a schematic diagram illustrating the steps of the non-contact interaction method of the present invention;
[0063] Figure 2 This is a schematic diagram showing the angle between the first plane and the second plane of the present invention;
[0064] Figure 3 This is a schematic diagram of the non-contact interactive head posture angle rotation of the present invention;
[0065] Figure 4 This is a schematic diagram of the non-contact interactive device of the present invention. Detailed Implementation
[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0067] The following describes a specific contactless interaction method and apparatus provided in an embodiment of this application, with reference to the accompanying drawings.
[0068] like Figure 1 As shown, the contactless interaction method provided in this application embodiment includes:
[0069] S101, acquire human body image data, identify key joint points in the human body image data, determine the three-dimensional coordinates of each key joint point, and obtain the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint point.
[0070] This application uses a depth camera to photograph the human body, obtaining human image data; the depth camera is positioned above the smart terminal. The human image data includes RGB image data and depth image data of the human body.
[0071] Specifically, a controlled indoor environment needs to be established before acquiring image data: place the depth camera above the computer screen, about 1.5m above the ground; the user needs to be 1.0m to 1.5m in front of the depth camera; when using it, the user follows the pre-set usage plan and moves their head slowly in the horizontal (yaw) and vertical (pitch) directions.
[0072] S102, the three-dimensional coordinates are transformed using a preset first rotation matrix and a preset translation vector, converting the three-dimensional coordinates of the camera coordinate system into three-dimensional coordinates of the world coordinate system;
[0073] S103, calculate the absolute pose of the human body based on the two-dimensional coordinates of key joints, the three-dimensional coordinates of the world coordinate system, and the intrinsic parameter matrix of the camera;
[0074] S104, using the absolute pose, control the device to be controlled to interact with it.
[0075] The working principle of the non-contact interaction method is as follows: This application uses a depth camera to take pictures of the human body, obtaining RGB image data and depth image data of the human body. Key joints in the human body image data are identified through the human body image data, and the three-dimensional coordinates of each key joint are determined. Based on the three-dimensional coordinates of each key joint, the corresponding two-dimensional coordinates are obtained. Through translation and rotation transformations, the three-dimensional coordinate system of the joints is transformed from the camera as the origin to the nose tip as the origin, and the depth direction is transformed from the camera shooting direction to the nose tip facing direction. Using the two-dimensional points of the key joints and the three-dimensional coordinates of the world coordinate system, the absolute pose of the human body is estimated. The absolute pose of the human body includes the position of the human head and the head facing posture. Based on the absolute pose, the device to be controlled is controlled to interact, that is, to control the presentation changes of the scene in the three-dimensional display device.
[0076] The non-contact interaction method proposed in this application uses a depth camera to acquire RGB image data and depth image data of the human body. It does not require specific 3D glasses or any head-mounted devices. It adopts human pose estimation technology, which uses a depth camera to accurately capture human movements for human pose estimation, thus solving the shortcomings of traditional contact interaction and improving the user experience.
[0077] In some embodiments, it is assumed that the coordinates of point p in the two-dimensional image are (x, y), and the relationship between its corresponding three-dimensional coordinates (X, Y, Z) is as shown in formula (1):
[0078]
[0079] Among them, f x f y Let x be the focal length in the x and y directions. Let c be the intrinsic parameter matrix of the camera. x ,c y ) is the center of light, approximated by the coordinates of the image center, and s is a scaling factor.
[0080] The three-dimensional coordinates are transformed using a preset first rotation matrix and a preset translation vector in the following manner, converting the three-dimensional coordinates of the camera coordinate system into three-dimensional coordinates of the world coordinate system, specifically transforming the camera coordinates (X, Y, Z) into (U, V, W).
[0081]
[0082] Where R is the first rotation matrix of world coordinates relative to camera coordinates. t is the translation vector of world coordinates relative to camera coordinates.
[0083] It should be noted that the first rotation matrix R and the translation vector t of the world coordinates relative to the camera coordinates are both preset known quantities.
[0084] In some embodiments, the conversion of the camera coordinate system three-dimensional coordinates to the world coordinate system three-dimensional coordinates includes:
[0085] Transforming the 3D coordinates in the camera coordinate system with the camera center as the origin into the 3D coordinates in the world coordinate system with the nose tip as the origin includes:
[0086] With the tip of the nose as the origin, the plane formed by the tip of the nose, the left eye, and the right eye is defined as the first plane;
[0087] When the posture of the human head rotates, the new plane formed by the tip of the nose, the left eye, and the right eye is defined as the second plane;
[0088] Determine the first normal vector of the first plane and the second normal vector of the second plane, and calculate the rotation angle based on the first normal vector and the second normal vector;
[0089] Determine the first vector formed by the nose tip, left eye, and right eye before rotation and the second vector formed by the nose tip, left eye, and right eye after rotation, and calculate the rotation axis based on the first vector and the second vector;
[0090] The second rotation matrix is calculated based on the rotation angle and rotation axis;
[0091] The rotated coordinates are calculated based on the second vector and the second rotation matrix, and the rotated coordinates are determined as the three-dimensional coordinates of the world coordinate system.
[0092] Specifically, such as Figure 2 As shown, the coordinates of three points—the nose, left eye, and right eye—are obtained. These three points can define a new plane. In this application, the plane formed by the nose tip, left eye, and right eye is selected as the first plane α, with the nose tip as the origin. When the posture of the human head changes, a new plane, the second plane β, is generated. The angle between the two planes can be determined by calculating the angle between the two normal vectors of the first plane α and the second plane β. Vector n1 is the normal vector of β, and vector n2 is the normal vector of α. The formula for calculating the angle between the two planes is as follows:
[0093]
[0094] Then, based on the coordinates of the three points—nose, left eye, and right eye—before and after the plane is rotated, the axis of rotation is calculated. The plane containing the axis of rotation is the plane formed by the first plane α and the second plane β, and the axis of rotation must be perpendicular to this plane.
[0095] Specifically, this includes: before rotation, the vectors for the left and right eyes are fixed as a(a1,a2,a3), and after rotation, the vectors for the left and right eyes are b(b1,b2,b3).
[0096] set up From the definition of cross product,
[0097]
[0098] Therefore, the rotation axis N1(n) x ,n y ,n z )for
[0099]
[0100] The second rotation matrix R1 corresponding to the following formula is calculated based on the rotation axis and the included rotation angle:
[0101]
[0102] C(c1, c2, c3) is a new matrix C = R1 * b obtained by multiplying the rotation matrix R1 by the transformed coordinates b. The resulting C represents the rotated coordinates, which are the three-dimensional coordinates in the world coordinate system. The formula for calculating the coordinate transformation is as follows:
[0103]
[0104] In some embodiments, the key joint points include the tip of the nose, the left eye, the right eye, the left clavicle, the right clavicle, and the neck.
[0105] As a preferred embodiment, this application uses two-dimensional points and three-dimensional coordinate points of a total of six points: the tip of the nose, the left eye, the right eye, the left clavicle, the right clavicle, and the neck, to estimate the absolute pose of the human body;
[0106] The `solvePnP` function in OpenCV can be used to solve for the third rotation matrix R2 and the translation vector t2. By using the coordinates of six points—the tip of the nose, left eye, right eye, left clavicle, right clavicle, and neck—in the 3D scene, along with the camera intrinsic parameter matrix, the absolute pose relationship between the camera coordinate system and the world coordinate system representing the 3D scene structure can be determined. In this application, `solvePnP` is used to solve for the rotation matrix R2 and the translation vector t2.
[0107]
[0108] (R2, t2) represents the absolute pose.
[0109] like Figure 3 As shown, once the third rotation matrix is obtained, the head orientation can be calculated, as follows:
[0110] Assumption
[0111] Where, r 11 ~r 33 All values are known; head attitude is represented by three Euler angles: pitch, roll, and yaw, as shown below. Figure 3 As shown. Therefore, in order to calculate the head pose, three sets of predicted coordinates are used to calculate the three angles.
[0112] pitch = atan2(r) 32 ,r 33 (10)
[0113]
[0114] yaw = atan2(r 21 ,r 11 (12)
[0115] Where, r 11 ~r 33 All values are known values. Pitch is the pitch angle, roll is the roll angle, yaw is the yaw angle, and atan2 is the azimuth function.
[0116] It should be noted that the horizontal range of head movement during testing of this invention was -76° to 76°, and the vertical range was -33° to 33°, with a test distance ranging from 0.8m to 5m. The test results are as follows:
[0117]
[0118] Table 1 Absolute Pose
[0119] like Figure 4 As shown, this application provides a contactless interactive device, comprising:
[0120] The acquisition module 201 is used to acquire human image data, identify key joint points in the human image data, determine the three-dimensional coordinates of each key joint point, and obtain the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint point.
[0121] The conversion module 202 is used to convert the three-dimensional coordinates using a preset first rotation matrix and a preset translation vector, thereby converting the three-dimensional coordinates of the camera coordinate system into three-dimensional coordinates of the world coordinate system.
[0122] Calculation module 203 is used to calculate the absolute pose based on the two-dimensional coordinates of the camera coordinate system and the three-dimensional coordinates of the world coordinate system;
[0123] The interaction module 204 is used to control the device to be controlled for interaction through absolute pose.
[0124] The working principle of the non-contact interactive device provided in this application embodiment is as follows: the acquisition module 201 acquires human image data, identifies key joints in the human image data, determines the three-dimensional coordinates of each key joint, and obtains the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint; the conversion module 202 uses a preset first rotation matrix and a preset translation vector to convert the three-dimensional coordinates of the camera coordinate system to the three-dimensional coordinates of the world coordinate system; the calculation module 203 calculates the absolute pose based on the two-dimensional coordinates of the camera coordinate system and the three-dimensional coordinates of the world coordinate system; and the interaction module 204 controls the device to be controlled to perform interaction based on the absolute pose.
[0125] In summary, this invention provides a non-contact interaction method and apparatus. This application employs human posture estimation to accurately capture human posture using a camera, thereby achieving non-contact interaction with the device and improving the user experience. Furthermore, this application uses a non-contact method to interact with the device, eliminating the need for wearing specific 3D glasses and solving the problem of single-camera positioning limitations.
[0126] It is understood that the method embodiments provided above correspond to the device embodiments described above, and the specific details can be referred to each other, which will not be repeated here.
[0127] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.
[0128] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0129] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction methods implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0130] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0131] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A contactless interaction method, characterized in that, include: Acquire human body image data, identify key joint points in the human body image data, determine the three-dimensional coordinates of each key joint point, and obtain the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint point. The three-dimensional coordinates are transformed using a preset first rotation matrix and a preset translation vector, converting the camera coordinate system three-dimensional coordinates into the world coordinate system three-dimensional coordinates; The absolute pose of the human body is calculated based on the two-dimensional coordinates of key joints, the three-dimensional coordinates of the world coordinate system, and the intrinsic parameter matrix of the camera. The absolute pose is used to control the device to perform interaction; A depth camera is used to photograph the human body to obtain human image data; the depth camera is mounted on top of the smart terminal. The three-dimensional coordinates are transformed using a preset first rotation matrix and a preset translation vector in the following manner, converting the camera coordinate system three-dimensional coordinates into the world coordinate system three-dimensional coordinates. =R +t in,( , , Let P be the 3D coordinates of the joint point P in the world coordinate system, and R be the first rotation matrix of the world coordinates relative to the camera coordinates. t is the translation vector of world coordinates relative to camera coordinates. ; The process of converting the camera coordinate system's three-dimensional coordinates to the world coordinate system includes: Transforming the 3D coordinates in the camera coordinate system with the camera center as the origin into the 3D coordinates in the world coordinate system with the nose tip as the origin includes: With the tip of the nose as the origin, the plane formed by the tip of the nose, the left eye, and the right eye is defined as the first plane; When the posture of the human head rotates, the new plane formed by the tip of the nose, the left eye, and the right eye is defined as the second plane; Determine the first normal vector of the first plane and the second normal vector of the second plane, and calculate the rotation angle based on the first normal vector and the second normal vector; Determine the first vector formed by the nose tip, left eye, and right eye before rotation and the second vector formed by the nose tip, left eye, and right eye after rotation, and calculate the rotation axis based on the first vector and the second vector; The second rotation matrix is calculated based on the rotation angle and rotation axis; The rotated coordinates are calculated based on the second vector and the second rotation matrix, and the rotated coordinates are determined as the three-dimensional coordinates of the world coordinate system.
2. The method according to claim 1, characterized in that, The corresponding two-dimensional coordinates of each key joint are obtained from the three-dimensional coordinates of each joint in the following manner. Where P is a critical joint, ( , ) are the two-dimensional coordinates of P, ( , , Let P be the three-dimensional coordinates. , They are respectively , Focal length of direction, ( , ) is the center of light, Let be the intrinsic parameter matrix of the camera, and s be a scaling factor.
3. The method according to claim 1, characterized in that, The rotation angle is calculated based on the first and second normal vectors in the following manner. Where, vector Let be the normal vector of β, and let be the vector. Let α be the normal vector. ; The determination of the first vector formed by the nose tip, left eye, and right eye before rotation and the second vector formed by the nose tip, left eye, and right eye after rotation includes: Rotate the front nose, left eye, and right eye with fixed vectors. After rotation, the vectors for the nose, left eye, and right eye are: ; set up + k, + k is derived from the definition of the cross product. The rotation axis is calculated based on the first and second vectors in the following manner. = Among them, N1( , , () is the axis of rotation; The second rotation matrix is calculated based on the rotation angle and rotation axis in the following manner. The rotated coordinates are calculated using the second vector and the second rotation matrix in the following manner. =R1 C( , , Let C = R1 be a new matrix obtained by multiplying the rotation matrix R1 by the transformed coordinates b. b and C are the coordinates after rotation.
4. The method according to claim 1, characterized in that, Based on the two-dimensional coordinates of key joints, the three-dimensional coordinates in the world coordinate system, and the camera's intrinsic parameter matrix, the absolute pose of the human body is calculated using the following method. (R2, t2)=slovePnP( , , ) Where R2 is the third rotation matrix, t2 is the translation vector, and (R2, t2) is the absolute pose.
5. The method according to claim 4, characterized in that, Based on the absolute pose, the device to be controlled is controlled and interacted with in the following manner. The head orientation is calculated based on the absolute pose and the position of the human head, assuming... R2= It can be calculated pitch= atan2( , ) roll= atan2( , ) yaw=atan2( , ) in, ~ All values are known values. Pitch is the pitch angle, roll is the roll angle, yaw is the yaw angle, and atan2 is the azimuth function.
6. The method according to claim 1, characterized in that, The key joints include the tip of the nose, left eye, right eye, left clavicle, right clavicle, and neck.
7. A contactless interactive device, applied to the contactless interactive method according to any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire human image data, identify key joint points in the human image data, determine the three-dimensional coordinates of each key joint point, and obtain the corresponding two-dimensional coordinates based on the three-dimensional coordinates of each key joint point. The transformation module is used to transform the three-dimensional coordinates using a preset first rotation matrix and a preset translation vector, converting the three-dimensional coordinates of the camera coordinate system into three-dimensional coordinates of the world coordinate system. The calculation module is used to calculate the absolute pose based on the two-dimensional coordinates of the camera coordinate system and the three-dimensional coordinates of the world coordinate system; The interaction module is used to control the device to be controlled and interact with it by means of absolute pose.