Gesture recognition method, device, system and readable storage medium of wearable system

Abstract

The application relates to the technical field of three-dimensional gesture recognition, and discloses a gesture recognition method, device and system of a wearable system and a readable storage medium. The method comprises the following steps: acquiring image data and IMU data collected by a wearable system, the wearable system comprising a head-mounted device and a hand-mounted IMU device; converting the IMU data to a coordinate system where a wrist is located, obtaining target IMU data under the coordinate where the wrist is located, fusing the target IMU data and the image data based on the corresponding relationship between the image data and the target IMU data, and obtaining hand pose information; and determining a corresponding target gesture based on the hand pose information. Through the implementation of the application, the IMU data and the image data are fused, the finger joint movement data and the wrist movement data are combined, the hand movement data is determined, the hand movement information is enriched, the hand movement data can accurately represent the hand action, and the recognition precision of the gesture is improved.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional gesture recognition technology, and more specifically to a gesture recognition method, device, system, and readable storage medium for a wearable system. Background Technology

[0002] Most existing head-mounted devices (AR / VR glasses) rely on controllers for button-based interaction. Gesture interaction technology is still immature, and due to unavoidable occlusion, limited camera field of view, and limited mobile computing power, relying solely on the visual sensors on head-mounted devices for gesture interaction results in low accuracy and robustness. Therefore, visual sensors need to be integrated with other sensors to improve the accuracy of 3D gesture recognition. With the increasing use of Inertial Measurement Units (IMUs), the current common approach to address the above issues is to fuse visual sensors with IMUs to provide rich hand motion information. In gesture recognition, the 3D pose of the wrist is crucial for the accurate estimation of the entire hand's 3D pose. However, current 3D gesture estimation algorithms based on visual sensors (whether 2D or 3D cameras) do not consider the accurate estimation of the wrist's 3D pose, making it difficult for existing 3D gesture estimation algorithms to accurately recognize the hand's 3D pose. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a gesture recognition method, device, system and readable storage medium for wearable systems to solve the problem that the prior art does not take into account the three-dimensional pose of the wrist and therefore cannot accurately recognize gestures.

[0004] According to a first aspect, embodiments of the present invention provide a gesture recognition method for a wearable system, comprising: acquiring image data and IMU data collected by the wearable system, wherein the wearable system includes a head-mounted device and a wrist-worn IMU device; converting the IMU data to the coordinate system of the wrist to obtain target IMU data in the coordinate system of the wrist; fusing the target IMU data and the image data based on the correspondence between the image data and the target IMU data to obtain hand pose information; and determining the corresponding target gesture based on the hand pose information.

[0005] The gesture recognition method for wearable systems provided in this invention utilizes image data and IMU data collected by the wearable system. The IMU data is converted to the coordinate system of the wrist to obtain target IMU data in that coordinate system, which represents wrist movement data. Then, based on the correspondence between image data and target IMU data, the target IMU data and image data are fused to obtain hand pose information. Finally, the corresponding target gesture is determined based on this hand pose information. This method converts IMU data to the wrist coordinate system when determining the target gesture, ensuring that the IMU data accurately represents wrist movements. It further determines hand movements based on these wrist movements and improves gesture recognition accuracy by fusing target IMU data and image data.

[0006] In conjunction with the first aspect, in the first embodiment of the first aspect, the step of fusing the target IMU data and the image data based on the correspondence between the image data and the target IMU data to obtain hand pose information includes: acquiring the acquisition time of the image data, determining the correspondence between the image data and the target IMU data at the acquisition time; and performing data synchronization fusion of the image data and the target IMU data based on the correspondence to obtain hand pose information at each acquisition time.

[0007] The gesture recognition method for wearable systems provided in this invention determines the correspondence between image data and target IMU data at the acquisition time by determining the correspondence between the image data and the target IMU data at the acquisition time, and then fuses the image data and the target IMU data according to the correspondence to obtain rich hand movement information and improve the robustness of gesture interaction.

[0008] In conjunction with the first embodiment of the first aspect, in the second embodiment of the first aspect, the step of performing data synchronization fusion of the image data and the target IMU data based on the correspondence to obtain hand pose information at each acquisition time includes: calculating the finger joint motion state quantity of finger joint motion data at the acquisition time, and the wrist motion state quantity of wrist motion data at the acquisition time, wherein the finger joint motion state quantity and the wrist motion state quantity constitute the hand motion state quantity; determining the hand motion observation quantity at the acquisition time based on a preset observation model and the image data; fusing the hand motion observation quantity and the hand motion state quantity at the same acquisition time to obtain the target motion state quantity of the hand at the acquisition time; and determining the hand pose information based on the target motion state quantity.

[0009] The gesture recognition method for wearable systems provided in this invention calculates the motion state quantities of finger joints and wrists to facilitate the determination of the current hand motion state quantity. Then, based on the hand motion state quantity and hand motion observation quantity, the target motion state quantity of the hand is determined to determine the current hand pose information. This method improves the accuracy of hand motion state recognition by fusing hand motion state quantity and hand motion observation quantity.

[0010] In conjunction with the second embodiment of the first aspect, in the third embodiment of the first aspect, the calculation of the finger joint motion state quantity at the acquisition time includes: constructing a motion prediction model of the finger joint based on the motion state of the finger joint; discretizing and integrating the motion prediction model, and calculating the finger joint motion state quantity at the acquisition time based on the extended Kalman filter method.

[0011] In conjunction with the second embodiment of the first aspect, in the fourth embodiment of the first aspect, calculating the wrist motion state quantity at the acquisition time of the wrist motion data includes: acquiring a wrist motion model; performing discretization and integration processing on the wrist motion model; and calculating the wrist motion state quantity at the acquisition time based on the extended Kalman filter method.

[0012] The gesture recognition method for wearable systems provided in this invention calculates the finger joint motion state quantity corresponding to the finger joint motion data and the wrist motion state quantity corresponding to the wrist motion data, which facilitates accurate acquisition of hand motion state and improves the accuracy of subsequent gesture recognition.

[0013] In conjunction with the second embodiment of the first aspect, in the fifth embodiment of the first aspect, determining the hand motion observations at the acquisition time based on the preset observation model and the image data includes: acquiring multiple finger joint key points and wrist key points corresponding to the target IMU data; calculating the coordinate positions of the finger joint key points and the wrist key points mapped to the image data; and determining the observations corresponding to the hand motion state quantity based on the preset observation model and the coordinate positions.

[0014] The gesture recognition method for wearable systems provided in this invention maps multiple finger joint key points and wrist key points into image data to determine the observations of finger joint movement and wrist movement relative to the image data. This facilitates updating the hand movement state based on the observations, resulting in more accurate hand movement information and facilitating subsequent gesture recognition.

[0015] In conjunction with the first aspect or any of the first to fifth embodiments of the first aspect, in the sixth embodiment of the first aspect, the method further includes: controlling the head-mounted device to perform different operations based on the different target gestures.

[0016] The gesture recognition method for wearable systems provided in this invention controls the head-mounted device to perform different operations based on different target gestures, thereby improving the gesture interaction experience between the user and the wearable device.

[0017] According to a second aspect, embodiments of the present invention provide a gesture recognition device for a wearable system, comprising: an acquisition module for acquiring image data and IMU data collected by the wearable system, wherein the wearable system includes a head-mounted device and a wrist-worn IMU device; a conversion module for converting the IMU data to a coordinate system in which the wrist is located, to obtain target IMU data in the coordinate system in which the wrist is located; a fusion module for fusing the target IMU data and the image data based on the correspondence between the image data and the target IMU data, to obtain hand pose information; and a determination module for determining the corresponding target gesture based on the hand pose information.

[0018] According to a third aspect, embodiments of the present invention provide a gesture recognition system for a wearable system, comprising: a wearable system including a head-mounted device and a wristband device; a memory and a processor, wherein the wearable system, the memory, and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the gesture recognition method of the wearable system according to the first aspect or any embodiment of the first aspect.

[0019] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the gesture recognition method of the wearable system described in the first aspect or any embodiment of the first aspect.

[0020] It should be noted that the beneficial effects of the gesture recognition device, gesture recognition system, and computer-readable storage medium of the wearable system provided in the embodiments of the present invention can be found in the description of the corresponding content in the gesture recognition method of the wearable system, and will not be repeated here. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 A schematic diagram of the wearable system in an embodiment of the present invention is shown;

[0023] Figure 2 This is a flowchart of a gesture recognition method for a wearable system according to an embodiment of the present invention;

[0024] Figure 3 This is another flowchart of a gesture recognition method for a wearable system according to an embodiment of the present invention;

[0025] Figure 4 This is yet another flowchart of a gesture recognition method for a wearable system according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of hand joints according to an embodiment of the present invention;

[0027] Figure 6 This is a structural block diagram of a gesture recognition device for a wearable system according to an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the hardware structure of the gesture recognition system of the wearable system provided in an embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] According to an embodiment of the present invention, an embodiment of a gesture recognition method for a wearable system is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] This embodiment provides a gesture recognition method for a wearable system, which can be used in a gesture recognition system for wearable systems. The gesture recognition system includes a wearable system and a processing device. The wearable system may include a head-mounted device and a wristband device. The processing device can be a mobile terminal such as a mobile phone or tablet computer, or an electronic device such as a desktop computer or laptop computer. Figure 2 This is a flowchart of a gesture recognition method for a wearable system according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0032] S11, acquire image data and IMU data collected by the wearable system, which includes a head-mounted device and a wrist-worn IMU device.

[0033] Head-mounted devices are AR / VR / MR devices with visual sensors, such as AR glasses; wrist-worn IMU devices are wristband devices or back-of-hand adhesive devices with IMUs, such as wristbands with IMUs. Taking wristband devices as an example, the head-mounted and wristband devices of a wearable system are communicatively connected. The head-mounted device is used to capture motion images of the hand, and the wristband device is used to capture IMU data generated during hand movements. Wrist and finger movement data are calculated from the IMU data. The wearable system can send the captured image and IMU data to the corresponding processing device for gesture determination. Correspondingly, the processing device of the wearable system can receive the image and IMU data captured by the wearable system.

[0034] S12, convert the IMU data to the coordinate system of the wrist to obtain the target IMU data in the coordinate system of the wrist.

[0035] Target IMU data is used to represent wrist motion data. IMU data has an IMU coordinate system, which is not consistent with the wrist coordinate system. After obtaining the IMU data, to ensure that it accurately represents the wrist motion state, the IMU data can be transformed to the wrist's coordinate system, resulting in target IMU data in that coordinate system. This target IMU data reflects the true wrist motion state. Since changes in wrist motion are crucial to changes in finger joint motion, accurately determining the true finger joint motion state through the true wrist motion is essential.

[0036] It should be noted that if the wristband IMU device is worn on the wrist, i.e., a wristband IMU device, the IMU data can be approximated as representing wrist motion data because the wristband IMU device is very close to the wrist. However, if the wristband IMU device is not worn on the wrist, for example, if the IMU device is attached to the back of the hand or other parts of the hand, the IMU data cannot be directly used to represent wrist motion data. In this case, it is necessary to further obtain the extrinsic parameters between the IMU device and the wrist to convert the IMU data to the coordinate system of the wrist and obtain target IMU data that can represent the wrist's operating state.

[0037] S13, based on the correspondence between image data and target IMU data, the target IMU data and image data are fused to obtain hand pose information.

[0038] Hand pose information is three-dimensional pose data used to characterize the hand during movement. Since the acquisition frequencies of image data and IMU data are different, the acquisition times of these two types of data will not completely overlap. Therefore, after receiving the image data and target IMU data, the processing device synchronizes these two types of data in time, determines the correspondence between the image data and target IMU data at the acquisition time, and fuses the target IMU data and image data at the same acquisition time according to this correspondence, obtaining fused data of the target IMU data and image data at the same acquisition time, i.e., hand pose information.

[0039] Specifically, the data output frequency of an IMU is typically much higher than that of a vision sensor. An IMU's data output frequency is approximately 200–1000 Hz, while a vision sensor's is typically 10–30 Hz. Since the IMU data output process is continuous, the IMU data conversion process is also continuous. When the processing device receives image data from the vision sensor, it can synchronize the target IMU data (after coordinate system transformation) with the image data and perform data fusion to update the data and obtain hand pose information that can represent more hand movement information.

[0040] S14, determine the corresponding target gesture based on hand pose information.

[0041] Target gestures are used to control head-mounted devices to perform corresponding operations. Taking VR glasses as an example, target gestures are used to control VR glasses to perform corresponding display operations, such as video switching and video playback. Specifically, the processing device can analyze hand posture information to determine the current position and movement of the wrist and the finger joints. Based on the wrist posture and finger joint posture, the target gesture issued by the user at the current moment can be determined.

[0042] The gesture recognition method for wearable systems provided in this embodiment converts IMU data to the coordinate system of the wrist when determining the target gesture, so that the IMU data can accurately represent the wrist movement of the hand. Furthermore, the hand movement is determined based on the wrist movement, and then the gesture recognition accuracy is improved by fusing the target IMU data and image data.

[0043] This embodiment provides a gesture recognition method for a wearable system, which can be used in a gesture recognition system for wearable systems. The gesture recognition system includes a wearable system and a processing device. The wearable system may include a head-mounted device and a wristband device. The processing device can be a mobile terminal such as a mobile phone or tablet computer, or an electronic device such as a desktop computer or laptop computer. Figure 3 This is a flowchart of a gesture recognition method for a wearable system according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0044] S21, acquire image data and IMU data collected by the wearable system, which includes a head-mounted device and a wrist-worn IMU device. For detailed explanations, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0045] S22, the IMU data is converted to the coordinate system of the wrist to obtain the target IMU data in the coordinate system of the wrist. For detailed explanation, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0046] S23, based on the correspondence between image data and target IMU data, the target IMU data and image data are fused to obtain hand pose information.

[0047] Specifically, step S23 above may include:

[0048] S231, obtain the acquisition time of the image data and determine the correspondence between the image data and the target IMU data at the acquisition time.

[0049] The acquisition time of image data is determined based on the frequency of image data output by the vision sensor. This acquisition time can be characterized by the timestamp of the acquired image data. By synchronizing the timestamp associated with the image data and the timestamp associated with the target IMU data, the correspondence between the image data and the target IMU data at the acquisition time can be determined, that is, the time correspondence between the image data and the target IMU data.

[0050] S232, based on the correspondence, performs data synchronization fusion of image data and target IMU data to obtain hand pose information at each acquisition time.

[0051] The image data consists of two-dimensional images of the user's hand movements captured by the head-mounted device. The processing device can map multiple finger joint motion data and wrist motion data onto the image data based on the correspondence between the image data and the target IMU data to obtain the hand motion observation data at the current acquisition time. By fusing the hand motion observation data with the wrist motion data and finger joint motion data corresponding to the target IMU data, the hand pose information at the current acquisition time can be obtained.

[0052] Specifically, step S232 above may include:

[0053] (1) Calculate the finger joint motion state at the time of acquisition and the wrist motion state at the time of acquisition. The finger joint motion state and the wrist motion state constitute the hand motion state.

[0054] Finger joints refer to the metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints of the five fingers, totaling 15 joints, such as... Figure 5 The 21 key hand points shown are: k0 represents the wrist-metacarpal joint (sometimes referred to as "wrist" in this scheme), k1, k5, k9, k... 13 k 17 These represent the metacarpophalangeal joints (MCPs) of the five fingers from the thumb to the little finger: k2, k6, k... 10 k 14 k 18 These represent the proximal interphalangeal joints (PIP) of the five fingers mentioned above, k3, k7, k... 11 k 15 k 19 These represent the distal interphalangeal joints (DIPs) of the five fingers mentioned above, k4, k8, k9, k10, k11, k12, k13, k14, k15, k16, k17, k18, k19, k19, k10 ... 12 k 16 k 20 These represent the five fingertips. The finger joints are k1, k2, k3, k5, k6, k7, k9, k... 10 k 11 k 13 k 14 k 15 k 17 k 18 k 19 .

[0055] This embodiment draws on robot kinematics to model the kinematics of the finger joints and the wrist joints (i.e., the wrist). The modeled hand has a total of 26 degrees of freedom (DoF) in three-dimensional space, specifically including: (1) The wrist joint is similar to the ball joint of a robot and has 6 DoF, including 3 DoF representing position and 3 DoF representing direction; (2) When the state quantity of the finger joint is defined relative to the wrist coordinate system, the state quantity only has rotation and no translation. That is, it is generally believed that the metacarpophalangeal joint has 2 DoF, including flexion / extension and adduction / abduction; (3) The proximal and distal finger joints only have 1 DoF of flexion / extension.

[0056] The motion state of the finger joints at the same moment X F and wrist motion state quantity X r The quantity representing the hand's motion state X = [X r X F ].

[0057] Wherein, IMU state variable X IMU for:

[0058] X IMU =[ G p IMU G v IMU G q IMU b a b g ]

[0059] in, G p IMU , G v IMU , G q IMU These represent the position, velocity, and rotation of the IMU coordinate system relative to the world coordinate system, respectively. In this technology, the camera coordinate system is set as the world coordinate system. The rotation can be expressed using rotation matrices, Euler angles, or quaternions, and these can be converted to each other pairwise. In this embodiment, for the sake of compactness and to avoid gimbal lock, quaternions are preferably used to express the rotation. a and b g These represent the offsets of the accelerometer and gyroscope, respectively.

[0060] Next, calculate the wrist motion state quantity X. r Assuming the relative pose of the IMU and the wrist is known. IMU T r That is, the external parameter calibration of both has been completed. After the calculation of the IMU state variables is completed, the wrist state variables can be obtained. The wrist motion state variables are represented as X. r =[c T r c v r ],in c T r Including translation c p r and rotation amount c q r Two parts, translation part c p r = c p IMU · IMU T r The amount of rotation and speed follow the same principle.

[0061] In the motion parameters of the hand, the motion parameter X of the finger joints is... F Composed of state quantities for each finger and each joint, represented as:

[0062] X F =[X F1 X F2 X F3 X F4 X F5 ]

[0063] Each X Fi (i = 1, ..., 5) represents the rotation of the i-th finger, including the metacarpophalangeal joint, proximal interphalangeal joint, and distal interphalangeal joint, for a total of 4 degrees of freedom. Here, quaternions can be used to represent the finger joint state quantities, which can be converted to and from rotation matrices.

[0064] Specifically, the steps for calculating the finger joint motion state at the time of data acquisition may include:

[0065] 1) Construct a motion prediction model for finger joints based on their motion state.

[0066] Based on the continuous movement of finger joints over time, a continuous-time stochastic differential equation for the finger joints is constructed. This differential equation is used to model the movement of the finger joints in order to predict their movement state at various times.

[0067] Specifically, suppose X Fi Assuming the motion of the finger follows Gaussian processes and that the motion in the time domain conforms to a constant-velocity model, the state transition equation for the finger joint has the following form:

[0068]

[0069]

[0070]

[0071]

[0072] in, Let ω represent the state variables of all joints on the i-th finger. In this embodiment, it is specifically set as a rotation variable, representing the rotation of the finger joint. k Represents angular velocity, w k For noise, Q C δ represents the power spectral density matrix, and δ() represents the Dirac delta function.

[0073] 2) Discretize and integrate the motion prediction model, and calculate the finger joint motion state at the acquisition time based on the extended Kalman filter method.

[0074] For a given t k The time and its corresponding image data can prove that at time t k Within a sufficiently small area around the time Since it is linear, the continuous-time stochastic differential equation can be locally discretized, and then numerical integration methods, such as Euler integrals and Runge-Kutta integrals, can be used within the framework of the extended Kalman filter to calculate... The mean and variance are denoted as . and Assume the timestamps of the camera data are t1, t2, ..., t k , ..., t C Based on the above calculation method, the corresponding timestamp can be obtained. The mean and variance of these variables can be used to determine the motion state quantity X of the finger joints. F .

[0075] Specifically, the steps for calculating the wrist motion state at the time of data acquisition may include:

[0076] 1) Obtain a wrist motion model.

[0077] The wrist motion model is a continuous-time IMU system model, which is constructed based on the IMU kinematic model. This wrist motion model can predict the real-time motion state of the wrist.

[0078] 2) Discretize and integrate the wrist motion model, and calculate the wrist motion state variables at the acquisition time based on the extended Kalman filter method.

[0079] By discretizing the continuous-time IMU system model and employing integration or pre-integration methods within the framework of extended Kalman filtering, t can be calculated. k Wrist motion state quantity X at time moment IMU mean and the variance matrix

[0080] Specifically, assuming Follow the mean And the covariance is If the data follows a multidimensional Gaussian distribution, then the control quantity u in the state transition equation corresponding to the wrist motion data is the linear acceleration and angular velocity of the IMU, and the true value is the sensor reading. Subtracting the deviation and error, the control quantity u is expressed as follows:

[0081]

[0082] The state transition equation can then be expressed as follows:

[0083]

[0084]

[0085]

[0086]

[0087]

[0088] Given two times t k-1 and t k Given the position, velocity, and orientation state variables in the above state transition equation, and their corresponding image data, the IMU measurements can be obtained over a time interval [t]. k-1 , t k The integration process is performed; the specific integration method is known to those skilled in the art and will not be elaborated here. After the IMU data is output, the processing device can calculate t. k Mean of IMU state variables at time t Covariance Matrix Assume the timestamps of the camera data are t1, t2, ..., t k , ..., t C Based on the above calculation method, the corresponding timestamp can be obtained. The mean and variance of the IMU state variables are calculated, and then, using the method described above, the IMU state variables are transformed into wrist motion state variables. From this, the wrist motion state variable X can be determined. r .

[0089] By calculating the finger joint motion state corresponding to the finger joint motion data, and the wrist motion state corresponding to the wrist motion data, it is easier to accurately obtain the hand motion state and improve the accuracy of subsequent gesture recognition.

[0090] (2) Based on the preset observation model and image data, determine the hand movement observation at the acquisition time.

[0091] The hand motion observation is the coordinate value of the hand motion data in the image data. The prediction observation model is a pre-established observation model of the hand motion. Through this observation model, the hand motion data can be mapped to the image data collected by the vision sensor, thereby obtaining the hand motion observation at the current moment.

[0092] Specifically, when the vision sensor is a monocular vision sensor, the hand motion observation z is a two-dimensional hand key point; when the vision sensor is a binocular vision sensor, the hand motion observation z can be the coordinates of a three-dimensional hand key point obtained through binocular reconstruction; of course, other sensors can also be used, and no specific limitation is made here.

[0093] Specifically, step (2) above may include:

[0094] (21) Obtain multiple finger joint key points and wrist key points corresponding to the target IMU data.

[0095] The key points of the hand are the key points of the finger joints and the key points of the wrist, such as... Figure 5 The 21 hand key points shown are described in detail in the corresponding descriptions of the above embodiments, and will not be repeated here. Hand movement changes are determined by the positional changes of these key points. When acquiring target IMU data, the processing device can analyze the target IMU data to determine the coordinate positions of the wrist key points corresponding to the target IMU data.

[0096] The three-dimensional coordinates of the remaining 20 finger joint keypoints can be obtained through binocular triangulation. Specifically, two or more monocular cameras are used to capture images of the hand, and the captured two-dimensional images are sent to a processing device. The processing device can extract the two-dimensional coordinates of the finger joint keypoints from the captured two-dimensional images, and then use the binocular triangulation method to reconstruct the three-dimensional coordinate system corresponding to the finger joint keypoints, thereby determining the three-dimensional coordinate positions of the finger joint keypoints. Alternatively, during gesture tracking, the three-dimensional coordinate positions of the finger joint keypoints can be determined directly from the data of the previous frame.

[0097] The finger joint key points can also be calculated based on the finger joint key points extracted from the previous frame of the two-dimensional image as initial values, and combined with the gesture tracking process, to calculate the position of the finger joint key points in the current frame of the two-dimensional image, and then map them to the three-dimensional coordinate system corresponding to the finger joint key points to determine the three-dimensional coordinate position of the current finger joint key points.

[0098] (22) Calculate the coordinates of the finger joint key points and wrist key points mapped to the image data.

[0099] The coordinates of each hand keypoint (i.e., finger joint keypoints and wrist keypoints) in the image data are calculated separately. Taking a monocular vision sensor as an example, the processing device can calculate the pixel coordinates of each hand keypoint in the two-dimensional image, denoted as z. j Typically, head-mounted devices are equipped with two or more visual sensor cameras, and there are multiple key points on the hand. j represents the pixel coordinates of all the i-th key points on the hand observed by the j-th visual sensor.

[0100] (23) Based on the preset observation model and coordinate position, determine the observation quantity corresponding to the hand motion state quantity.

[0101] The preset observation model is:

[0102] z j =π j (trans(kin(X k ))

[0103] Where, π j (·) represents the projection process of the j-th visual sensor, which is to project the three-dimensional coordinates of the hand key points in the visual sensor coordinate system onto the key points in the two-dimensional image. trans(·) represents taking the pose matrix T i The translation component represents the 3D position of the i-th hand keypoint. `kin(·)` represents the robot's kinematic process (different kinematic equations for different finger joints, but the same kinematic equation for the same finger joint at different positions). Here, forward kinematics is used to model the hand keypoints to represent the pose transformation relationship of the finger bones. The following example uses the fingertip of one finger; other joints are analogous:

[0104] The coordinates of the fingertip in the visual sensor coordinate system are:

[0105] c p tip = c T r · r T mcp · mcp T pip ·pip T dip · dip P tip

[0106] The pose matrix describing the motion of a three-dimensional rigid body is typically represented as: R represents the rotation matrix, and p is the three-dimensional coordinate of the rigid body relative to the origin of the selected coordinate system, also known as translation or position (corresponding to the position p mentioned above). The rotation matrix and the quaternion q mentioned above can be converted to each other, for example in... pip T dip In the middle, R( mcp q pip ) indicates that it corresponds to X F The quaternion representing the rotation amount mcp q pip The rotation matrix; for example, c T r The translation part in the text corresponds to the above. G p IMU Here, the visual sensor coordinate system is selected as the global coordinate system, that is... c p IMU The rotation matrix can be written as R( c q IMU ).

[0107] r T mcp This indicates the relative pose of the MCP with respect to the wrist (corresponding to...). Figure 5 (MCP of metacarpophalangeal joint 9, pose relative to wrist coordinate system);

[0108] mcp T pip This indicates the relative pose of the PIP with respect to the MCP.

[0109] pip T dip This indicates the relative pose of the DIP with respect to the PIP.

[0110] dip p tip fingertips (corresponding to) Figure 5 The initial value of the three-dimensional coordinates of the distal interphalangeal joint (reference number 12) relative to the distal interphalangeal joint can be calculated by triangulation using two monocular vision sensors.

[0111] Next, using the matrix partial derivative method, we can calculate the hand motion observation z of the vision sensor relative to the hand motion state quantity X = [X]. r X F The Jacobian matrix of H is denoted as H. k .

[0112] By mapping hand motion data to image data through multiple hand key points, the observations of hand motion data relative to image data are determined. This allows for updating of hand motion state quantities based on these observations, resulting in more accurate hand motion information and facilitating subsequent gesture recognition.

[0113] (3) The hand motion observation and hand motion state quantity at the same acquisition time are fused to obtain the target motion state quantity of the hand at the acquisition time.

[0114] The target motion state variable is the hand motion state variable at the current moment. Specifically, the mean of the target motion state variable updated with hand motion observations can be calculated using the extended Kalman filter method. and its covariance matrix The mean and its covariance matrix are used to characterize the hand movement state at the current moment.

[0115] (4) Determine hand pose information based on target motion state variables.

[0116] The target motion state variables can characterize the three-dimensional position, velocity, rotation, offset of the wrist, and the rotation of the finger joints relative to the wrist. By analyzing these target motion state variables, the processing device can determine the hand pose information at the current moment.

[0117] S24, determine the corresponding target gesture based on the hand pose information. For detailed explanation, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0118] The gesture recognition method for wearable systems provided in this embodiment acquires image data at a lower frequency than IMU data, meaning the output frequency of image data is lower than the output frequency of the target IMU data. By determining the correspondence between image data and target IMU data at the acquisition time, the image data and target IMU data are then fused according to this correspondence to obtain rich hand motion information. By fusing hand motion state quantities and hand motion observations, the accuracy of hand motion state recognition is improved, and the robustness of gesture interaction is enhanced.

[0119] This embodiment provides a gesture recognition method for a wearable system, which can be used in a gesture recognition system for wearable systems. The gesture recognition system includes a wearable system and a processing device. The wearable system may include a head-mounted device and a wristband device. The processing device can be a mobile terminal such as a mobile phone or tablet computer, or an electronic device such as a desktop computer or laptop computer. Figure 4 This is a flowchart of a gesture recognition method for a wearable system according to an embodiment of the present invention, such as... Figure 4 As shown, the process includes the following steps:

[0120] S31, acquire image data and IMU data collected by the wearable system, which includes a head-mounted device and a wrist-worn IMU device. For detailed explanations, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0121] S32, the IMU data is converted to the coordinate system of the wrist to obtain the target IMU data in the coordinate system of the wrist. For detailed explanation, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0122] S33, based on the correspondence between image data and target IMU data, the target IMU data and image data are fused to obtain hand pose information. For detailed explanation, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0123] S34, determine the corresponding target gesture based on the hand pose information. For detailed explanation, please refer to the relevant descriptions in the above embodiments; they will not be repeated here.

[0124] S35 controls the head-mounted device to perform different operations based on different target gestures.

[0125] Different target gestures correspond to different operations. The correspondence between target gestures and their set operations is pre-stored in the memory of the wearable system. After the processor of the wearable system determines the current target gesture, it can send corresponding control commands to the head-mounted device according to the set operation corresponding to the target gesture, so as to control the head-mounted device to perform different operations according to the different target gestures.

[0126] The gesture recognition method for wearable systems provided in this embodiment controls the head-mounted device to perform different operations based on different target gestures, thereby improving the gesture interaction experience between the user and the wearable device.

[0127] This embodiment also provides a gesture recognition device for a wearable system, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0128] This embodiment provides a gesture recognition device for a wearable system, such as... Figure 6 As shown, it includes:

[0129] The acquisition module 41 is used to acquire image data and IMU data collected by the wearable system, which includes a head-mounted device and a wrist-worn IMU device. Detailed explanations are available in the descriptions corresponding to the above method embodiments, and will not be repeated here.

[0130] The conversion module 42 is used to convert the IMU data to the coordinate system of the wrist, thereby obtaining the target IMU data in the coordinate system of the wrist. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0131] The fusion module 43 is used to fuse the target IMU data and the image data based on the correspondence between the image data and the target IMU data to obtain hand pose information. Detailed explanations are available in the descriptions corresponding to the above method embodiments, and will not be repeated here.

[0132] The determination module 44 is used to determine the corresponding target gesture based on hand pose information. Detailed descriptions of the methods described in the above embodiments are provided and will not be repeated here.

[0133] The gesture recognition device of the wearable system provided in this embodiment converts the IMU data to the coordinate system of the wrist when determining the target gesture, so that the IMU data can accurately represent the hand movement. Then, the target IMU data and image data are fused to improve the recognition accuracy and precision of the gesture.

[0134] In this embodiment, the gesture recognition device of the wearable system is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0135] Further functional descriptions of the above modules are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0136] This invention also provides a gesture recognition system for a wearable system, having the above-described features. Figure 6 The wearable system shown has a gesture recognition device.

[0137] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a gesture recognition system for a wearable system provided in an optional embodiment of the present invention, as shown below. Figure 7As shown, the gesture recognition system of this wearable system may include: a wearable system 605, at least one processor 601, such as a CPU (Central Processing Unit), at least one communication interface 603, a memory 604, and at least one communication bus 602. The wearable system 605 includes a head-mounted device, such as MR / AR / VR glasses; and a wristband device, such as a bracelet. The communication bus 602 is used to enable communication between these components. The communication interface 603 may include a display screen and a keyboard; optionally, the communication interface 603 may also include a standard wired interface or a wireless interface. The memory 604 may be high-speed RAM (Random Access Memory) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 604 may also be at least one storage device located remotely from the aforementioned processor 601. The processor 601 may be combined with... Figure 6 The described apparatus has an application program stored in memory 604, and a processor 601 calls the program code stored in memory 604 to perform any of the above method steps.

[0138] The communication bus 602 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 602 can be divided into an address bus, a data bus, and a control bus, etc. For ease of representation, Figure 7 It is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0139] The memory 604 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 604 may also include a combination of the above types of memory.

[0140] The processor 601 can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP.

[0141] The processor 601 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0142] Optionally, memory 604 is also used to store program instructions. Processor 601 can call the program instructions to implement the functions described in this application. Figures 2 to 4 The gesture recognition method of the wearable system shown in the embodiment.

[0143] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the processing method of the gesture recognition method of the wearable system in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0144] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A gesture recognition method for a wearable system, characterized in that, include: The system acquires image data and IMU data collected by a wearable system, which includes a head-mounted device and a wrist-worn IMU device. The IMU data is converted to the coordinate system of the wrist to obtain the target IMU data in the coordinate system of the wrist. Based on the correspondence between the image data and the target IMU data, the target IMU data and the image data are fused to obtain hand pose information, including: acquiring the acquisition time of the image data and determining the correspondence between the image data and the target IMU data at the acquisition time; performing data synchronization fusion of the image data and the target IMU data based on the correspondence to obtain hand pose information at each acquisition time, including: calculating the finger joint motion state quantity of finger joint motion data at the acquisition time and the wrist motion state quantity of wrist motion data at the acquisition time, wherein the finger joint motion state quantity and the wrist motion state quantity constitute the hand motion state quantity; determining the hand motion observation quantity at the acquisition time based on a preset observation model and the image data; fusing the hand motion observation quantity and the hand motion state quantity at the same acquisition time to obtain the target motion state quantity of the hand at the acquisition time; and determining the hand pose information based on the target motion state quantity. The corresponding target gesture is determined based on the hand position information.

2. The method according to claim 1, characterized in that, The calculation of finger joint motion data at the time of acquisition includes: Based on the motion state of the finger joints, a motion prediction model for the finger joints is constructed; The motion prediction model is discretized and integrated, and the motion state of the finger joint at the acquisition time is calculated based on the extended Kalman filter method.

3. The method according to claim 1, characterized in that, Calculate the wrist motion state quantities at the time of acquisition, including: Obtain a wrist motion model; The wrist motion model is discretized and integrated, and the wrist motion state variables at the acquisition time are calculated based on the extended Kalman filter method.

4. The method according to claim 1, characterized in that, The determination of the hand movement observation at the acquisition time based on the preset observation model and the image data includes: Acquire multiple finger joint key points and wrist key points corresponding to the target IMU data; Calculate the coordinate positions of the key points of the finger joints and the key points of the wrist in the image data; Based on the preset observation model and the coordinate position, the observation quantity corresponding to the hand movement state quantity is determined.

5. The method according to any one of claims 1-4, characterized in that, Also includes: Based on the different target gestures, the head-mounted device is controlled to perform different operations.

6. A gesture recognition device for a wearable system, characterized in that, include: The acquisition module is used to acquire image data and IMU data collected by the wearable system, wherein the wearable system includes a head-mounted device and a wrist-worn IMU device; The conversion module is used to convert the IMU data to the coordinate system of the wrist to obtain the target IMU data in the coordinate system of the wrist. A fusion module is used to fuse the target IMU data and the image data based on the correspondence between the image data and the target IMU data to obtain hand pose information. This includes: acquiring the acquisition time of the image data; determining the correspondence between the image data and the target IMU data at the acquisition time; performing synchronous data fusion of the image data and the target IMU data based on the correspondence to obtain hand pose information at each acquisition time; calculating the finger joint motion state quantity of finger joint motion data and the wrist motion state quantity of wrist motion data at the acquisition time, wherein the finger joint motion state quantity and the wrist motion state quantity constitute the hand motion state quantity; determining the hand motion observation quantity at the acquisition time based on a preset observation model and the image data; fusing the hand motion observation quantity and the hand motion state quantity at the same acquisition time to obtain the target motion state quantity of the hand at the acquisition time; and determining the hand pose information based on the target motion state quantity. The determination module is used to determine the corresponding target gesture based on the hand position information.

7. A gesture recognition system for a wearable system, characterized in that, include: A wearable system, comprising a head-mounted device and a wristband device; The wearable system, the memory, and the processor are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the gesture recognition method of the wearable system according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the gesture recognition method of the wearable system according to any one of claims 1-5.

Images