Method and system for detecting gestures, and computer readable storage medium

Abstract

Embodiments of the present invention provide a method and system for detecting a gesture, and a computer readable storage medium. The method includes determining whether information of a hand is sufficient for recognizing a gesture of the hand, in response to determining that the information of the hand is sufficient for recognizing the gesture, recognizing the gesture, receiving first gesture information from at least one external posture information provider, and correcting the gesture based on the first gesture information, in response to determining that the information of the hand is not sufficient for recognizing the gesture, receiving second gesture information from the at least one external posture information provider, obtaining a predicted gesture, and obtaining the gesture based on the predicted gesture and the second gesture information.

Description

Technical Field

[0001] This invention generally relates to a tracking mechanism, and more specifically, to a method and system for detecting gestures, as well as a computer-readable storage medium. Background Technology

[0002] See Figure 1 It illustrates gesture tracking. Figure 1 In the left portion, a head-mounted display (HMD) 100 may be equipped with a tracking element (e.g., a tracking camera on the HMD 100), and the tracking element may have a field of view (FOV) 199. When the object to be tracked (e.g., hand 102) is within the FOV 199, the HMD 100 can acquire an image of the object to be tracked, thus enabling the tracking of the object's posture (e.g., the gesture of hand 102).

[0003] However, when the object to be tracked leaves FOV 199, as illustrated in the example... Figure 1 On the right side, the HMD100 will be unable to track the pose of the object being tracked because it cannot acquire an image of the object. Summary of the Invention

[0004] Therefore, the present invention relates to a method and system for detecting gestures, and a computer-readable storage medium, which can be used to solve the above-mentioned technical problems.

[0005] An embodiment of the present invention provides a method for detecting gestures applicable to a host computer, comprising: determining whether hand information is sufficient for recognizing hand gestures; in response to determining that hand information is sufficient for recognizing gestures, recognizing gestures, receiving first gesture information from at least one external gesture information provider, and correcting gestures based on the first gesture information; in response to determining that hand information is insufficient for recognizing gestures, receiving second gesture information from at least one external gesture information provider, obtaining a predicted gesture, and obtaining a gesture based on the predicted gesture and the second gesture information.

[0006] Embodiments of the present invention provide a system for detecting gestures. The system includes a host. The host is configured to: determine whether hand information is sufficient for recognizing a hand gesture; in response to determining that the hand information is sufficient for recognizing a gesture, recognize the gesture; receive first gesture information from at least one external gesture information provider, and correct the gesture based on the first gesture information; in response to determining that the hand information is insufficient for recognizing a gesture, receive second gesture information from at least one external gesture information provider, obtain a predicted gesture, and obtain the gesture based on the predicted gesture and the second gesture information.

[0007] Embodiments of the present invention provide a non-transitory computer-readable storage medium recording an executable computer program, which is loaded by a host to perform the following steps: determining whether hand information is sufficient for recognizing a hand gesture; in response to determining that the hand information is sufficient for recognizing a gesture, recognizing the gesture, receiving first gesture information from at least one external gesture information provider, and correcting the gesture based on the first gesture information; in response to determining that the hand information is insufficient for recognizing a gesture, receiving second gesture information from at least one external gesture information provider, obtaining a predicted gesture, and obtaining the gesture based on the predicted gesture and the second gesture information. Attached Figure Description

[0008] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0009] Figure 1 A schematic diagram illustrating gesture tracking.

[0010] Figure 2A This is a functional diagram of a system for detecting gestures according to an embodiment of the present invention.

[0011] Figure 2B for Figure 2A A schematic diagram.

[0012] Figure 3 A flowchart illustrating a method for detecting gestures according to an embodiment of the present invention is shown.

[0013] Explanation of icon numbers

[0014] 100:HMD

[0015] 102, O1: Hands

[0016] 199, 299: Field of vision

[0017] 200: System

[0018] 21: Host

[0019] 212: Storage circuit

[0020] 214: Processor

[0021] 221~22N: External posture information provider

[0022] CS1: First control signal

[0023] CS2: Second control signal

[0024] P1~PN: First gesture information

[0025] C1~CN: Second gesture information

[0026] G1: Predicting Gestures

[0027] G2: Previous gesture

[0028] S310, S321~S323, S331~S333: Steps Detailed Implementation

[0029] Referring now to preferred embodiments of the invention, examples of which are illustrated in the accompanying drawings. Where possible, the same reference numerals are used in the drawings and description to refer to the same or similar parts.

[0030] See Figure 2A and Figure 2B ,in Figure 2A This is a functional diagram of a system for detecting gestures according to an embodiment of the present invention, and Figure 2B for Figure 2A A schematic diagram.

[0031] exist Figure 2A In the system 200, there are host 21 and external posture information providers 221 to 22N, wherein host 21 is connected to each of the external posture information providers 221 to 22N via a wireless communication protocol such as Bluetooth.

[0032] In various embodiments, the host 21 can be any device capable of tracking one or more objects to be tracked (e.g., hand O1) within the FOV 299 (e.g., inside-out tracking and / or outside-in tracking). In embodiments of the invention, the FOV 299 can be the image capture range of one or more cameras (e.g., tracking cameras) on the host 21. When the object to be tracked (e.g., hand O1) is within the FOV 299, the cameras on the host 21 can capture images of the object to be tracked, and the host 21 can track the pose of each object to be tracked based on the captured images, but the invention is not limited thereto.

[0033] In embodiments of the present invention, the host 21 may be an HMD for providing reality services to its users, wherein reality services include, but are not limited to, virtual reality (VR) services, augmented reality (AR) services, extended reality (XR) services, and / or mixed reality (MR) services. In these cases, the host 21 may display corresponding visual content for the user to view, such as VR / AR / XR / MR visual content.

[0034] exist Figure 2A In this configuration, host 21 includes storage circuitry 212 and processor 214. Storage circuitry 212 is one or a combination of static or mobile random access memory (RAM), read-only memory (ROM), flash memory, hard disk, or any other similar device, and storage circuitry 212 records multiple modules executable by processor 214.

[0035] Processor 214 may be coupled to storage circuitry 212, and processor 214 may be, for example, a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like.

[0036] In some embodiments, each of the external posture information providers 221 to 22N may be an (external) camera capable of tracking one or more objects to be tracked, such as a tracking camera outside the host 21. The external posture information providers 221 to 22N may be located in the environment where the host 21 is located or in an environment in which the host 21 can move.

[0037] In an embodiment of the invention, hand O1 is used as an instance of the object to be tracked, and each external pose information provider 221 to external pose information provider 22N can track the pose (i.e., gesture) of hand O1. In one embodiment, hand O1 is pre-constructed from a specific number (e.g., 21) of predetermined joints, and each external pose information provider 221 to external pose information provider 22N can be used to track the pose (e.g., 3D position) of each predetermined joint.

[0038] In one embodiment, host 21 may send a first control signal CS1 to each external posture information provider 221 to external posture information provider 22N to request each external posture information provider 221 to external posture information provider 22N to provide the posture of some specific joints of predetermined joints on hand O1. For example, host 21 may request each external posture information provider 221 to external posture information provider 22N to provide the host 21 only the posture of the fingertip joints on hand O1. In an embodiment, the posture information provided by each external posture information provider 221 to external posture information provider 22N in response to the first control signal CS1 may be referred to as first gesture information. In one embodiment, the first gesture information may be partial gesture information, which indicates that each external posture information provider 221 to external posture information provider 22N does not provide the tracked posture of all predetermined joints on hand O1.

[0039] For example, an external gesture information provider 221 may provide the host 21 with first gesture information P1, wherein the first gesture information P1 may exemplify the posture of the fingertip joints on the hand O1 tracked by the external gesture information provider 221. As another example, an external gesture information provider 222 may provide the host 21 with first gesture information P2, wherein the first gesture information P2 may exemplify the posture of the fingertip joints on the hand O1 tracked by the external gesture information provider 222. Similarly, an external gesture information provider 22N may provide the host 21 with first gesture information PN, wherein the first gesture information PN may exemplify the posture of the fingertip joints on the hand O1 tracked by the external gesture information provider 22N.

[0040] In one embodiment, the host 21 may send a second control signal CS2 to each of the external posture information providers 221 to 22N to request each of the external posture information providers 221 to 22N to provide the tracked postures of all predetermined joints on the hand O1. In this embodiment, the posture information provided by each of the external posture information providers 221 to 22N in response to the second control signal CS2 may be referred to as second gesture information. In one embodiment, the second gesture information may be complete gesture information, which represents the tracked postures of all predetermined joints on the hand O1 provided by each of the external posture information providers 221 to 22N.

[0041] For example, external gesture information provider 221 may provide host 21 with second gesture information C1, wherein first gesture information P1 may exemplify the poses of all predetermined joints on hand O1 tracked by external gesture information provider 221. As another example, external gesture information provider 222 may provide host 21 with first gesture information P2, wherein first gesture information P2 may exemplify the poses of all predetermined joints on hand O1 tracked by external gesture information provider 222. Similarly, external gesture information provider 22N may provide host 21 with first gesture information PN, wherein first gesture information PN may exemplify the poses of all predetermined joints on hand O1 tracked by external gesture information provider 22N.

[0042] In an embodiment of the invention, processor 214 may access modules stored in storage circuit 212 to implement a method for detecting gestures provided in the invention, which will be further discussed below.

[0043] join Figure 3 The diagram illustrates a flowchart of a method for detecting gestures according to an embodiment of the present invention. This method of the embodiment can be derived from... Figure 2A and Figure 2B The host 21 in the middle executes, and will be used in the following text. Figure 2A and Figure 2B The components shown in the figure are used to describe Figure 3 Details of each step in the process.

[0044] First, in step S310, the processor 214 determines whether the information of the hand O1 is sufficient to recognize the gesture of the hand O1.

[0045] In one embodiment, processor 214 detects joints in a gesture and determines whether the number of detected joints in the gesture exceeds a number threshold. In different embodiments, the number threshold may be determined by the designer as a sufficient number for processor 214 to track the pose of hand O1.

[0046] In one embodiment, in response to determining that the number of joints on the detected gesture is higher than a number threshold, the processor 214 determines that the information of the hand O1 is sufficient to recognize the gesture. In this case, even if some joints are untrackable for some reason (e.g., outside the FOV 299 of the host 21 or obstructed when the hand O1 is at certain angles), the processor 214 can predict / guess the pose of the trackable joints based on the trackable joints, but the invention is not limited thereto.

[0047] On the other hand, in response to determining that the number of joints on the detected gesture is not higher than the number threshold, the processor 214 determines that the information of the hand O1 is insufficient for recognizing the gesture.

[0048] In one embodiment, processor 214 may perform inside-out tracking to detect joints in a gesture based on an image of the hand O1 captured by a tracking camera, but the invention is not limited thereto.

[0049] In another embodiment, the processor 214 may determine whether the hand O1 is within the field of view 299 of the host 21's camera (e.g., a tracking camera). In one embodiment, in response to determining that the hand O1 is within the field of view 299 of the host 21's camera, the processor 214 determines that the information of the hand O1 is sufficient for recognizing a gesture. On the other hand, in response to determining that the hand O1 is not within the field of view 299 of the host 21's camera, the processor 214 determines that the information of the hand O1 is insufficient for recognizing a gesture.

[0050] In a first embodiment where it is determined that the information of hand O1 is sufficient to recognize the gesture of hand O1 (e.g., hand O1 is within FOV 299), processor 214 may then perform steps S321 to S323 to obtain the gesture of hand O1.

[0051] In step S321, processor 214 recognizes the gesture. In one embodiment, processor 214 may track the gesture based, for example, on a conventional inside-out tracking mechanism, which will not be discussed in detail below.

[0052] Therefore, after step S321, the hand gesture of O1 has been obtained (e.g., 6 degrees of freedom (6DOF)).

[0053] In step S322, processor 214 receives first gesture information P1 to first gesture information PN from external gesture information providers 221 to 22N. As mentioned above, each first gesture information P1 to first gesture information PN may only contain the posture of some specific joints on hand O1 (e.g., fingertip joints on hand O1). For better understanding, the posture of the specific joints carried in the first gesture information P1 to first gesture information PN will be referred to as the first posture.

[0054] In step S323, processor 214 corrects the gesture based on first gesture information P1 to first gesture information PN. In one embodiment, processor 214 obtains the posture of each specific joint in the gesture detected by host 21 in step S321. For example, if a specific joint is considered to be a fingertip joint on hand O1, then processor 214 can obtain the posture of the fingertip joint from the gesture detected in step S321, but the invention is not limited thereto. For better understanding, the posture of the specific joint obtained from the gesture detected in step S321 will be referred to as the second posture.

[0055] Next, the processor 214 can correct the gesture obtained in step S321 by combining the second pose of each specific joint with the corresponding first pose.

[0056] In simple terms, since the processor 214 can obtain the first posture of a specific joint based on the first gesture information P1 to the first gesture information PN, the processor 214 can optimize the second posture of the specific joint on the gesture obtained in step S321 based on the first posture of the specific joint.

[0057] For example, assuming the thumb's fingertip joint on hand O1 is one of the specific joints under consideration, processor 214 can obtain a second pose of the thumb's fingertip joint from the gesture obtained in step S321. Additionally, processor 214 can obtain a first pose of the thumb's fingertip joint from each of the first gesture information P1 to the first gesture information PN. Then, processor 214 can combine the second pose of the thumb's fingertip joint carried in each of the first gesture information P1 to the first gesture information PN with the first pose of the thumb's fingertip joint to form a corrected pose of the thumb's fingertip joint. In one embodiment, when combining the first pose and the second pose, processor 214 can obtain a linear or nonlinear combination of the first pose and the second pose, but the invention is not limited thereto.

[0058] After performing similar operations on other specific joints, the corrected postures of those other specific joints can be obtained. Therefore, the gestures obtained in step S321 can be more accurate because the postures of the specific joints on the hand O1 have already been corrected in step S323.

[0059] Specifically, when the information of hand O1 is sufficient to recognize the gesture of hand O1 (e.g., hand O1 is in FOV 299), processor 214 may need to display the gesture at a higher frame rate to provide a better visual experience for the user. In this case, processor 214 may not have sufficient resources to receive second gesture information C1 to second gesture information CN from external gesture information providers 221 to 22N and / or to perform more complete correction on the gesture obtained in step S321.

[0060] Therefore, when the information of hand O1 is sufficient to recognize the gesture of hand O1, processor 214 can request external posture information providers 221 to 22N to provide only the first gesture information P1 to PN, and correct the posture of some specific joints (e.g., some important joints and / or some joints that are more likely to be occluded). In this case, processor 214 can provide better gesture tracking results without consuming too many additional resources.

[0061] In a second embodiment where the information of hand O1 is determined to be insufficient for recognizing the gesture of hand O1 (e.g., hand O1 is outside FOV 299), processor 214 may continue to steps S331 to S333 to obtain the gesture of hand O1.

[0062] In the second embodiment, it is assumed that hand O1 is within FOV 299 and leaves FOV 299 at time point i, such as Figure 2B As shown, i is the index value. In this case, before the i-th time point, the processor 214 can determine the gesture based on steps S321 to S323.

[0063] At time point i, since hand O1 has left FOV 299, processor 214 can determine that the information of hand O1 is insufficient to recognize the gesture of hand O1 and continue to steps S331 to S333.

[0064] In step S331, the processor 214 receives second gesture information C1 to second gesture information CN from external gesture information provider 221 to external gesture information provider 22N.

[0065] In one embodiment, in response to the determination that the information of hand O1 changes from sufficient to recognize a gesture to insufficient to recognize a gesture, processor 214 may send a second control signal CS2 to each external gesture information provider 221 to external gesture information provider 22N to request the external gesture information providers 221 to external gesture information provider 22N to provide the corresponding second gesture information C1 to second gesture information, but the present invention is not limited thereto.

[0066] As mentioned above, each of the second gesture information C1 to the second gesture information CN contains the poses of all predetermined joints on the hand O1. For better understanding, the poses of all predetermined joints carried in the second gesture information C1 to the second gesture information CN will be referred to as the third pose.

[0067] In step S332, the processor 214 obtains the predicted gesture G1. In the second embodiment, in response to the determination that the information of hand O1 has changed from sufficient for gesture recognition to insufficient for gesture recognition at time point i, the processor 214 may obtain the previous gesture G2 recognized at time point (ik), where k is a positive integer. Then, the processor 214 may predict the gesture at time point i as the predicted gesture G1 based on the previous gesture.

[0068] In different embodiments, k can be any value desired by the designer. To better predict the predicted gesture G1, k can be determined to be 1, such that the processor 214 acquires the recognized gesture as the previous gesture G2 just before the hand O1 leaves the FOV 299, but the invention is not limited thereto.

[0069] In the second embodiment, the mechanism for predicting the gesture G1 based on the previous gesture G2 can be referred to in relevant gesture prediction techniques, which will not be discussed further herein.

[0070] In step S333, the processor 214 obtains the gesture based on the predicted gesture G1 and the second gesture information C1 to the second gesture information CN.

[0071] In one embodiment, the processor 214 obtains a reference gesture based on second gesture information C1 to second gesture information CN from each external gesture information provider 221 to external gesture information provider 22N. In another embodiment, the processor 214 obtains a first gesture detected by each external gesture information provider 221 to external gesture information provider 22N based on the corresponding second gesture information C1 to second gesture information CN. For example, the processor 214 may obtain the third posture of each predetermined joint from the second gesture information C1 and construct a first gesture corresponding to external gesture information provider 221 by connecting the predetermined joints. As another example, the processor 214 may obtain the third posture of each predetermined joint from the second gesture information C2 and construct a first gesture corresponding to external gesture information provider 222 by connecting the predetermined joints. Similarly, the processor 214 may obtain the third posture of each predetermined joint from the second gesture information CN and construct a first gesture corresponding to external gesture information provider 22N by connecting the predetermined joints, but the invention is not limited thereto.

[0072] Next, the processor 214 combines the first gestures detected by each of the external gesture information providers 221 to 22N into a reference gesture. In one embodiment, the processor 214 may use a linear / non-linear combination of the first gestures corresponding to each of the external gesture information providers 221 to 22N as the reference gesture. In another embodiment, the processor 214 may use a first average value of the first gestures detected by each of the external gesture information providers 221 to 22N as the reference gesture, but the invention is not limited thereto.

[0073] Then, the processor 214 can combine the predicted gesture G1 with the reference gesture to form the gesture of hand O1 at time point i.

[0074] In one embodiment, the processor 214 may use a linear / non-linear combination of the predicted gesture G1 and the reference gesture as the gesture of hand O1. In another embodiment, the processor 214 may use a second average of the predicted gesture and the reference gesture as the gesture of hand O1 at time point i.

[0075] Specifically, when the information of hand O1 is insufficient to recognize the gesture of hand O1 (e.g., hand O1 is not within FOV 299), processor 214 may need to display the gesture at a lower frame rate to prepare more resources for processing the second gesture information C1 to the second gesture information CN, thereby optimizing the gesture of hand O1. Therefore, the solution provided by the embodiments of the present invention can be used to determine the gesture of hand O1 when the information of hand O1 is insufficient to recognize the gesture of hand O1 (e.g., hand O1 is not within FOV 299).

[0076] In the second embodiment, the processor 214 may further correct the first gesture detected by each of the external gesture information providers 221 to 22N based at least on the correction factors of each of the external gesture information providers 221 to 22N before combining the first gesture into a reference gesture.

[0077] In embodiments where each external gesture information provider 221 to external gesture information provider 22N is an external camera, the correction factor considered by each external gesture information provider 221 to external gesture information provider 22N may include external parameters of each external gesture information provider 221 to external gesture information provider 22N. Details regarding the correction of the first gesture detected by each external gesture information provider 221 to external gesture information provider 22N based on these external parameters can be found in related art, which will not be discussed further herein.

[0078] In one embodiment, in response to a change in the determination that the information of hand O1 changes from insufficient to recognize a gesture to sufficient for recognizing a gesture, processor 214 may send a first control signal CS1 to each of the external gesture information providers 221 to 22N to request the external gesture information providers 221 to 22N to provide corresponding first gesture information P1 to PN, but the invention is not limited thereto. In this case, processor 214 may then proceed to steps S321 to S323, the details of which can be found in the above description.

[0079] In one embodiment, after the hand gesture of O1 has been determined in step S321 or step S333, the processor 214 can adjust the visual content of the real-world service based on the hand gesture of O1. For example, the processor 214 can display the hand gesture of O1 in the visual content for the user to see and interact with the real-world service, but the invention is not limited thereto.

[0080] Furthermore, although the concept of the present invention is explained above using only one hand O1, embodiments of the present invention can be used to simultaneously track the gestures of multiple hands. For example, for some hands within the FOV 299, the processor 214 can perform steps S321 to S323 to obtain their gestures. For some hands outside the FOV 299, the processor 214 can perform steps S331 to S333 to obtain their gestures, but the present invention is not limited thereto.

[0081] The present invention further provides a computer-readable storage medium for performing a method for detecting gestures. The computer-readable storage medium comprises a plurality of program instructions (e.g., setup program instructions and deployment program instructions) implemented herein. These program instructions can be loaded into and executed by a host 21 to perform the method for detecting gestures and the functions of the host 21 described above.

[0082] In summary, embodiments of the present invention provide a mechanism for enabling the host to determine gestures while cooperating with one or more external gesture information providers when the hand to be tracked is not visible to the host. Furthermore, when the hand to be tracked is visible to the host, embodiments of the present invention provide a mechanism for achieving higher accuracy in gesture tracking by considering first gesture information provided by external gesture information providers.

[0083] Furthermore, embodiments of the present invention can be used to implement gesture tracking for multiple hands. Therefore, the accuracy of gesture tracking can be improved, enabling the host to provide users with better real-world services.

[0084] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the invention without departing from its scope or spirit. In view of the foregoing, it is intended that this invention cover modifications and variations thereof, provided that such modifications and variations fall within the scope of the appended claims and their equivalents.

Claims

1. A method for detecting gestures, applicable to a host computer, characterized in that, include: Information about the hand is tracked by a tracking camera on the host computer; Determining whether the information about the hand is sufficient to recognize the hand gesture includes: Detect the joints on the gesture; In response to determining that the number of joints on the detected gesture is higher than a number threshold, it is determined that the information of the hand is sufficient to identify the gesture; In response to determining that the number of joints on the detected gesture is not higher than the number threshold, it is determined that the information of the hand is insufficient to recognize the gesture; In response to determining that the information of the hand is sufficient to recognize the gesture, the gesture is recognized based on the image of the hand captured by the tracking camera to obtain an initial recognized gesture, first gesture information is received from at least one external gesture information provider, and the initial recognized gesture is corrected based on the first gesture information to obtain a corrected recognized gesture, wherein the at least one external gesture information provider includes at least one external tracking camera; In response to determining that the information of the hand is insufficient for recognizing the gesture, second gesture information is received from the at least one external gesture information provider, a predicted gesture is determined, and the gesture is recognized based on the predicted gesture and the second gesture information to obtain an auxiliary recognition gesture, wherein the second gesture information includes more gesture information than the first gesture information; and The host adjusts the visual content of the reality service based on the corrected recognition gesture or the assisted recognition gesture.

2. The method for detecting a gesture according to claim 1, wherein the step of determining whether the information of the gesture is sufficient for recognizing the gesture includes: Determine whether the hand is within the field of view of the host's camera; In response to determining that the hand is within the field of view of the camera of the host computer, it is determined that the information of the hand is sufficient to recognize the gesture; In response to determining that the hand is not in the field of view of the camera on the host, it is determined that the information about the hand is insufficient to recognize the gesture.

3. The method for detecting gestures according to claim 1, wherein the first gesture information includes partial gesture information from each of the external gesture information providers, the partial gesture information including a first posture of at least one specific joint on the hand detected by each of the external gesture information providers.

4. The method for detecting a gesture according to claim 3, wherein the step of correcting the initial recognized gesture based on the first gesture information includes: Obtain the second pose of each specific joint in the initial recognition gesture detected by the host; The second posture of each specific joint is combined with the corresponding first posture.

5. The method for detecting gestures according to claim 3, wherein the at least one specific joint includes the fingertip joints of the hand.

6. The method for detecting gestures according to claim 1, further comprising: In response to determining that the information of the hand changes from being insufficient to recognize the gesture to being sufficient to recognize the gesture, a first control signal is sent to each of the external gesture information providers, wherein the first control signal controls each of the external gesture information providers to provide the corresponding first gesture information; In response to determining that the information of the hand changes from sufficient to recognize the gesture to insufficient to recognize the gesture, a second control signal is sent to each of the external gesture information providers, wherein the second control signal controls each of the external gesture information providers to provide corresponding second gesture information.

7. The method for detecting a gesture according to claim 1, wherein the step of obtaining the predicted gesture includes: In response to determining that the information of the hand has changed from being sufficient to recognize the gesture to being insufficient to recognize the gesture at time point i, the previous gesture recognized at time point (ik) is obtained, where i is an index value and k is a positive integer; The gesture at the i-th time point is predicted based on the previous gesture as the predicted gesture.

8. The method for detecting gestures according to claim 1, wherein the hand is pre-constructed from a specific number of pre-defined joints, and the second gesture information includes complete gesture information from each of the external gesture information providers, the complete gesture information including a third posture of each of the pre-defined joints on the hand detected by each of the external gesture information providers.

9. The method for detecting a gesture according to claim 1, wherein the step of obtaining the auxiliary recognition gesture based on the predicted gesture and the second gesture information comprises: A reference gesture is obtained based on the second gesture information from each of the external gesture information providers; as well as The predicted gesture is combined with the reference gesture to form the auxiliary recognition gesture.

10. The method for detecting a gesture according to claim 9, wherein the step of obtaining the reference gesture based on the second gesture information of each of the external gesture information providers comprises: The first gesture detected by each of the external gesture information providers is obtained based on the corresponding second gesture information; The first gestures detected by each of the external gesture information providers are combined into the reference gesture.

11. The method for detecting a gesture according to claim 10, wherein the step of combining the first gestures detected by each of the external gesture information providers into the reference gesture comprises: The first average value of the first gesture detected by each of the external gesture information providers is used as the reference gesture.

12. The method for detecting a gesture according to claim 10, wherein prior to the step of combining the first gestures detected by each of the external gesture information providers into the reference gesture, the method further comprises: The first gesture detected by each of the external gesture information providers is corrected at least based on the correction factor of each of the external gesture information providers.

13. The method for detecting gestures according to claim 12, wherein each of the external gesture information providers is an external camera, and the correction factor of each of the external gesture information providers includes external parameters of each of the external gesture information providers.

14. The method for detecting a gesture according to claim 9, wherein the step of combining the predicted gesture with the reference gesture to form the gesture comprises: The second average of the predicted gesture and the reference gesture is taken as the gesture.

15. A system for detecting gestures, characterized in that, include: The host is configured to perform: Information about the hand is tracked by a tracking camera on the host computer; Determine whether the information about the hand is sufficient to recognize the hand gesture; Detect the joints on the gesture; In response to determining that the number of joints on the detected gesture is higher than a number threshold, it is determined that the information of the hand is sufficient to identify the gesture; In response to determining that the number of joints on the detected gesture is not higher than the number threshold, it is determined that the information of the hand is insufficient to recognize the gesture; In response to determining that the information of the hand is sufficient to recognize the gesture, the gesture is recognized based on the image of the hand captured by the tracking camera to obtain an initial recognized gesture, first gesture information is received from at least one external gesture information provider, and the initial recognized gesture is corrected based on the first gesture information to obtain a corrected recognized gesture, wherein the at least one external gesture information provider includes at least one external tracking camera; In response to determining that the information of the hand is insufficient to recognize the gesture, second gesture information is received from the at least one external gesture information provider, a predicted gesture is determined, and the gesture is recognized based on the predicted gesture and the second gesture information to obtain an auxiliary recognition gesture, wherein the second gesture information includes more gesture information than the first gesture information; as well as Adjust the visual content of the reality service based on the corrected recognition gesture or the assisted recognition gesture.

16. The system for detecting gestures according to claim 15, further comprising the at least one external gesture information provider.

17. The system for detecting gestures according to claim 16, wherein each of the external gesture information providers is an external camera.

18. The system for detecting gestures according to claim 15, wherein the host performs: In response to determining that the information of the hand changes from being insufficient to recognize the gesture to being sufficient to recognize the gesture, a first control signal is sent to each of the external gesture information providers, wherein the first control signal controls each of the external gesture information providers to provide the corresponding first gesture information; In response to determining that the information of the hand changes from sufficient to recognize the gesture to insufficient to recognize the gesture, a second control signal is sent to each of the external gesture information providers, wherein the second control signal controls each of the external gesture information providers to provide the corresponding second gesture information.

19. A computer-readable storage medium, characterized in that, The computer-readable storage media records an executable computer program, which is loaded by a host computer to perform the following steps: Information about the hand is tracked by a tracking camera on the host computer; Determining whether the information about the hand is sufficient to recognize the hand gesture includes: Detect the joints on the gesture; In response to determining that the number of joints on the detected gesture is higher than a number threshold, it is determined that the information of the hand is sufficient to recognize the gesture; In response to determining that the number of joints on the detected gesture is not higher than the number threshold, it is determined that the information of the hand is insufficient to recognize the gesture; in response to determining that the information of the hand is sufficient to recognize the gesture, the gesture is recognized based on the image of the hand captured by the tracking camera to obtain an initial recognized gesture, first gesture information is received from at least one external gesture information provider, and the initial recognized gesture is corrected based on the first gesture information to obtain a corrected recognized gesture, wherein the at least one external gesture information provider includes at least one external tracking camera; In response to determining that the information of the hand is insufficient for recognizing the gesture, second gesture information is received from the at least one external gesture information provider, a predicted gesture is determined, and the gesture is recognized based on the predicted gesture and the second gesture information to obtain an auxiliary recognition gesture, wherein the second gesture information includes more gesture information than the first gesture information; and Adjust the visual content of the reality service based on the corrected recognition gesture or the assisted recognition gesture.

Images