Method, device and storage medium for input recognition in a virtual scene

Abstract

A method, device, and storage medium for input recognition in a virtual scene are provided. [Solution] An input recognition method, which uses a stereo vision algorithm to calculate the coordinates of fingertips based on the recognized positions of key points of the hand, compares the coordinates of the fingertips with at least one virtual input interface in a virtual scene, and determines that the user performs an input operation through the target virtual input interface if the positions of the fingertips and a target virtual input interface in the at least one virtual input interface satisfy a set position rule. With such a configuration, the positions of the user's fingertips can be calculated by the stereo vision algorithm, and the user does not need to interact with a controller or a special sensor device in the real world, further improving the immersiveness and realism of the virtual scene.

Description

[Technical field]

[0001] <CROSS REFERENCE TO RELATED APPLICATIONS> This application cites Chinese patent application No. 202210261992.8, entitled "Method, device, and storage medium for input recognition in virtual scenes," filed on March 16, 2022, the entire contents of which are incorporated herein by reference.

[0002] TECHNICAL FIELD Embodiments of the present application relate to the technical field of virtual reality or augmented reality, and in particular to a method, device and storage medium for input recognition in a virtual scene. [Background technology]

[0003] With the rapid development of related technologies such as virtual reality, augmented reality, and mixed reality, head-mounted smart devices, including smart glasses such as head-mounted virtual reality glasses and head-mounted mixed reality glasses, are being developed one after another, gradually improving user experience.

[0004] In the prior art, smart glasses are used to generate virtual interfaces such as holographic keyboards and holographic screens, and controllers or special sensor devices can be used to determine whether a user has interacted with the virtual interface, allowing the user to use the keyboard or screen in the virtual world.

[0005] However, this method requires the user to interact with controllers and special sensor devices in the real world, which reduces the user's sense of immersion and realism. Summary of the Invention [Problem to be solved by the invention]

[0006] The embodiments of the present application provide a method, device, and storage medium for input recognition in a virtual scene, which enable a user to perform an input operation without using additional hardware, thereby reducing hardware costs. [Means for solving the problem]

[0007] The embodiment of the present application is A method for input recognition in a virtual scene applied to a smart device, comprising: Provided is a method for input recognition in a virtual scene, the method including the steps of: recognizing key points of a user's hand in a binocular image captured by a binocular camera; calculating the coordinates of fingertips using a stereo vision algorithm based on the positions of the key points of the hand in the binocular image; comparing the coordinates of the fingertips with at least one virtual input interface in the virtual scene; and determining that a user performs an input operation through the target virtual input interface if the positions of the fingertips and a target virtual input interface in the at least one virtual input interface satisfy a set position rule.

[0008] More preferably, the step of recognizing key points of a user's hand in a binocular image taken with a binocular camera includes the steps of: detecting a hand area from any monocular image in the binocular image using a target detection algorithm; segmenting a foreground image corresponding to the hand area from the monocular image; and recognizing the foreground image using a predetermined hand key point recognition model and obtaining the hand key points in the monocular image.

[0009] More preferably, the step of calculating the coordinates of the fingertip using a stereo vision algorithm based on the positions of the hand key points in the binocular image includes the steps of: determining, for any of the user's fingers, whether or not the recognized hand key points include a fingertip joint point of the finger; and if the hand key points include the fingertip joint point of the finger, calculating the position of the fingertip joint point of the finger in the virtual scene using a stereo vision algorithm based on the positions of the fingertip joint points in the binocular image, and setting the position as the coordinates of the fingertip.

[0010] More preferably, the method further includes, when the hand key points do not include fingertip joint points, a step of calculating a finger bending angle based on the positions of visible finger key points in the binocular image and phalangeal association features when performing an input operation, and a step of calculating the coordinates of the fingertip from the finger bending angle and the positions of visible finger key points in the binocular image.

[0011] More preferably, the finger includes a first phalange close to a palm, a second phalange connected to the first phalange, and a tip phalange connected to the second phalange, and the step of calculating a bending angle of the finger based on positions of visible key points of the finger in the binocular image and phalange association features when performing an input operation includes the steps of: determining actual lengths of the first phalange, the second phalange, and the tip phalange of the finger; calculating observed lengths of the first phalange, the second phalange, and the tip phalange according to coordinates of the recognized key points of the hand; determining that the bending angle of the finger is less than 90 degrees if the observed lengths of the second phalange and / or the tip phalange are less than the corresponding actual lengths, and calculating the bending angle of the finger from the observed length and actual length of the second phalange and / or the observed length and actual length of the tip phalange; and determining that the bending angle of the finger is 90 degrees if the observed length of the second phalange and / or the tip phalange is 0.

[0012] More preferably, the step of calculating coordinates of the fingertip from the bend angle of the finger and positions of visible key points of the finger in the binocular image includes the steps of: calculating coordinates of the fingertip from a position of a start joint point of the second phalange, the bend angle of the finger, an actual length of the second phalange, and an actual length of the tip phalange when the bend angle of the finger is less than 90 degrees; and calculating a position of the fingertip from the position of a start joint point of the second phalange and a moving distance of the first phalange to the at least one virtual input interface when the bend angle of the finger is 90 degrees.

[0013] More preferably, when the position of the fingertip and a target virtual input interface in the at least one virtual input interface satisfy a set position rule, the step of determining that the user performs an input operation through the target virtual input interface includes a step of determining that the user is touching the target virtual input interface when the position of the fingertip is on the target virtual input interface, and / or determining that the user is clicking on the target virtual input interface when the position of the fingertip is on the side of the target virtual input interface away from the user and the distance to the target virtual input interface is greater than a preset distance threshold.

[0014] More preferably, an infrared sensor is attached to the smart device, and the method further includes a step of collecting distance values ​​between the infrared sensor and key points of the hand by the infrared sensor, and a step of performing position correction on the calculated position of the user's fingertip using the distance values.

[0015] The embodiment of the present application is A memory and a processor, the memory storing one or more computer instructions; There is further provided a terminal device, wherein the processor is for performing steps of a method for input recognition in a virtual scene by executing the one or more computer instructions.

[0016] An embodiment of the present application further provides a computer readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform steps of a method for recognizing an input in a virtual scene. Effect of the Invention

[0017] In the input recognition method, device, and storage medium in a virtual scene according to the embodiment of the present application, a stereo vision algorithm is used to calculate the coordinates of a fingertip based on the position of the recognized key points of the hand, the coordinates of the fingertip are compared with at least one virtual input interface in the virtual scene, and if the position of the fingertip and the target virtual input interface in the at least one virtual input interface satisfy a set position rule, it is determined that the user performs an input operation through the target virtual input interface. With such a configuration, the position of the user's fingertip can be calculated using a stereo vision algorithm, and the user does not need to interact with a controller or a special sensor device in the real world, and the immersiveness and realism of the virtual scene can be further improved. [Brief description of the drawings]

[0018] In order to more clearly describe the embodiments of the present invention or the technical solutions in the prior art, the drawings necessary for the description of the embodiments or the prior art are briefly described below. The drawings in the following description show some embodiments of the present invention, and it is obvious that those skilled in the art can derive other drawings based on these drawings without exerting creative efforts. [Figure 1] 1 is a flow diagram of an input recognition method according to an embodiment of the present application; [Diagram 2]FIG. 2 is a schematic diagram of key points of a hand shown as an example of the present application. [Diagram 3] FIG. 2 is a schematic diagram of target detection as an example of the present application. [Figure 4] FIG. 2 is a schematic diagram of foreground image segmentation shown as an example in the present application. [Diagram 5] FIG. 2 is a schematic diagram of a stereo vision algorithm presented as an example in the present application; [Figure 6] FIG. 2 illustrates the image formation principle of a stereo vision algorithm presented as an example in the present application. [Figure 7] FIG. 2 is a schematic diagram of a phalanx shown as an example in the present application. [Figure 8] FIG. 2 is a schematic diagram of calculation of key point positions of a hand shown as an example of the present application. [Figure 9] FIG. 2 is a schematic diagram of a virtual input interface shown as an example in the present application. [Figure 10] FIG. 2 is a schematic diagram of the parallax of a binocular camera shown as an example of the present application. [Figure 11] FIG. 2 is a schematic diagram of a terminal device shown as an example of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the drawings in the embodiments of the present invention. However, it is clear that the described embodiments are only a part of the present invention and do not cover all the embodiments. All other embodiments that can be obtained by those skilled in the art without any creative efforts based on the embodiments of the present invention are included in the protection scope of the present invention.

[0020] In the conventional technology, smart glasses are used to generate virtual interfaces such as holographic keyboards and holographic screens, and controllers or special sensor devices can be used to determine whether a user has interacted with the virtual interface. This allows the user to use the keyboard or screen in the virtual world. However, this method still requires the user to interact with controllers or special sensor devices in the real world, which reduces the user's sense of immersion and realism.

[0021] In order to address the above technical problems, some embodiments of the present application provide solutions. The technical solutions according to the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0022] FIG. 1 is a schematic flow diagram of a method for input recognition in a virtual scene shown as an example of the present application. As shown in FIG. 1, the method includes the following steps 11 to 14.

[0023] Step 11: Recognize key points of the user's hand in the binocular image captured by the binocular cameras.

[0024] Step 12: Based on the positions of the hand key points in the binocular images, calculate the coordinates of the fingertips using a stereo vision algorithm.

[0025] Step 13: Compare the fingertip coordinates with at least one virtual input interface in the virtual scene.

[0026] Step 14: If the position of the fingertip and the target virtual input interface in the at least one virtual input interface satisfy the set position rule, it is determined that the user performs an input operation through the target virtual input interface.

[0027] The present embodiment can be executed by a smart device, which may include, but is not limited to, wearable devices such as Virtual Reality (VR) glasses, Mixed Reality (MR) glasses, and VR Head-Mounted Display (HMD). Taking the VR glasses as an example, when the VR glasses display a virtual scene, at least one virtual input interface including at least one virtual input interface such as a virtual keyboard and / or a virtual screen can be generated in the virtual scene. A user can interact with these virtual input interfaces in the virtual scene.

[0028] In this embodiment, the smart device can use a binocular image of the hands captured by a binocular camera. Here, the binocular camera may be mounted on the smart device or in another position capable of capturing both hands, but this embodiment is not limited thereto. The binocular camera includes two monocular cameras, and the binocular image is composed of two monocular images.

[0029] The smart device may recognize key points of the user's hand in a binocular image of the hand captured by a binocular camera, which may be the knuckles of the user's fingers, fingertips, or any position on the hand, as shown in the schematic diagram of FIG.

[0030] After recognizing the key points of the user's hand, a stereo vision algorithm may be used to calculate the coordinates of the fingertips based on the positions of the key points of the hand in the binocular image. Here, the stereo vision algorithm is also known as the binocular vision algorithm, which is an algorithm that simulates the principle of human vision to passively detect distance by a computer, and its main principle is as follows: an object is observed from two points, images are obtained at different visual angles, and the position of the object is calculated according to the pixel matching relationship between the images and the triangulation principle.

[0031] After calculating the position of the fingertip, the coordinates of the fingertip may be compared with at least one virtual input interface in the virtual scene. If the position of the fingertip and the target virtual input interface in the at least one virtual input interface satisfy the set position rule, it is determined that the user performs an input operation through the target virtual input interface. Here, the input operation by the user includes at least a click, a long press, or a touch, etc.

[0032] In this embodiment, the smart device uses a stereo vision algorithm to calculate the coordinates of the fingertip based on the recognized position of the key points of the hand, compares the coordinates of the fingertip with at least one virtual input interface in the virtual scene, and determines that the user performs an input operation through the target virtual input interface if the position of the fingertip and the target virtual input interface in the at least one virtual input interface satisfy the set position rule. According to this configuration, the position of the user's fingertip can be calculated by the stereo vision algorithm, eliminating the need for the user to interact with a controller or special sensor device in the real world, and further improving the immersiveness and realism of the virtual scene.

[0033] Furthermore, in this embodiment, an image of the hand is captured using an existing binocular camera installed in the smart device or the environment, and the user can perform input operations without using additional hardware, thereby reducing hardware costs.

[0034] In some preferred embodiments, the operation of "recognizing key points of a user's hand in a binocular image captured by a binocular camera" described in the above embodiment may be realized by the following steps.

[0035] As shown in Fig. 3, for any monocular image in the binocular image, the smart device may detect a hand region from the monocular image using a target detection algorithm (Object Detection), where the target detection algorithm may be realized based on a Region-Convolutional Neural Network (R-CNN).

[0036] The target detection algorithm is further described below.

[0037] For one picture, the algorithm generates about 2000 candidate regions based on the picture, then converts each candidate region to a predetermined size, and sends the modified candidate regions to a Convolutional Neural Network (CNN) model. Furthermore, a feature vector corresponding to each candidate region can be obtained through the model. The feature vector is then sent to a classifier including multiple categories, and the value of the probability that the image in the candidate region belongs to each category can be predicted. For example, if the prediction result by the classifier is that the image in the total of 10 candidate regions, candidate regions 1 to 10, has a 95% probability of belonging to the hand region and a 20% probability of belonging to the face, candidate regions 1 to 10 can be detected as hand regions. With this configuration, the smart device can accurately detect the hand region in any monocular image.

[0038] In a real scene, when a user uses a smart device to interact with a virtual scene, the user's hand is often the most visible object to the user, so the foreground image in a monocular image captured by any camera is usually the region of the user's hand. Therefore, as shown in FIG. 4, the smart device can segment the foreground image corresponding to the hand region from the monocular image. According to such an embodiment, the smart device can segment the hand region to reduce interference of other regions on subsequent recognition and target recognition of the hand region, thereby improving recognition efficiency.

[0039] Based on the above steps, as shown in FIG. 2, the smart device can use a pre-set hand keypoint recognition model to recognize the foreground image and obtain the hand keypoints in the monocular image. The hand keypoint recognition model can be pre-trained. For example, a hand image is input into the model to obtain the model recognition result of the hand keypoints, and the model parameters are further adjusted according to the error between the model recognition result and the expected result, and the hand keypoints are recognized again using the model with the adjusted parameters. By repeating in this way, the hand keypoint recognition model can accurately recognize the foreground image corresponding to the hand region and obtain the hand keypoints in the monocular image.

[0040] In an actual scene, when a user performs an input operation, the fingertip may be blocked by other parts of the hand, and as a result, the binocular camera cannot capture the user's fingertip, and the fingertip joint points are missing from the recognized key points of the hand. Here, the fingertip joint points are shown at 4, 8, 12, 16, and 20 in FIG. 2. On the other hand, if the user's fingertip can be captured by one of the binocular cameras, the recognized key points of the hand may include the fingertip joint points.

[0041] Preferably, after recognizing the key points of the user's hand as described in the above embodiment, the step of calculating the coordinates of the fingertips using a stereo vision algorithm based on the positions of the key points of the hand in the binocular images may be realized according to the following steps S1 and S2.

[0042] Step S1: For any of the user's fingers, it is determined whether the recognized hand key points include the fingertip joint point of the finger.

[0043] Step S2: If the key points of the hand include the fingertip joint points, calculate the positions of the fingertip joint points in the virtual scene using a stereo vision algorithm based on the positions of the fingertip joint points in the binocular images, and use this as the coordinates of the fingertip.

[0044] The stereo vision algorithm will now be described in detail with reference to FIGS.

[0045] The two rectangles on the left and right sides of FIG. 5 represent the camera planes of the two cameras on the left and right sides, respectively. Point P indicates the target object (the knuckle point of the user's finger), P1 and P2 are the projections of point P on the two camera planes, respectively, and the image points on the image formation planes of the two cameras on the left and right sides of one point P (X, Y, Z) in the world space are P1 (ul, vl) and P2 (ur, vr), respectively. These two image points are images of the same object point P in the world space (world coordinate system), and are called "conjugate points." The two conjugate image points are connected by projection lines PlOl and P2Or that connect the respective image points and the optical centers Ol and Or of the cameras, and the intersection of these lines is the object point P (X, Y, Z) in the world space (world coordinate system).

[0046] Specifically, Figure 6 shows the principle of simple head-up binocular stereoscopic imaging. Let T be the baseline distance, which is the distance of the connecting line between the projection centers of the two cameras. The origin of the camera coordinate system is at the optical center of the camera lens, and the coordinate system is shown in Figure 6. The image-forming original plane of the camera is behind the optical center of the lens, and the left and right image-forming planes are located in front of the optical center of the lens at f, and the u and v axes of this virtual image plane coordinate system O1uv are in the same direction as the x and y axes of the camera coordinate system. This simplifies the calculation process. The origins of the left and right image coordinate systems are at the optical axis of the camera and the focal points O1 and O2 of the plane, and the corresponding coordinates of point P in the left and right images are xl(u1,v1) and xr(u2,v2), respectively. Assuming that the images of the two cameras are on the same plane, the Y coordinate of the image coordinates of point P is the same, that is, v1=v2, and the following equation can be obtained from the geometric relationship of the triangle: JPEG2025508595000002.jpg18128

[0047] In the above, (x,y,z) are the coordinates of point P in the left camera coordinate system, T is the baseline distance, f is the focal length of the two cameras, and (u1,v1) and (u1,v2) are the coordinates of point P in the left and right images, respectively.

[0048] Disparity is defined as the difference in position d between a particular point and corresponding points in the two images. JPEG2025508595000003.jpg20125

[0049] As a result, the coordinates of point P in the left camera coordinate system are calculated as follows: JPEG2025508595000004.jpg20115

[0050] Based on the above process, the points on the image forming planes of the two left and right cameras corresponding to the fingertip joint points (i.e., the positions of the fingertip joint points in the binocular images) are identified, and the internal and external parameters of the cameras are obtained through camera calibration, so that the 3D coordinates of the fingertip joint points in the world coordinate system can be determined using the above formula.

[0051] Preferably, a correspondence relationship may be set in advance between the coordinate system of the virtual scene generated by the smart device and the three-dimensional coordinates in the world coordinate system. Based on this correspondence relationship, the three-dimensional coordinates of the fingertip joint point obtained above are converted into the coordinate system of the virtual scene to obtain the position of the fingertip joint point in the virtual scene, which is set as the coordinates of the fingertip of the finger.

[0052] According to the above embodiment, even if the fingertip joint points are occluded, the smart device can accurately calculate the coordinates of the fingertip of the user's finger using stereo vision algorithms.

[0053] As shown in FIG. 7, the finger includes a first phalange close to the palm, a second phalange connected to the first phalange, and a tip phalange connected to the second phalange. Each phalange of a human finger has a certain bending rule when bending. For example, most people usually cannot bend the tip phalange without moving the second phalange and the first phalange, and when the tip phalange bends downward by 20°, the second phalange also usually bends by a certain angle as the tip phalange bends. The reason for such bending rule is that each phalange of a human finger is related, that is, there is a feature of phalange association.

[0054] Based on the above, in some preferred embodiments, when a user's fingertip joint points are occluded, the step of calculating the coordinates of the fingertip using a stereo vision algorithm based on the positions of the hand key points in the binocular images may be realized according to the following steps:

[0055] Step S3: If the hand key points do not include the fingertip joint points, calculate the finger bending angle from the positions of the visible key points of the fingers in the binocular image and the phalangeal association features when performing the input operation.

[0056] Step S4: Calculate the coordinates of the fingertip from the finger bending angle and the positions of visible key points of the finger in the binocular image.

[0057] In such an embodiment, the coordinates of a finger's tip may be calculated by visible keypoint and phalange-associated features even when the finger's phalange is occluded and blocked.

[0058] The visible key point in step S3 refers to a key point that can be detected in the binocular image. For example, when a user's little finger is bent at a certain angle and the tip phalanx of the little finger is blocked by the palm, the fingertip joint point of the user's little finger cannot be recognized in this state, that is, the fingertip joint point of the little finger is an invisible key point. Other key points of the hand other than the tip joint point are recognized normally, that is, the other key points of the hand are visible key points. Here, the bending angle of the finger includes each bending angle of one or more phalanges.

[0059] In some preferred embodiments, the above step S3 may be realized according to the following embodiments.

[0060] The actual length of each of the first, second and tip phalanges of the fingers is determined, and the observed length of each of the first, second and tip phalanges is calculated based on the coordinates of the recognized hand keypoints.

[0061] As shown in Figure 8, the observed length is the length of the finger observed from the angle of the binocular camera, and this observed length is the length of each phalanx calculated from the hand keypoints, that is, the projected length to the camera. For example, when two hand keypoints R1 and R2 corresponding to the first phalanx of a finger are recognized, the observed length of the first phalanx can be calculated from the coordinates of these two hand keypoints.

[0062] Preferably, the finger bending angle is determined to be less than 90 degrees if the observed length of the second phalange is less than the actual length corresponding to the second phalange, or the observed length of the tip phalange is less than the actual length corresponding to the tip phalange, or the observed lengths of both the second phalange and the tip phalange are less than their respective actual lengths. In this case, the finger bending angle can be calculated from the observed length and actual length of the second phalange, or the finger bending angle can be calculated from the observed length and actual length of the tip phalange, or the finger bending angle can be calculated from the observed length of the second phalange, the actual length of the second phalange, the observed length of the tip phalange and the actual length of the tip phalange.

[0063] Hereinafter, calculation of the bending angle from the observed length and the actual length will be described by way of example with reference to FIG.

[0064] FIG. 8 exemplarily shows the state of the phalanges when the finger is bent and the key points R1, R2, and R3 corresponding to the phalanges, where the first phalange is R1R2, the second phalange is R2R5, and the tip phalange is R5R6. As shown in FIG. 8, in a triangle formed by R1, R2, and R3, if R2R3 (observed length) and R1R2 (actual length) are known, the bending angle a of the first phalange can be obtained. Similarly, in a triangle formed by R4, R2, and R5, if R2R5 (actual length) and R2R4 (observed length) are known, the bending angle b of the second phalange can be obtained. Similarly, the bending angle c of the third phalange can be obtained.

[0065] Preferably, if the observation length of the second phalange and / or the tip phalange is 0, the binocular camera cannot observe the second phalange and / or the tip phalange, in which case it may be assumed that the finger bending angle is 90 degrees based on the finger bending characteristics.

[0066] Preferably, based on the above bending angle calculation process, the step of "calculating the coordinates of the fingertip from the bending angle of the finger and the position of the visible key point of the finger in the binocular image" described in the above embodiment may be realized based on the following embodiment. EMBODIMENT 1

[0067] If the bending angle of the finger is less than 90 degrees, the coordinates of the fingertip may be calculated from the position of the starting joint point of the second phalanx, the bending angle of the finger, the actual length of the second phalanx, and the actual length of the tip phalanx.

[0068] As shown in Figure 8, when the start joint point R2 of the second phalange, R2, can be observed, the position of R2 can be calculated using a stereo vision algorithm. When the position of R2, the actual length of the second phalange, and the bending angle b of the second phalange are known, the position of the start joint point R5 of the tip phalange can be obtained. Furthermore, the position of the fingertip R6 can be calculated from the position of R5, the bending angle c of the tip phalange, and the actual length of the tip phalange. EMBODIMENT 2

[0069] When the finger is bent at a 90 degree angle, the position of the fingertip is calculated from the position of the starting joint point of the second phalanx and the distance of the movement of the first phalanx to at least one virtual input interface.

[0070] In addition, when the finger bending angle is 90 degrees, the user's fingertip may move in the same manner as the first phalanx, for example, when the first phalanx moves 3 cm downward, the fingertip also moves 3 cm. Based on this, when the position of the starting joint point of the second phalanx and the moving distance of the first phalanx to at least one virtual input interface are known, the position of the fingertip is calculated, and the problem of calculating the fingertip position is converted into a geometric problem of calculating the end point position when the position of the starting point, the moving direction of the starting point and the moving distance are known. This will not be described in detail here.

[0071] In some preferred embodiments, after calculating the position of the fingertip, the position of the fingertip is compared with at least one virtual input interface in the virtual scene, and whether the user performs an input operation is determined based on the comparison result. The following description will be given by taking one of the at least one virtual input interface as an example.

[0072] EMBODIMENT 1 If the position of the fingertip is on the target virtual input interface, it is determined that the user is touching the target virtual input interface.

[0073] EMBODIMENT 2 When the position of the fingertip is on the side of the target virtual input interface away from the user and the distance between the target virtual input interface and the fingertip is greater than a preset distance threshold, it is determined that the user clicks on the target virtual input interface, where the distance threshold may be preset to 1 cm, 2 cm, 5 cm, etc., but is not limited thereto in this embodiment.

[0074] The above two embodiments may be implemented individually or in combination, but the present embodiment is not limited to this.

[0075] Preferably, an infrared sensor may be attached to the smart device. After calculating the position of the user's fingertip, the smart device may use the infrared sensor to collect distance values ​​between the infrared sensor and key points on the hand. The calculated position of the user's fingertip can be corrected using the distance values.

[0076] According to such a position correction method, it is possible to reduce the error between the calculated fingertip position and the actual fingertip position, thereby further improving the recognition accuracy of recognizing an input operation by a user.

[0077] The above input recognition method will be further described below with reference to Figs. 9 and 10 and an actual application scene.

[0078] As shown in Figure 9, the virtual screen and virtual keyboard generated by the VR glasses (smart device) are virtual stereoscopic planes, which also function as a boundary (i.e., a virtual input interface). A user can interact with the virtual plane and virtual keyboard by appropriately adjusting the positions of the virtual plane and virtual keyboard and clicking or pressing or pulling an action button. If the user's fingertip crosses the boundary, it is determined that the user is clicking. If the user's fingertip is on the boundary, it is determined that the user is touching.

[0079] In order to determine whether the user's fingertip is beyond the boundary, it is necessary to calculate the position of the user's fingertip. At least two cameras outside the VR glasses may be used for this calculation. The following describes the case where the number of cameras is two.

[0080] In addition, when a user interacts with the VR glasses, the user's hands are generally located closest to the camera. It is assumed that the user's two hands are the objects closest to the camera, and there are no obstacles between the camera and the hands. In addition, as shown in FIG. 10, by providing a binocular camera, parallax occurs between the monocular images of different visions captured by the two monocular cameras, and the VR glasses can calculate the position of the object based on the pixel matching relationship between the two monocular images and the triangulation principle.

[0081] If the user's fingertips are not occluded by other parts of the hand, the VR glasses use a stereo vision algorithm to directly calculate the position of the user's fingertips and then determine whether the user's fingertips are over a screen, keyboard, or other virtual input interface.

[0082] Usually, each human finger has three lines (hereinafter, simply called tri-phasic lines), and the finger is divided into three parts by the tri-phasic lines: the first phalanx close to the palm, the second phalanx connected to the first phalanx, and the tip phalanx connected to the second phalanx. In addition, there is a bending correlation (phalanx association feature) between each phalanx of the user.

[0083] Based on the above, when the user's fingertip is occluded by other parts of the hand, the terminal device can determine the actual lengths of the user's first phalanges, second phalanges, and tip phalanges, and the actual lengths may be pre-measured based on the three-phalange line of the hand. In a real scene, the back of the user's hand and the first phalanges are usually visible, and the VR glasses can calculate the positions of the second phalanges and tip phalanges from the bending correlation, the observed length and the actual length of the first phalanges, and further calculate the coordinates of the fingertip.

[0084] For example, if only the first phalanx is visible, and the second and tip phalanxes are not visible at all, then it is assumed that the user's finger is bent 90 degrees. This means that the distance the first phalanx of the finger moves downward is equal to the distance the tip phalanx moves downward. Based on this, the position of the fingertip can be calculated.

[0085] After calculating the position of the fingertip, the position of the fingertip may be compared with the positions of the screen, keyboard and other virtual input interfaces. If the fingertip is over the virtual input interface and does not exceed a preset touch depth, it is determined that the user is performing a click operation, and if the fingertip is on the virtual input interface, it is determined that the user is performing a touch operation. If the fingertip is over the virtual input interface and does not exceed a preset touch depth, it is determined that the user is performing a cancel operation.

[0086] Preferably, an infrared sensor is attached to the VR glasses, which obtains the distance between the infrared sensor and the fingertip position by a time of flight (ToF) method, so that the fingertip position can be further corrected to reduce the error between the calculated fingertip position and the actual fingertip position.

[0087] FIG. 11 is a structural schematic diagram of a terminal device shown as an example of the present application. As shown in FIG. 11, the terminal device includes: a memory 1101 and a processor 1102.

[0088] The memory 1101 is used to store computer programs and is configured to store various other data to support operation on the terminal device, including instructions for any applications or methods running on the terminal device, contact data, phone book data, messages, photos, videos, etc.

[0089] Here, the memory 1101 may be implemented with any type of volatile or non-volatile storage device, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, or any combination thereof.

[0090] The processor 1102 is coupled to the memory 1101 and executes a computer program in the memory 1101 to recognize key points of a user's hand in a binocular image captured by a binocular camera, calculate the coordinates of the fingertip using a stereo vision algorithm based on the positions of the key points of the hand in the binocular image, compare the coordinates of the fingertip with at least one virtual input interface in the virtual scene, and determine that the user will perform an input operation through the target virtual input interface if the position of the fingertip and a target virtual input interface in the at least one virtual input interface satisfy a set position rule.

[0091] More preferably, when processor 1102 recognizes key points of a user's hands in a binocular image captured by a binocular camera, specifically, for any monocular image in the binocular image, it is used to detect a hand area from the monocular image using a target detection algorithm, to segment a foreground image corresponding to the hand area from the monocular image, and to recognize the foreground image using a preset hand key point recognition model and obtain hand key points in the monocular image.

[0092] More preferably, when calculating the coordinates of the fingertip using a stereo vision algorithm based on the positions of the hand key points in the binocular image, the processor 1102 is used to determine, for any of the user's fingers, whether or not the recognized hand key points include the fingertip joint point of the finger, and if the hand key points include the fingertip joint point of the finger, calculate the position of the fingertip joint point of the finger in the virtual scene using a stereo vision algorithm according to the position of the fingertip joint point in the binocular image, and use the calculated position as the coordinate of the fingertip of the finger.

[0093] More preferably, the processor 1102 is further used for, if the hand key points do not include the fingertip joint points of the fingers, calculating a bending angle of the fingers based on the positions of the visible key points of the fingers in the binocular image and the phalangeal association features when performing an input operation, and calculating the coordinates of the fingertip of the fingers from the bending angle of the fingers and the positions of the visible key points of the fingers in the binocular image.

[0094] More preferably, the finger includes a first phalange close to the palm, a second phalange connected to the first phalange, and a tip phalange connected to the second phalange, and the processor 1102, when calculating the bending angle of the finger based on the position of the visible key point of the finger in the binocular image and the phalange association feature when performing the input operation, specifically, determines the actual length of each of the first phalange, the second phalange, and the tip phalange of the finger, calculates the observed length of each of the first phalange, the second phalange, and the tip phalange based on the coordinates of the recognized key point of the hand, and determines that the bending angle of the finger is less than 90 degrees when the observed length of the second phalange and / or the tip phalange is shorter than the corresponding actual length. Calculates the bending angle of the finger from the observed length and actual length of the second phalange and / or the observed length and actual length of the tip phalange, and determines that the bending angle of the finger is 90 degrees when the observed length of the second phalange and / or the tip phalange is 0.

[0095] More preferably, when calculating the coordinates of the fingertip from the bending angle of the finger and the positions of the visible key points of the finger in the binocular image, the processor 1102 calculates the coordinates of the fingertip from the position of the start joint point of the second phalange, the bending angle of the finger, the actual length of the second phalange, and the actual length of the tip phalange when the bending angle of the finger is less than 90 degrees, and calculates the position of the fingertip from the position of the start joint point of the second phalange and the moving distance of the first phalange to the at least one virtual input interface when the bending angle of the finger is 90 degrees.

[0096] More preferably, the processor 1102 determines that the user performs an input operation through the target virtual input interface when the position of the fingertip and a target virtual input interface in the at least one virtual input interface satisfy a set position rule. Specifically, when the position of the fingertip is on the target virtual input interface, the processor 1102 determines that the user is touching the target virtual input interface, and / or when the position of the fingertip is on the side of the target virtual input interface away from the user and the distance between the target virtual input interface and the target virtual input interface is greater than a preset distance threshold, the processor 1102 determines that the user is clicking the target virtual input interface.

[0097] More preferably, the smart device is equipped with an infrared sensor, and the processor 1102 can further use the infrared sensor to collect distance values ​​between the infrared sensor and key points on the hand, and perform position correction on the calculated position of the user's fingertip with the distance values.

[0098] The memory in FIG. 11 may be implemented with any type of volatile or non-volatile storage device, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk, or any combination thereof.

[0099] The display 1103 in FIG. 11 refers to a screen including a liquid crystal display (LCD) and a touch panel (TP). When the screen includes a touch panel, the screen may be realized as a touch screen that receives input signals from a user. The touch panel includes one or more touch sensors that detect touches, swipes, and gestures on the touch panel. The touch sensors can detect the boundaries of a touch or slide motion as well as the duration and pressure associated with the touch or slide motion.

[0100] The audio component 1104 of FIG. 11 may be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive an external audio signal when the device in which the audio component is located is in an operational mode such as a call mode, a record mode, a voice recognition mode, etc. The received audio signal may be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component further includes a speaker for outputting the audio signal.

[0101] 11, the electronic device further includes other components such as a communication component 1105 and a power supply component 1106. In FIG. 11, only some components are shown diagrammatically, which does not mean that the electronic device includes only the components shown in FIG.

[0102] The communication component 1105 of FIG. 11 is configured to facilitate wired or wireless communication between the device on which the communication component is located and other devices. The device on which the communication component is located may be connected to a wireless network based on a communication standard such as WiFi, 2G, 3G, 4G, or 5G, or a combination thereof. In one exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component may be implemented based on Near Field Communication (NFC) technology, Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0103] Of these, the power supply component 1106 provides power to various components of the device in which it is located, and may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing electrical power to the device in which it is located.

[0104] In this embodiment, the terminal device uses a stereo vision algorithm to calculate the coordinates of the fingertip based on the position of the recognized key point of the hand, compares the coordinates of the fingertip with at least one virtual input interface in the virtual scene, and determines that the user performs an input operation through the target virtual input interface if the position of the fingertip and the target virtual input interface in the at least one virtual input interface satisfy the set position rule. With this configuration, the position of the user's fingertip is calculated by the stereo vision algorithm, and the immersiveness and realism of the virtual scene are further improved without the user using a controller or special sensor device to interact with the real world.

[0105] Therefore, the embodiments of the present application further provide a computer-readable storage medium storing a computer program implementing each step executable by a terminal device in the above method embodiments.

[0106] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. The present invention may also take the form of a computer program product embodied on one or more computer usable storage media (including, but not limited to, magnetic disk memories, CD-ROMs, optical memories, etc.) that contain computer usable program code.

[0107] The present invention will be described with reference to flowcharts and / or block diagrams of a method, a device (system), and a computer program product according to an embodiment of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions are provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, and the instructions executed by the processor of the computer or other programmable data processing device can generate a machine to generate means for implementing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0108] These computer program instructions may also be stored in a computer readable memory that can cause a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including an instruction apparatus that implements the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0109] These computer program instructions may be loaded into a computer or other programmable data processing device, whereby a sequence of operational steps is executed on the computer or other programmable device to generate a computer-implemented process, whereby the instructions executing on the computer or other programmable device provide steps for realizing the functions specified in one or more flows of the flowcharts and / or one or more blocks of the block diagrams.

[0110] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0111] Memory may include any form of non-persistent memory in a computer readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read only memory (ROM) or flash memory (flash RAM). Memory is one example of a computer readable medium.

[0112] Computer readable media includes permanent and non-permanent, removable and non-removable media, and may achieve storage of information by any method or technology. Information may be computer readable instructions, data structures, modules of programs, or other data. Computer storage media may include, for example, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital multifunction disk (DVD) or other optical storage, magnetic cartridges, magnetic disks or other magnetic storage devices, or other non-transmission media that may be used to store information accessible by a computing device. As defined herein, computer readable media does not include transitory computer readable media, such as modulated data signals or carriers.

[0113] It is noted that the terms "comprise," "include," or any variation thereof, are intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that includes a set of elements includes not only those elements, but also other elements not expressly listed or that are inherent to such process, method, article, or device. Unless further limited, an element qualified by the phrase "comprises..." does not exclude the presence of other identical elements in the process, method, article, or device that includes the element.

[0114] The above description is merely an embodiment of the present application, and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art in the present application. Any amendments, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for input recognition in a virtual scene applied to a smart device, comprising: Recognizing key points of a user's hand in a binocular image captured by a binocular camera; Calculating the coordinates of fingertips based on the positions of key points of the hand in the binocular images using a stereo vision algorithm; comparing the coordinates of the fingertip with at least one virtual input interface in the virtual scene; and determining that the user will perform an input operation through the target virtual input interface if the position of the fingertip and a target virtual input interface in the at least one virtual input interface satisfy a set position rule.

2. The step of recognizing key points of a user's hand in a binocular image captured by a binocular camera includes: detecting a hand region from any one of the monocular images in the binocular images using a target detection algorithm; Segmenting a foreground image corresponding to the hand region from the monocular image; and utilizing a pre-established hand keypoint recognition model to recognize the foreground image and obtain hand keypoints in the monocular image.

3. Calculating fingertip coordinates using a stereo vision algorithm based on the positions of key points of the hand in the binocular images, determining, for any finger of the user, whether the recognized hand key points include a fingertip knuckle point of the finger; and if the hand key points include a fingertip joint point of the finger, calculating a position of the fingertip joint point of the finger in the virtual scene using a stereo vision algorithm based on the position of the fingertip joint point in the binocular image, and setting the position as the coordinate of the fingertip of the finger.

4. if the hand key points do not include the fingertip joint points of the fingers, calculating a bending angle of the fingers based on the positions of the visible key points of the fingers in the binocular image and phalangeal association features when performing an input operation; 4. The method of claim 3, further comprising: calculating coordinates of the fingertip from the finger bend angle and the positions of the finger's visible key points in the binocular image.

5. the finger includes a first phalange close to the palm, a second phalange connected to the first phalange, and a tip phalange connected to the second phalange; Calculating a bending angle of the finger based on the positions of the visible key points of the finger in the binocular image and the phalangeal association features when performing an input operation, determining an actual length of each of the first phalanges, the second phalanges and the tip phalanges of the finger; calculating observed lengths of the first phalange, the second phalange and the tip phalange based on the coordinates of the recognized key points of the hand; determining that the bend angle of the finger is less than 90 degrees if the observed length of the second phalange and / or the tip phalange is shorter than the corresponding actual length, and calculating the bend angle of the finger from the observed length and actual length of the second phalange and / or the observed length and actual length of the tip phalange; and determining that the bend angle of the finger is 90 degrees if the observed length of the second phalange and / or the tip phalange is zero.

6. Calculating the coordinates of the fingertip from the bending angle of the finger and the positions of the visible key points of the finger in the binocular image, if the bending angle of the finger is less than 90 degrees, calculating the coordinates of the fingertip from the position of the start joint point of the second phalange, the bending angle of the finger, the actual length of the second phalange, and the actual length of the tip phalange; and calculating a position of the fingertip from a position of a start joint point of the second phalanx and a moving distance of the first phalanx to the at least one virtual input interface when the finger bend angle is 90 degrees.

7. determining that the user performs an input operation through the target virtual input interface when the position of the fingertip and a target virtual input interface in the at least one virtual input interface satisfy a set position rule; If the position of the fingertip is on the target virtual input interface, determining that the user is touching the target virtual input interface; and / or 2. The method of claim 1, further comprising: determining that the user is clicking on the target virtual input interface if the position of the fingertip is on a side of the target virtual input interface away from the user and the distance to the target virtual input interface is greater than a preset distance threshold.

8. The smart device is equipped with an infrared sensor, using the infrared sensor to collect distance values ​​between the infrared sensor and key points on the hand; The method according to any one of claims 1 to 7, further comprising the step of performing a position correction on the calculated position of the user's fingertip using the distance value.

9. A terminal device, comprising: a memory; and a processor; the memory storing one or more computer instructions; A terminal device, characterized in that said processor is for carrying out the steps of the method according to any one of claims 1 to 8 by executing said one or more computer instructions.

10. A computer-readable storage medium storing a computer program, the computer program being characterized in that, when executed by a processor, the computer program causes the processor to perform the steps of the method according to any one of claims 1 to 8.

Images