A virtual gesture control method and device

The user's hand images are acquired and processed through a panoramic camera and reference gestures are generated, which solves the problem of virtual gesture deviation in indirect gesture interactions, and improves the accuracy and user experience of virtual interactions.

CN117648035BActive Publication Date: 2024-11-05SHANGHAI ZUIAN NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311720763.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-14
Publication Date
2024-11-05
Estimated Expiration
2043-12-14

AI Technical Summary

Technical Problem

In non-direct gesture interaction, obstruction of the user's hand causes deviation from the virtual gestures and the actual gestures, reducing the user's user experience.

Method used

A panoramic camera is used to obtain real-time images of user hand activities, and a reference gesture is generated through contour recognition and optimization processing, combining multi-angle and continuous shooting image data to reduce the deviation of virtual gestures.

Benefits of technology

Through accurate virtual gesture control, the accuracy and user experience of virtual interaction are improved, and the deviation between virtual gestures and actual gestures is reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117648035B_ABST
    Figure CN117648035B_ABST
Patent Text Reader

Abstract

The present invention discloses a control method, device, electronic device and computer-readable storage medium for virtual gestures. The method is applicable to a smart wearable device equipped with a panoramic camera, and the method comprises: when it is determined that the user has hand activities, calling the panoramic camera to obtain a real-time image set; after extracting a number of hand images from the real-time image set, respectively performing contour recognition on each hand image to obtain a hand contour; using a number of hand contours to generate a reference gesture, and controlling the movement of the virtual gesture according to the reference gesture. The present invention can call a panoramic camera to obtain a hand image of the user's hand activities; generate a reference gesture according to the hand contour of the hand image, and control the movement of the virtual gesture according to the reference gesture. By combining the actual hand image to generate and control the virtual gesture, the deviation between the virtual gesture and the user's actual gesture can be reduced, thereby reducing the action deviation of the virtual interaction, so as to improve the accuracy of the virtual action and enhance the user's experience.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0002] Spatial interaction design is also called interactive space design. Specifically, it refers to dynamically expressing various virtual devices based on interactive experience to achieve multi-dimensional interaction between people and space. Currently, common spatial interaction methods include: gestures, eyes, voice and peripherals (such as handles, keyboards or mice, etc.).

[0003] Among them, gesture interaction can be roughly divided into direct gesture interaction and indirect gesture interaction according to the definition of its interaction with the spatial environment. Indirect gesture interaction is to lock an object or component in space through hand rays at a certain distance to obtain the focus of the object or component; then detect the movement of the user's hand or finger and convert it into a virtual gesture, so that the component can be quickly operated to achieve spatial interaction.

[0004] However, the above method has the following problems: when detecting the movement of the user's hand or finger, the hand may be blocked, causing the converted virtual gesture to deviate from the user's actual gesture, which in turn causes the interaction to deviate, reducing the user's experience. Summary of the invention

[0005] The present invention provides a virtual gesture control method, device, electronic device and computer-readable storage medium, and the method can solve one or more of the above-mentioned technical problems.

[0006] A first aspect of an embodiment of the present invention provides a method for controlling a virtual gesture, the method being applicable to a smart wearable device equipped with a panoramic camera, the method comprising:

[0007] When it is determined that the user has hand activities, calling the panoramic camera to obtain a real-time image set, wherein the real-time hand image set consists of a plurality of hand images related to the user's hand activities;

[0008] After extracting a plurality of hand images from the real-time image set, contour recognition is performed on each of the hand images to obtain a hand contour;

[0009] A reference gesture is generated using a plurality of the hand contours, and a virtual gesture movement is controlled according to the reference gesture.

[0010] In a possible implementation manner of the first aspect, performing contour recognition on each of the hand images to obtain the hand contour includes:

[0011] After identifying the wrist interval in each of the hand images, the hand image is segmented to obtain a segmented image;

[0012] After the segmented image is optimized to obtain an optimized image, contour extraction is performed on the optimized image to obtain a hand contour, wherein the optimization process includes: smoothing, denoising and anti-aliasing.

[0013] In a possible implementation manner of the first aspect, the step of generating a reference gesture using the plurality of hand contours includes:

[0014] If the plurality of hand images are images taken at the same time point and at different angles, then the contour similarities between the plurality of hand contours are calculated to obtain a plurality of hand similarity values;

[0015] If each of the hand similarity values ​​is greater than a preset similarity value, a hand similarity value with the largest value is selected from the plurality of hand similarity values ​​to obtain a target similarity value;

[0016] The two hand contours corresponding to the target similarity values ​​are used to perform three-dimensional modeling to obtain a reference gesture.

[0017] In a possible implementation manner of the first aspect, the generating a reference gesture using the plurality of hand contours further includes:

[0018] If any of the hand similarity values ​​is less than a preset similarity value, a hand contour corresponding to the hand similarity value less than the preset similarity value is obtained, and a three-dimensional modeling is performed on the hand contour corresponding to the hand similarity value less than the preset similarity value to obtain a plurality of gestures to be selected;

[0019] After obtaining the user's confirmation information, a corresponding gesture is filtered from the plurality of candidate gestures as a reference gesture according to the confirmation information, wherein the confirmation information is information in which the user filters any one of the candidate gestures and replies to confirm after displaying the plurality of candidate gestures.

[0020] In a possible implementation manner of the first aspect, the step of generating a reference gesture using the plurality of hand contours includes:

[0021] If the plurality of hand images are images taken continuously at the same angle, obtaining a shooting time sequence of the plurality of hand images;

[0022] Generating a movement trajectory of the user's hand according to the shooting time sequence and the plurality of hand contours;

[0023] According to the activity track, a corresponding gesture template is searched in preset gesture templates as a reference gesture.

[0024] In a possible implementation manner of the first aspect, controlling the movement of the virtual gesture according to the reference gesture includes:

[0025] Acquiring user movement information, wherein the movement information is information about a movable activity trajectory of a virtual object selected by the user;

[0026] After a virtual gesture is generated according to the shape of the reference gesture, the movement of the virtual gesture is controlled based on a trajectory of the movement information.

[0027] In a possible implementation manner of the first aspect, obtaining the user's movement information includes:

[0028] Obtaining the object coordinate point corresponding to the virtual object and the visual coordinate point corresponding to the user's visual focus;

[0029] The object coordinate point and the visual coordinate point are connected to obtain movement information.

[0030] A second aspect of an embodiment of the present invention provides a virtual gesture control device, the device being applicable to a smart wearable device equipped with a panoramic camera, the device comprising:

[0031] An image acquisition module, used for calling the panoramic camera to acquire a real-time image set when determining that the user has hand activities, wherein the real-time hand image set consists of a plurality of hand images related to the user's hand activities;

[0032] A contour recognition module, used for extracting a plurality of hand images from the real-time image set and performing contour recognition on each of the hand images to obtain a hand contour;

[0033] The gesture control module is used to generate a reference gesture using a plurality of the hand contours, and control the movement of the virtual gesture according to the reference gesture.

[0034] In a possible implementation manner of the second aspect, performing contour recognition on each of the hand images to obtain the hand contour includes:

[0035] After identifying the wrist interval in each of the hand images, the hand image is segmented to obtain a segmented image;

[0036] After the segmented image is optimized to obtain an optimized image, contour extraction is performed on the optimized image to obtain a hand contour, wherein the optimization process includes: smoothing, denoising and anti-aliasing.

[0037] In a possible implementation manner of the second aspect, the step of generating a reference gesture using the plurality of hand contours includes:

[0038] If the plurality of hand images are images taken at the same time point and at different angles, then the contour similarities between the plurality of hand contours are calculated to obtain a plurality of hand similarity values;

[0039] If each of the hand similarity values ​​is greater than a preset similarity value, a hand similarity value with the largest value is selected from the plurality of hand similarity values ​​to obtain a target similarity value;

[0040] The two hand contours corresponding to the target similarity values ​​are used to perform three-dimensional modeling to obtain a reference gesture.

[0041] In a possible implementation manner of the second aspect, the generating a reference gesture using the plurality of hand contours further includes:

[0042] If any of the hand similarity values ​​is less than a preset similarity value, a hand contour corresponding to the hand similarity value less than the preset similarity value is obtained, and a three-dimensional modeling is performed on the hand contour corresponding to the hand similarity value less than the preset similarity value to obtain a plurality of gestures to be selected;

[0043] After obtaining the user's confirmation information, a corresponding gesture is filtered from the plurality of candidate gestures as a reference gesture according to the confirmation information, wherein the confirmation information is information in which the user filters any one of the candidate gestures and replies to confirm after displaying the plurality of candidate gestures.

[0044] In a possible implementation manner of the second aspect, the step of generating a reference gesture using the plurality of hand contours includes:

[0045] If the plurality of hand images are images taken continuously at the same angle, obtaining a shooting time sequence of the plurality of hand images;

[0046] Generating a movement trajectory of the user's hand according to the shooting time sequence and the plurality of hand contours;

[0047] According to the activity track, a corresponding gesture template is searched in preset gesture templates as a reference gesture.

[0048] In a possible implementation manner of the second aspect, controlling the movement of the virtual gesture according to the reference gesture includes:

[0049] Acquiring user movement information, wherein the movement information is information about a movable activity trajectory of a virtual object selected by the user;

[0050] After a virtual gesture is generated according to the shape of the reference gesture, the movement of the virtual gesture is controlled based on a trajectory of the movement information.

[0051] In a possible implementation manner of the second aspect, obtaining the user's movement information includes:

[0052] Obtaining the object coordinate point corresponding to the virtual object and the visual coordinate point corresponding to the user's visual focus;

[0053] The object coordinate point and the visual coordinate point are connected to obtain movement information.

[0054] Compared with the prior art, the control method and device of a virtual gesture provided by the embodiment of the present invention has the following beneficial effects: when determining that the user has hand movement, the present invention can call a panoramic camera to obtain a hand image of the user's hand movement; after extracting the hand contour from the hand image, a reference gesture is generated using the hand contour and the virtual gesture movement is controlled according to the reference gesture. By combining the actual hand image to generate and control the virtual gesture, the deviation between the virtual gesture and the user's actual gesture can be reduced, thereby reducing the deviation of the virtual interactive movement, thereby improving the accuracy of the virtual movement and enhancing the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a virtual gesture control method provided by an embodiment of the present invention;

[0056] Figure 2 It is a structural schematic diagram of a virtual gesture control device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In order to solve the above problems, a virtual gesture control method provided by an embodiment of the present application will be introduced and explained in detail through the following specific embodiments.

[0059] Reference Figure 1 , which shows a flow chart of a virtual gesture control method provided by an embodiment of the present invention.

[0060] The virtual gesture control method can be applied to smart wearable devices, such as Apple Vision Pro, VR helmets or VR glasses, and the smart wearable devices are provided with a panoramic camera that can capture 360-degree images of the wearer and the surrounding area.

[0061] Optionally, multiple panoramic cameras may be provided. For example, if the smart wearable device is a VR helmet, a panoramic camera may be provided on the front and on the left and right sides of the VR helmet, for a total of three panoramic cameras.

[0062] In the spatial interaction of VR or AR, users can provide a more realistic and immersive interactive experience in the virtual environment, and realize natural interaction with the real world in the augmented reality environment.

[0063] In an actual application scenario, after the user wears the smart wearable device, he can move his hands in the real environment, and the smart wearable device can collect the user's hand movements in the real environment, generate a virtual gesture, and click, slide or grab virtual movements in the virtual environment. However, due to the deviation in detecting the user's hand movements in the real environment, the virtual movements in the virtual environment are also deviated, and then the operation in the virtual environment is wrong. In order to solve the above technical problems, as an example, the control method of the virtual gesture may include:

[0064] S11. When it is determined that the user has hand activities, the panoramic camera is called to obtain a real-time image set, where the real-time hand image set consists of multiple hand images about the user's hand activities.

[0065] In one embodiment, after the user wears the smart wearable device, his or her hand may or may not have smart accessories. If there are smart accessories (such as a handle), the sensor built into the handle can detect whether the user's handle has any activity. If there is activity, a trigger information can be generated and sent to the smart wearable device, so that the smart wearable device determines that the user has hand activity.

[0066] If no smart accessories are used, the smart wearable device can call on the panoramic camera to detect the user's hands in real time to determine whether the hands can move.

[0067] When there is an activity, the panoramic camera can be called to obtain images of the user's hands. Since the duration of the user's actions may be long or short, the hand image of the user's hand while continuously performing an action can be obtained, thereby obtaining multiple hand images, and then the multiple hand images are combined into an image set to obtain a real-time image set.

[0068] Referring to the above example, a panoramic camera is provided on the front and left and right sides of the VR helmet, for a total of three panoramic cameras. After determining that the user's hands are moving, the panoramic camera on the front can be called to shoot the image of the user's hands moving from a top-down angle. Similarly, the panoramic camera on the left can be called to shoot the image of the user's left hand moving from a top-down angle; and then the panoramic camera on the right can be called to shoot the image of the user's right hand moving from a top-down angle.

[0069] The hand images taken by the three panoramic cameras are then combined into a real-time image set, or the hand images taken by each panoramic camera are individually combined into a real-time image set.

[0070] S12: After extracting a number of hand images from the real-time image set, perform contour recognition on each of the hand images to obtain a hand contour.

[0071] In one embodiment, since there may be multiple panoramic cameras, the real-time image set captured contains multiple hand images. If each hand image is identified and processed, it will not only take a long time, but also be inefficient. The user's current action may have been completed and the next action needs to be performed, but the image analysis and processing has not been completed. In order to reduce the number of images to be processed, several hand images can be extracted from the real-time image set.

[0072] Optionally, the sampling method may be random sampling or interval sampling, for example, sampling one frame of image every one thousandth of a second. The specific sampling method may be adjusted according to actual needs.

[0073] After extracting several hand images, in order to accurately identify the specific gesture of the user performing the action, contour recognition can be performed on each hand image separately to identify the user's wrist, palm, fingers and other parts in the image, thereby obtaining the contour of the entire hand.

[0074] In order to accurately extract the hand contour, step S12 may include the following sub-steps as an example:

[0075] S121, after identifying the wrist interval in each of the hand images, segment the hand images to obtain segmented images.

[0076] S122. After optimizing the segmented image to obtain an optimized image, perform contour extraction on the optimized image to obtain a hand contour, wherein the optimization process includes smoothing, denoising and anti-aliasing.

[0077] Specifically, the hand image may include images of the user's wrist, fingers, and palm, etc., and may also include the environment where the user is located, and the surrounding environment may be the background of the image.

[0078] In an optional embodiment, a pre-trained model can be used to identify images based on color, texture, shape and other features to obtain the areas of the wrist, fingers and palm of the user's hand, and determine the position coordinates of these areas in the image. The image is segmented based on the position coordinates to obtain a segmented image of the wrist, fingers and palm.

[0079] In one embodiment, the present invention can determine the segmentation category through the portrait information, and intercept the corresponding entire human body area for further accurate segmentation.

[0080] Optionally, segmentation can also be performed based on other segmentation algorithms in deep learning, such as entity segmentation and panoptic segmentation.

[0081] After segmentation, the image may be biased, and the edges of the segmented images of the wrist, fingers, and palm may have small ripples. In order to eliminate the ripples, the image can be optimized.

[0082] In an optional embodiment, the optimization process may include: smoothing, denoising and anti-aliasing.

[0083] Since the segmented image may have various defects, the segmented image has different problems, which in turn leads to problems such as low clarity, blur, and many burrs in the subsequent image synthesis. In order to improve the segmented image.

[0084] In an optional embodiment, the smoothing process specifically includes: performing image smoothing process on the image using a preset Gaussian filter. The image clarity can be improved by the smoothing process, thereby improving the effect of subsequent recognition.

[0085] In one embodiment, the denoising process specifically includes: first determining the noise region of the image, and filling the color of the noise region with a corresponding color.

[0086] In actual operation, the noise block in the image can be filtered out first, and the part of the noise block can be filled with a color that matches the hand area (for example, the finger area is white, and the noise block can be filled with white). In this way, the local small block noise can be eliminated without affecting the image clarity, and the image integrity can be better maintained by locating and removing it in the corresponding area.

[0087] In one embodiment, the anti-aliasing process is specifically: performing a blur operator operation on the image to obtain a segmentation line region, binarizing the segmentation line region to obtain an image segmentation line, and then removing the aliasing of the image according to the image segmentation line.

[0088] Specifically, the image may be firstly blurred so that the wrist, finger and palm areas in the image may form a softer segmentation line contour area, and then the segmentation line area may be binarized so that the segmentation line of the contour area may produce a softer and smoother segmentation line.

[0089] The present invention aims at the problem of jagged edges of images, adds smoothing processing to the edges of images, and improves the accuracy of subsequent recognition.

[0090] After the optimization process is completed, the optimized image may be subjected to contour extraction. In order to accurately extract the portrait contour, in one embodiment, the contour extraction process is specifically as follows:

[0091] The amplitude and direction corresponding to the optimized image are calculated using first-order partial derivative finite difference.

[0092] The amplitude is subjected to non-maximum suppression and double-threshold algorithm detection to obtain the processed amplitude, and the directions are connected into contour edges according to the processed amplitude; the preset extraction algorithm is called to extract contour information from the contour edge according to the preset contour threshold, thereby obtaining the hand area.

[0093] Specifically, after using a Gaussian filter to smooth the image, the amplitude and direction of the gradient can be calculated using first-order partial derivative finite difference, and then the gradient amplitude is non-maximum suppressed and the double threshold algorithm is used to detect and connect edges.

[0094] Optionally, since the background of the image is relatively complex, the hand image is a relatively blurred area, and the contours of the wrist, fingers and palm areas are relatively weak, the size of the double threshold can be adjusted, and this threshold range can better extract weak edge contour information.

[0095] In an optional implementation, the Canny algorithm can be used for multi-level edge extraction. The Canny algorithm has a good effect in extracting multi-level strong and weak edge information and has a great advantage in running speed.

[0096] On this basis, R-Canny (Cyclic Canny Algorithm) can be used for processing, that is, based on the standard Canny algorithm, the idea of ​​cyclic operation is added to perform multi-level layer-by-layer edge extraction operations. Due to the use of the cyclic operation of extracting edges again on the edge image, the effect of weak edge extraction is further improved.

[0097] It should be noted that the above value ranges can be adjusted according to actual needs.

[0098] S13. Generate a reference gesture using a plurality of the hand contours, and control the movement of the virtual gesture according to the reference gesture.

[0099] After extracting the outline of the hand, three-dimensional modeling can be performed based on the outline and shape of the hand, so that a virtual gesture identical to the real gesture can be formed in the virtual environment and used as a reference gesture. Then, the virtual gesture in the virtual scene can be controlled to perform corresponding movements according to the reference gesture. For example, the user makes a gesture of four fingers combined and waves his arm. A reference gesture can be generated with the four fingers combined, and this gesture can be used as a virtual gesture to wave the arm in the virtual environment.

[0100] In one embodiment, when detecting the contour of the user's hand, various sections of the hand may be blocked or overlapped. If a virtual gesture is generated based on a handle image taken from a single angle, the gesture may deviate from the actual gesture.

[0101] In order to generate a reference gesture by combining the hand contours corresponding to the images taken at multiple angles, as an example, the generation of the reference gesture by using several hand contours may include the following sub-steps:

[0102] S21. If the plurality of hand images are images taken at the same time point and at different angles, calculate contour similarities between the plurality of hand contours to obtain a plurality of hand similarity values.

[0103] S22: If each of the hand similarity values ​​is greater than a preset similarity value, a hand similarity value with the largest value is selected from the plurality of hand similarity values ​​to obtain a target similarity value.

[0104] S23, performing three-dimensional modeling using the two hand contours corresponding to the target similarity values ​​to obtain a reference gesture.

[0105] Specifically, since there are multiple panoramic cameras, each panoramic camera collects multiple hand images, and each panoramic camera has a corresponding real-time image set, in order to extract the hand contour from multiple different angles and avoid gesture deviation due to occlusion, a hand image can be extracted from the real-time image set corresponding to each panoramic camera. Specifically, images at the same time node can be extracted.

[0106] In actual operation, if there are three panoramic cameras and the user only moves one hand, the central panoramic camera can shoot the user's moving hand from a top-down angle. A hand image at a certain time point can be obtained from the real-time image set of the central panoramic camera. At the same time, a hand image from the left side of the panoramic camera can be obtained at the same time point. A hand image from the right side of the panoramic camera can also be obtained. Finally, a reference gesture is generated by combining the three hand contours.

[0107] Similarly, if there are three panoramic cameras and the user moves both hands, the middle panoramic camera can take a hand image from top to bottom, and the hand image from the bird's-eye view can include both hands of the user; at the same time, the hand image of the left hand taken by the left panoramic camera can be obtained, and the hand image of the right hand taken by the right panoramic camera can be obtained. Finally, a reference gesture for the left hand is generated by combining the left hand contour of the hand image of the middle panoramic camera and the hand contour of the hand image of the left hand; similarly, a reference gesture for the right hand can be generated by combining the right hand contour of the hand image of the middle panoramic camera and the hand contour of the hand image of the right hand.

[0108] In the specific operation, in order to integrate the contours of each hand, the contour similarity between several hand contours can be calculated to obtain multiple hand similarity values. Although the shooting angles may be different, if the gestures are the same, the contours of the two gestures are roughly the same. Therefore, the contour similarity between the two can be calculated. The specific calculation method can be to call a preset neural network and calculate according to the image similarity method.

[0109] Next, each hand similarity value can be compared with a preset similarity value. If each hand similarity value is greater than the preset similarity value, it means that the hand contours of the images taken at each angle are similar. In order to reduce the amount of data to be processed, the hand similarity value with the largest value can be selected from multiple hand similarity values, and the hand similarity value with the largest value is used as the target similarity value.

[0110] Then, two hand contours corresponding to the target similarity values ​​are obtained, and three-dimensional modeling is performed using the two hand contours to finally obtain a reference gesture.

[0111] S24. If any of the hand similarity values ​​is smaller than a preset similarity value, a hand contour corresponding to the hand similarity value smaller than the preset similarity value is obtained, and a three-dimensional modeling is performed on the hand contour corresponding to the hand similarity value smaller than the preset similarity value to obtain a plurality of gestures to be selected.

[0112] S25. After obtaining the user's confirmation information, select a corresponding gesture from the plurality of gestures to be selected as a reference gesture according to the confirmation information, wherein the confirmation information is information in which the user selects any one of the gestures to be selected and replies to confirm after displaying the plurality of gestures to be selected.

[0113] Optionally, one or more hand similarity values ​​may be smaller than a preset similarity value, which indicates that the gestures taken at different angles are different. This may be due to occlusion of fingers or other occlusions, resulting in different gestures at different angles.

[0114] In order to avoid matching the actual operation or activity intention of the user, a hand similarity value smaller than a preset similarity value may be obtained, and then a hand contour corresponding to the hand similarity value smaller than the preset similarity value may be obtained.

[0115] If a hand similarity value is less than a preset similarity value, two hand contours corresponding to the hand similarity value are obtained, and then the two hand contours are used for three-dimensional modeling to obtain two gestures to be selected.

[0116] If there are multiple hand similarity values ​​less than the preset similarity value, two hand contours corresponding to each hand similarity value less than the preset similarity value are obtained, and then three-dimensional modeling is performed on each hand contour to obtain multiple gestures to be selected.

[0117] Then, multiple gestures to be selected can be displayed in the virtual world or virtual environment, and the user can directly view the multiple gestures to be selected, and then select a gesture to be selected, and reply a confirmation message, and finally use the gesture to be selected by the user as a reference gesture.

[0118] Optionally, if there are multiple gestures to be selected, and the gestures to be selected may be the same or similar to each other, the multiple gestures to be selected can be aggregated first. Specifically, they can be aggregated according to their shapes, so that the same gestures are aggregated together, and then the aggregated gestures to be selected are displayed. This can reduce the number of gestures that the user needs to view and improve processing efficiency.

[0119] In another optional embodiment, the user may only move one hand, and the gesture of the movement may be a set of actions, such as swinging the fingers up and down, or turning in circles. The series of actions may have changes in gestures, and it is difficult to accurately determine the user's actions and gestures through images taken at a single time node. In order to accurately identify gestures of dynamic actions, as an example, the generation of reference gestures using several hand contours may include the following sub-steps:

[0120] S31. If the plurality of hand images are images taken continuously at the same angle, obtain a shooting time sequence of the plurality of hand images.

[0121] S32: Generate a movement trajectory of the user's hand according to the shooting time sequence and the plurality of hand contours.

[0122] S33: searching for a corresponding gesture template in preset gesture templates according to the activity trajectory as a reference gesture.

[0123] In one embodiment, since the user gesture to be determined is a gesture of a series of operations and actions, a plurality of hand images may be acquired from a real-time image set, and the plurality of hand images are continuous.

[0124] Optionally, several hand images can also be extracted from the real-time image set according to a preset time interval. The extraction method can be to first determine the time of the user's action, for example, 1 second or 2 seconds.

[0125] The panoramic camera samples 100 frames per second. Assuming that the user's action is 1 second, image extraction can be performed at a preset time interval of 10 milliseconds. Then, 10 images can be extracted from the 100 frames to obtain 10 hand images. The shooting time sequence of these 10 hand images is then obtained and the 10 hand images are arranged according to the shooting time sequence.

[0126] Next, the hand contour of each hand image can be determined separately, and then multiple hand contours are played in the order of shooting time to form a dynamic gesture. Finally, the dynamic gesture is recognized to determine the specific area of ​​the user's activity, such as the finger area, wrist area or arm area.

[0127] Then determine the trajectory of the regional activity, thereby determining the activity trajectory of the user's hand.

[0128] Finally, the corresponding gesture template can be searched in the preset gesture templates according to the activity trajectory as a reference gesture.

[0129] It should be noted that there are multiple gestures in the preset gesture template, each gesture corresponds to an activity track, similar to the gesture setting of a mobile phone or smart terminal, two-finger rotation activity corresponds to a gesture template, single-finger swing corresponds to a gesture template, etc. Finally, the gesture corresponding to the gesture template can be used as a reference gesture.

[0130] After determining the user's gesture, in order to fit the user's actual activity intention, a response operation is performed in the virtual world, where, as an example, controlling the virtual gesture movement according to the reference gesture may include the following sub-steps:

[0131] S131. Obtain user movement information, where the movement information is information about a movable activity trajectory of a virtual object selected by the user.

[0132] In an application operation, the movement information may be the track information of a user moving a virtual object in the virtual world, or the operation track of the user when moving in the virtual world.

[0133] For example, in the virtual world of Apple Vision Pro, you can virtually design furniture or various household items. If you need to move a table from left to right in the virtual world, the movement information is from left to right.

[0134] In one embodiment, the movement of the virtual world may be the activity of the user's virtual character or the change of the user's viewpoint to determine the location of the activity. As an example, step S131 may include the following sub-steps:

[0135] S1311. Obtain the object coordinate point corresponding to the virtual object and the visual coordinate point corresponding to the user's visual focus.

[0136] S1312: Connect the object coordinate point and the visual coordinate point to obtain movement information.

[0137] Specifically, the coordinates of the virtual object in the virtual world can be obtained first to obtain the object coordinate point corresponding to the virtual object. Then, the coordinates corresponding to the user's visual focus in the virtual world can be obtained to obtain the visual coordinate point. Finally, the two coordinates can be connected to obtain the movement trajectory, thereby obtaining the movement information.

[0138] Optionally, after the coordinate points of two positions are determined, they may be connected together according to a certain trajectory to obtain movement information.

[0139] The certain trajectory can be a curve, up and down swing, etc., which can be adjusted according to actual needs.

[0140] S132: After generating a virtual gesture according to the shape of the reference gesture, control the movement of the virtual gesture based on the trajectory of the movement information.

[0141] Finally, a virtual gesture can be generated according to the shape of the reference gesture, and the movement of the virtual gesture can be controlled based on the trajectory of the movement information. In the virtual world, the virtual gesture can drive the object to be moved to move.

[0142] For example, in the above example, the table in the virtual world needs to be moved from the left to the right. The virtual gesture is a clenched fist gesture. You can clench your fist in the virtual world, and then the clenched fist can grab the table in the virtual world and move the virtual table from the left to the right.

[0143] In one embodiment, in a VR environment, after wearing the device for a long time, the user may experience dizziness, eye fatigue and other problems. In order to eliminate the user's physical discomfort, the method may further include:

[0144] Seamlessly integrate objects or different elements of the virtual world with the real environment to avoid interference or distress to users' cognition.

[0145] In addition, after the user has been active or worn for a period of time, a prompt message can be displayed to the user to remind the user that they need to take a corresponding rest.

[0146] In this embodiment, the embodiment of the present invention provides a method for controlling a virtual gesture, and its beneficial effect is that: when determining that the user has hand activity, the present invention can call a panoramic camera to obtain a hand image of the user's hand activity; after extracting the hand contour from the hand image, the hand contour is used to generate a reference gesture and the virtual gesture movement is controlled according to the reference gesture. By combining the actual hand image to generate and control the virtual gesture, the deviation between the virtual gesture and the user's actual gesture can be reduced, thereby reducing the action deviation of the virtual interaction, thereby improving the accuracy of the virtual action and improving the user's experience.

[0147] The embodiment of the present invention also provides a virtual gesture control device, see Figure 2 , showing a schematic structural diagram of a virtual gesture control device provided by an embodiment of the present invention.

[0148] Wherein, as an example, the control device of the virtual gesture may include:

[0149] The method is applicable to a smart wearable device equipped with a panoramic camera, and the method comprises:

[0150] The image acquisition module 201 is used to call the panoramic camera to acquire a real-time image set when it is determined that the user has hand activities, and the real-time hand image set consists of multiple hand images about the user's hand activities;

[0151] The contour recognition module 202 is used to extract a plurality of hand images from the real-time image set and then perform contour recognition on each of the hand images to obtain a hand contour;

[0152] The gesture control module 203 is used to generate a reference gesture using a plurality of the hand contours, and control the movement of the virtual gesture according to the reference gesture.

[0153] Optionally, performing contour recognition on each of the hand images to obtain a hand contour includes:

[0154] After identifying the wrist interval in each of the hand images, the hand image is segmented to obtain a segmented image;

[0155] After the segmented image is optimized to obtain an optimized image, contour extraction is performed on the optimized image to obtain a hand contour, wherein the optimization process includes: smoothing, denoising and anti-aliasing.

[0156] Optionally, the generating a reference gesture using a plurality of the hand contours includes:

[0157] If the plurality of hand images are images taken at the same time point and at different angles, then the contour similarities between the plurality of hand contours are calculated to obtain a plurality of hand similarity values;

[0158] If each of the hand similarity values ​​is greater than a preset similarity value, a hand similarity value with the largest value is selected from the plurality of hand similarity values ​​to obtain a target similarity value;

[0159] The two hand contours corresponding to the target similarity values ​​are used to perform three-dimensional modeling to obtain a reference gesture.

[0160] Optionally, the step of generating a reference gesture using a plurality of the hand contours further includes:

[0161] If any of the hand similarity values ​​is less than a preset similarity value, a hand contour corresponding to the hand similarity value less than the preset similarity value is obtained, and a three-dimensional modeling is performed on the hand contour corresponding to the hand similarity value less than the preset similarity value to obtain a plurality of gestures to be selected;

[0162] After obtaining the user's confirmation information, a corresponding gesture is filtered from the plurality of candidate gestures as a reference gesture according to the confirmation information, wherein the confirmation information is information in which the user filters any one of the candidate gestures and replies to confirm after displaying the plurality of candidate gestures.

[0163] Optionally, the generating a reference gesture using a plurality of the hand contours includes:

[0164] If the plurality of hand images are images taken continuously at the same angle, obtaining a shooting time sequence of the plurality of hand images;

[0165] Generating a movement trajectory of the user's hand according to the shooting time sequence and the plurality of hand contours;

[0166] According to the activity track, a corresponding gesture template is searched in preset gesture templates as a reference gesture.

[0167] Optionally, controlling the movement of the virtual gesture according to the reference gesture includes:

[0168] Acquiring user movement information, wherein the movement information is information about a movable activity trajectory of a virtual object selected by the user;

[0169] After a virtual gesture is generated according to the shape of the reference gesture, the movement of the virtual gesture is controlled based on a trajectory of the movement information.

[0170] Optionally, obtaining the user's mobile information includes:

[0171] Obtaining the object coordinate point corresponding to the virtual object and the visual coordinate point corresponding to the user's visual focus;

[0172] The object coordinate point and the visual coordinate point are connected to obtain movement information.

[0173] Those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0174] Furthermore, an embodiment of the present application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the virtual gesture control method as described in the above embodiment is implemented.

[0175] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer-executable program, and the computer-executable program is used to enable a computer to execute the virtual gesture control method as described in the above embodiment.

[0176] Those skilled in the art will appreciate that the embodiments of the present application may also provide computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes.

[0177] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), apparatuses, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0180] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A virtual gesture control method, characterized in that: The method is applicable to a smart wearable device equipped with a panoramic camera, and the method comprises: When it is determined that the user has hand activities, calling the panoramic camera to obtain a real-time image set, the real-time image set consisting of a plurality of hand images related to the user's hand activities; After extracting a plurality of hand images from the real-time image set, contour recognition is performed on each of the hand images to obtain a hand contour; A reference gesture is generated by using several of the hand contours, the user's movement information is obtained, a virtual gesture is generated according to the shape of the reference gesture, and the movement of the virtual gesture is controlled according to the movement information; the reference gesture is generated by using several of the hand contours, including: if the several hand images are images taken at the same time node and different angles, the contour similarities between several hand contours are calculated to obtain multiple hand similarity values; if each of the hand similarity values ​​is greater than a preset similarity value, the hand similarity value with the largest value is selected from the multiple hand similarity values ​​to obtain a target similarity value; and three-dimensional modeling is performed using two hand contours corresponding to the target similarity value to obtain a reference gesture.

2. The virtual gesture control method according to claim 1, characterized in that: The step of performing contour recognition on each of the hand images to obtain the hand contour comprises: After identifying the wrist interval in each of the hand images, the hand image is segmented to obtain a segmented image; After the segmented image is optimized to obtain an optimized image, contour extraction is performed on the optimized image to obtain a hand contour, wherein the optimization process includes: smoothing, denoising and anti-aliasing.

3. The virtual gesture control method according to claim 1, characterized in that: The step of generating a reference gesture by using a plurality of the hand contours further includes: If any of the hand similarity values ​​is less than a preset similarity value, a hand contour corresponding to the hand similarity value less than the preset similarity value is obtained, and a three-dimensional modeling is performed on the hand contour corresponding to the hand similarity value less than the preset similarity value to obtain a plurality of gestures to be selected; After obtaining the user's confirmation information, a corresponding gesture is filtered from the plurality of candidate gestures as a reference gesture according to the confirmation information, wherein the confirmation information is information in which the user filters any one of the candidate gestures and replies to confirm after displaying the plurality of candidate gestures.

4. The method for controlling a virtual gesture according to claim 1, characterized in that: The step of generating a reference gesture using a plurality of hand contours comprises: If the plurality of hand images are images taken continuously at the same angle, obtaining a shooting time sequence of the plurality of hand images; Generating a movement trajectory of the user's hand according to the shooting time sequence and the plurality of hand contours; According to the activity track, a corresponding gesture template is searched in preset gesture templates as a reference gesture.

5. The method for controlling a virtual gesture according to any one of claims 1 to 4, characterized in that: The controlling the virtual gesture movement according to the movement information comprises: The movement of the virtual gesture is controlled based on the trajectory of the movement information; the movement information is the trajectory information of the user moving the virtual object in the virtual world, or the operation trajectory of the user when moving in the virtual world.

6. The method for controlling a virtual gesture according to claim 1, characterized in that: The obtaining of the user's mobile information includes: Obtaining the object coordinate point corresponding to the virtual object and the visual coordinate point corresponding to the user's visual focus; The object coordinate point and the visual coordinate point are connected to obtain movement information.

7. A virtual gesture control device, characterized in that: The device is applicable to a smart wearable device equipped with a panoramic camera, and the device comprises: An image acquisition module, used to call the panoramic camera to acquire a real-time image set when it is determined that the user has hand activities, wherein the real-time image set is composed of a plurality of hand images related to the user's hand activities; A contour recognition module, used for extracting a plurality of hand images from the real-time image set and performing contour recognition on each of the hand images to obtain a hand contour; A gesture control module is used to generate a reference gesture using a plurality of the hand contours and obtain the user's movement information, generate a virtual gesture according to the shape of the reference gesture, and control the movement of the virtual gesture according to the movement information; the gesture control module is also used to: if the plurality of hand images are images taken at the same time node and at different angles, calculate the contour similarity between the plurality of hand contours to obtain a plurality of hand similarity values; if each of the hand similarity values ​​is greater than a preset similarity value, select the hand similarity value with the largest value from the plurality of hand similarity values ​​to obtain a target similarity value; and perform three-dimensional modeling using the two hand contours corresponding to the target similarity value to obtain a reference gesture.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for controlling a virtual gesture as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the virtual gesture control method according to any one of claims 1 to 6.