Speed determination method and device

Abstract

The embodiment of the invention provides a speed determination method and device, and relates to the technical field of intelligent equipment. The method is applied to the VR head-mounted display device, the VR head-mounted display device is provided with an image acquisition device, and the method comprises the following steps: performing grabbing gesture identification on each frame of image acquired by the image acquisition device; when the grabbing gesture is recognized for the first time, opening gesture recognition is carried out on each frame of image collected after the first current image; when the opening gesture is recognized for the first time, determining historical hand data within a specified time range; and determining the distribution condition of an identification result indicating that the hand position is identified in the historical hand data, and determining the throwing speed of the hand by using a determination method corresponding to a preset distribution condition. Based on this, by applying the scheme provided by the embodiment of the invention, when the user uses various applications of the VR head-mounted display device, when the real hand of the player makes the action of throwing the object, the throwing speed relatively close to the expectation of the player can be calculated.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent devices, and in particular to a speed determination method and device. Background Art

[0002] Currently, with the continuous development of smart devices, VR (Virtual Reality) head-mounted display devices are increasingly being used in users' daily entertainment, such as VR movie watching, VR games, etc.

[0003] Among them, currently, since gesture interaction is more natural and convenient than traditional handle interaction, users usually choose gesture interaction when using various applications of VR head display devices. For example, in VR games, the camera carried by the VR head display device can capture the player's real hand and display the virtual hand at the corresponding position in the game scene, and the movement and position of the virtual hand can change with the movement of the player's real hand, so that the player can complete various hand movements in the VR game more freely and easily, and enhance the player's gaming experience.

[0004] In some links of various applications of VR head-mounted display devices, users need to make grabbing and throwing actions, that is, users need to simulate the action of grabbing a virtual object in the application scene with their hands and throwing the virtual object with force, so as to realize the scene effect of the user throwing the virtual object to a certain object in the application scene. For example, in a shooting game, players can throw weapons to teammates. At this time, the player's real hand is needed to simulate the action of grabbing the weapon and throwing it to teammates, that is, to make a grabbing and throwing action. In order to provide users with a better gaming experience, when the user makes a grabbing and throwing action, it is usually necessary to determine the throwing speed of the user's real hand, and assign the throwing speed to the initial velocity of the virtual object thrown by the virtual hand in the application scene, so as to facilitate the subsequent related processing of the virtual object, for example, using the above initial velocity to predict the trajectory of the virtual object.

[0005] Based on this, when users use VR head-mounted display devices for various applications and their real hands make grabbing and throwing movements, how to determine the throwing speed of the hands so that the determined throwing speed is closer to the actual throwing speed of the above-mentioned grabbing and throwing movements is a problem that needs to be solved urgently. Summary of the invention

[0006] The purpose of the embodiment of the present invention is to provide a speed determination method and device, so as to achieve the determination of the throwing speed of the hand when the player's real hand performs the action of throwing an object in the process of using various applications of VR head display devices by the user, so that the determined throwing speed is closer to the actual throwing speed of the above-mentioned grab and throw action. The specific technical solution is as follows:

[0007] In a first aspect of the present invention, a speed determination method is provided, which is applied to a VR head display device, wherein the VR head display device is equipped with an image acquisition device; the method comprises:

[0008] Performing grasping gesture recognition on each frame of image captured by the image acquisition device; wherein the posture of each finger in the grasping gesture conforms to a first preset posture;

[0009] When the grasping gesture is recognized for the first time, an open gesture recognition is performed on each frame image collected after the first current image; wherein the first current image is: an image in which the grasping gesture is recognized for the first time, and the posture of each finger in the open gesture conforms to the second preset posture;

[0010] When the opening gesture is recognized for the first time, historical hand data within a specified time range is determined; wherein the historical hand data indicates each recognition moment of hand position recognition for each frame image according to a preset recognition frequency, and a recognition result at each recognition moment, the recognition result indicating whether the hand position is recognized, the hand position being the image position of at least part of the hand area present in each frame image; the starting moment of the specified time range is not earlier than the acquisition moment of the first current image and the ending moment of the specified time range is not later than the acquisition moment of the second current image, the second current image being: the image at which the opening gesture is recognized for the first time after the grasping gesture is recognized;

[0011] The distribution of the recognition results indicating the hand position in the historical hand data is determined, and the throwing speed of the hand is determined by using a preset determination method corresponding to the distribution.

[0012] Optionally, in a specific implementation, before determining the historical hand data within a specified time range, the method further includes:

[0013] Determine a specified time range with the acquisition time of the first current image as the starting time and the acquisition time of the second current image as the ending time; or,

[0014] A specified time range is determined, with the capture time of the second current image as the end time and the duration being a preset duration.

[0015] Optionally, in a specific implementation, the distribution is: in the historical hand data, the recognition results at each recognition moment all indicate that the hand position is recognized;

[0016] The method of determining the throwing speed of the hand by using the preset determination method corresponding to the distribution condition includes:

[0017] Calculate each candidate speed by using the position difference of the hand positions recognized at each two adjacent recognition moments in the historical hand data and the time difference between the two adjacent recognition moments;

[0018] The maximum speed among the candidate speeds is selected as the throwing speed of the hand.

[0019] Optionally, in a specific implementation, the distribution is as follows: in the historical hand data, the content of the recognition results at each recognition moment does not uniformly indicate that the hand position is recognized, and in the historical hand data, the recognition results at the recognition moments of the first specified number of digits are arranged in order from earliest to latest, including the recognition results indicating that the hand position is recognized;

[0020] The method of determining the throwing speed of the hand by using the preset determination method corresponding to the distribution condition includes:

[0021] The moving speed of the hand is calculated as the first speed by using the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment; wherein the first recognition moment is: the last recognition moment at which the recognition result indicates that the hand position is recognized, and the second recognition moment is: the recognition result indicates that the hand position is recognized, the recognition moment that is closest to the first recognition moment and earlier than the first recognition moment;

[0022] Based on the first speed, a throwing speed of the hand is determined.

[0023] Optionally, in a specific implementation, determining the throwing speed of the hand based on the first speed includes:

[0024] Calculating the product of the first speed and a preset empirical coefficient as a first reference speed;

[0025] Based on the first reference speed, the throwing speed of the hand is determined.

[0026] Optionally, in a specific implementation, before determining the throwing speed of the hand based on the first reference speed, the method further includes:

[0027] Calculate the hand movement speeds as the second speeds using the position difference of the hand position between two adjacent recognition moments and the time difference between the two adjacent recognition moments at each recognition moment when the recognition result indicates the hand position, and select the maximum second speed as the second reference speed;

[0028] The step of determining the throwing speed of the hand based on the first reference speed includes:

[0029] The maximum speed between the first reference speed and the second reference speed is selected as the throwing speed of the hand.

[0030] Optionally, in a specific implementation, the distribution is: in the historical hand data, in order from early to late, the recognition results of the recognition moments arranged in the first specified number of digits all indicate that the hand position is not recognized;

[0031] The method of determining the throwing speed of the hand by using the preset determination method corresponding to the distribution condition includes:

[0032] The preset rate is determined as the rate of the hand throwing speed, and the user's line of sight direction is determined as the direction of the hand throwing speed, so as to obtain the hand throwing speed.

[0033] In a second aspect of an embodiment of the present invention, a speed determination device is further provided, which is applied to a VR head display device, wherein the VR head display device is equipped with an image acquisition device; the device comprises:

[0034] A first recognition module, used for performing grasping gesture recognition on each frame of image captured by the image acquisition device; wherein the posture of each finger in the grasping gesture conforms to a first preset posture;

[0035] A second recognition module is used to perform an open gesture recognition on each frame image collected after the first current image when the grasping gesture is recognized for the first time; wherein the first current image is: an image in which the grasping gesture is recognized for the first time, and the posture of each finger in the open gesture conforms to the second preset posture;

[0036] A data determination module, for determining historical hand data within a specified time range when the opening gesture is first recognized; wherein the historical hand data indicates each recognition moment of hand position recognition for each frame image according to a preset recognition frequency, and a recognition result at each recognition moment, wherein the recognition result indicates whether the hand position is recognized, and the hand position is the image position of at least part of the hand area existing in each frame image; the starting moment of the specified time range is not earlier than the acquisition moment of the first current image and the ending moment of the specified time range is not later than the acquisition moment of the second current image, and the second current image is: the image at which the opening gesture is first recognized after the grasping gesture is recognized;

[0037] The speed determination module is used to determine the distribution of the recognition results indicating the hand position in the historical hand data, and determine the throwing speed of the hand using a preset determination method corresponding to the distribution.

[0038] Optionally, in a specific implementation, the data determination module is further used to:

[0039] Before determining the historical hand data within the specified time range, determining a specified time range with the acquisition time of the first current image as the starting time and the acquisition time of the second current image as the ending time; or,

[0040] Before determining the historical hand data within the specified time range, a specified time range is determined with the acquisition time of the second current image as the end time and the duration being a preset duration.

[0041] Optionally, in a specific implementation, the distribution is: in the historical hand data, the recognition results at each recognition moment indicate that the hand position is recognized; and the speed determination module is specifically used to:

[0042] Calculate each candidate speed by using the position difference of the hand positions recognized at each two adjacent recognition moments in the historical hand data and the time difference between the two adjacent recognition moments;

[0043] The maximum speed among the candidate speeds is selected as the throwing speed of the hand.

[0044] Optionally, in a specific implementation, the distribution is as follows: in the historical hand data, the content of the recognition results at each recognition moment does not uniformly indicate that the hand position is recognized, and in the historical hand data, the recognition results at the recognition moments of the first specified number of digits are arranged in order from early to late, including the recognition results indicating that the hand position is recognized; the speed determination module includes:

[0045] The speed calculation submodule calculates the moving speed of the hand as the first speed by using the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment; wherein the first recognition moment is: the last recognition moment at which the recognition result indicates that the hand position is recognized, and the second recognition moment is: the recognition result indicates that the hand position is recognized, the recognition moment closest to the first recognition moment and earlier than the first recognition moment;

[0046] The speed determination submodule is used to determine the throwing speed of the hand based on the first speed.

[0047] Optionally, in a specific implementation, the speed determination submodule includes:

[0048] a speed calculation unit, used to calculate the product of the first speed and a preset empirical coefficient as a first reference speed;

[0049] The speed determination unit is used to determine the throwing speed of the hand based on the first reference speed.

[0050] Optionally, in a specific implementation, the device further includes:

[0051] A speed selection module, used for calculating each moving speed of the hand as each second speed by using the position difference of the hand position between each two adjacent recognition moments at each recognition moment when the hand position is recognized indicated by the recognition result and the time difference between the two adjacent recognition moments before determining the throwing speed of the hand based on the first reference speed, and selecting the maximum second speed as the second reference speed;

[0052] The speed determination unit is specifically used for:

[0053] The maximum speed between the first reference speed and the second reference speed is selected as the throwing speed of the hand.

[0054] Optionally, in a specific implementation, the distribution is: in the historical hand data, in the order of time from early to late, the recognition results of the recognition moments arranged in the first specified number of digits all indicate that the hand position is not recognized; the speed determination module is specifically used to:

[0055] The preset rate is determined as the rate of the hand throwing speed, and the user's line of sight direction is determined as the direction of the hand throwing speed, so as to obtain the hand throwing speed.

[0056] In another aspect of the present invention, a VR head display device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0057] Memory, used to store computer programs;

[0058] The processor is used to implement the method steps of any of the speed determination methods provided in the first aspect of the present invention when executing the program stored in the memory.

[0059] In another aspect of the implementation of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the method steps of any speed determination method provided in the first aspect of the implementation of the present invention are implemented.

[0060] In another aspect of the implementation of the present invention, a computer program product containing instructions is also provided. When the computer program product is run on a computer, the computer executes the method steps of any of the speed determination methods provided in the first aspect of the implementation of the present invention.

[0061] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, in the process of users using various applications of VR head display devices, since the user's hands will experience the process of grasping objects, moving quickly and opening and throwing when performing grabbing and throwing actions, the process of the user performing a grabbing and throwing action can be identified by identifying the grabbing gesture and opening gesture in each frame image captured by the image acquisition device carried by the VR head display device. Among them, when the grabbing gesture is first recognized, it can be regarded as the user starting to perform the grabbing and throwing action, and after the grabbing gesture is first recognized, when the opening gesture is first recognized, it can be regarded as the user completing the grabbing and throwing action. In addition, in the above-mentioned gesture recognition process, the hand position recognition can be performed on each frame image captured by the above-mentioned image acquisition device according to the preset recognition frequency, that is, it can be identified whether there is at least a part of the user's hand area in the image and the image position of the existing hand area is identified. In this way, the method for determining the distribution of the hand positions in the recognition results recorded within the specified time range between the acquisition time of the image of the first recognized posture and the acquisition time of the image of the first recognized open posture can be used to determine the throwing speed of the user performing the grab and throw action, that is, to obtain the throwing speed of the hand. Furthermore, the throwing speed can be assigned to the initial speed of the virtual object thrown by the virtual hand in the application scene of the VR head display device, that is, the throwing speed is determined as the initial speed of the virtual object thrown by the virtual hand in the application scene of the VR head display device, so as to facilitate the subsequent related processing of the virtual object, for example, using the initial speed to predict the trajectory of the virtual object.

[0062] Based on this, the scheme provided by the embodiment of the present invention is applied to determine the throwing speed of the user's hand, and considering that the movement speed of the hand of the user will experience a process of first accelerating and then decelerating when performing the grabbing and throwing action, therefore, compared with directly using the position difference between the hand position when the user starts to perform the grabbing and throwing action and the hand position when the user completes the grabbing and throwing action, dividing it by the time difference between the moment when the user starts to perform the grabbing and throwing action and the moment when the user completes the grabbing and throwing action to obtain the throwing speed of the hand, in the embodiment of the present invention, the throwing speed is determined by using the recognition results of the hand position at each recognition moment in the process of the user performing the grabbing and throwing action, which can more fully consider the influence of the acceleration process and deceleration process of the hand on the throwing speed during the execution of the grabbing and throwing action, thereby making the determined throwing speed closer to the actual throwing speed of the user's real hand when performing the grabbing and throwing action.

[0063] That is to say, by applying the solution provided by the embodiment of the present invention, when the user's real hand performs the action of throwing an object, the determined throwing speed can be made closer to the real throwing speed of the user's real hand when performing the grab and throw action, thereby achieving a more natural and more user-expected interaction effect between the user and the VR head display device. In addition, for applications of VR head display devices such as games that originally wanted to use the grab and throw operation but were not developed smoothly or even stagnant due to the unstable throwing feedback, developers can adopt the solution of the embodiment of the present invention to solve the problem of unstable throwing feedback and achieve a more natural interaction between the user and the VR head display device. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0065] Figure 1 Schematic diagram of the flow of the first speed determination method in an embodiment of the present invention.

[0066] Figure 2 A schematic diagram of the movement trajectory of a user's hand when performing a grabbing and throwing action while using a VR head-mounted display device.

[0067] Figure 3 Schematic diagram of a flow chart of a second speed determination method in an embodiment of the present invention.

[0068] Figure 4 4 is a flow chart of a third speed determination method in an embodiment of the present invention.

[0069] Figure 5 4 is a flow chart of a fourth speed determination method in an embodiment of the present invention.

[0070] Figure 6 4 is a flow chart of a fifth speed determination method in an embodiment of the present invention.

[0071] Figure 7 4 is a flow chart of a sixth speed determination method in an embodiment of the present invention.

[0072] Figure 8 The present invention provides a flowchart of a specific example.

[0073] Fig. 9 A schematic structural diagram of a speed determination device provided by an embodiment of the present invention.

[0074] Fig.10 A schematic structural diagram of a VR head display device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0076] At present, when the user uses various applications of VR head display devices, the user's real hand makes a grabbing and throwing action. Generally, the technical status of the related technology is that when the user's hand makes a grabbing and throwing action, the VR head display device always detects whether the user's gesture in the image captured by the image acquisition device it carries is an open gesture, and then uses the change in the position of the user's hand before and after the hand is opened, divided by the time before and after the user's hand is opened, to calculate the speed of the hand. However, when the user performs the action of opening his hand and throwing an object, the user's hand moves very fast, and due to the hardware limitations of the above-mentioned image acquisition device, it is difficult to obtain the position of the hand stably. Therefore, it often happens that when the user's hand makes a grabbing and throwing action, the position of the user's hand can only be detected when the throwing starts (at this time, the user's hand just starts to accelerate) and ends (at this time, the speed of the user's hand gradually decreases), resulting in the inability to calculate the throwing speed that meets the user's expectations. Based on this, when the user uses various applications of VR head display devices, the user's real hand makes a grabbing and throwing action, how to determine a throwing speed that is closer to the player's expectations is a problem that needs to be solved urgently.

[0077] In order to solve the above technical problem, an embodiment of the present invention provides a speed determination method.

[0078] The speed determination method can be applied to various application scenarios of VR head display devices that need to determine the throwing speed when the user makes a throwing action, for example, in a shooting game, determining the throwing speed when the user simulates shooting; in a gunfight game, determining the throwing speed when the user throws a weapon to a teammate, etc. The embodiment of the present invention does not limit the specific application scenario of the speed determination method.

[0079] Furthermore, the execution subject of the speed determination method may be various VR head display devices, such as external head display, integrated head display, mobile phone box head display, etc., and VR head display devices may also be called VR helmets, VR glasses, etc. Among them, various VR head display devices may be provided with processing modules with data processing capabilities such as processors and controllers, so that various VR head display devices can use the above processing modules to execute the speed determination method, hereinafter referred to as VR head display devices.

[0080] In addition, in the speed determination method, the VR head display device as the execution subject is equipped with an image acquisition device, for example, a camera. In this way, during the use of the VR head display device, when the user interacts with the VR head display device, the user's hand may appear within the field of view of the image acquisition device, so that the image acquisition device can acquire an image including the user's hand.

[0081] A speed determination method provided by an embodiment of the present invention may include the following steps:

[0082] Performing grasping gesture recognition on each frame of image captured by the image acquisition device; wherein the posture of each finger in the grasping gesture conforms to a first preset posture;

[0083] When the grasping gesture is recognized for the first time, an open gesture recognition is performed on each frame image collected after the first current image; wherein the first current image is: an image in which the grasping gesture is recognized for the first time, and the posture of each finger in the open gesture conforms to the second preset posture;

[0084] When the opening gesture is recognized for the first time, historical hand data within a specified time range is determined; wherein the historical hand data indicates each recognition moment of hand position recognition for each frame image according to a preset recognition frequency, and a recognition result at each recognition moment, the recognition result indicating whether the hand position is recognized, the hand position being the image position of at least part of the hand area present in each frame image; the starting moment of the specified time range is not earlier than the acquisition moment of the first current image and the ending moment of the specified time range is not later than the acquisition moment of the second current image, the second current image being: the image at which the opening gesture is recognized for the first time after the grasping gesture is recognized;

[0085] The distribution of the recognition results indicating the hand position in the historical hand data is determined, and the throwing speed of the hand is determined by using a preset determination method corresponding to the distribution.

[0086] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, in the process of users using various applications of VR head display devices, since the user's hands will experience the process of grasping objects, moving quickly and opening and throwing when performing grabbing and throwing actions, the process of the user performing a grabbing and throwing action can be identified by identifying the grabbing gesture and opening gesture in each frame image captured by the image acquisition device carried by the VR head display device. Among them, when the grabbing gesture is first recognized, it can be regarded as the user starting to perform the grabbing and throwing action, and after the grabbing gesture is first recognized, when the opening gesture is first recognized, it can be regarded as the user completing the grabbing and throwing action. In addition, in the above-mentioned gesture recognition process, the hand position recognition can be performed on each frame image captured by the above-mentioned image acquisition device according to the preset recognition frequency, that is, it can be identified whether there is at least a part of the user's hand area in the image and the image position of the existing hand area is identified. In this way, the method for determining the distribution of the hand positions in the recognition results recorded within the specified time range between the acquisition time of the image of the first recognized posture and the acquisition time of the image of the first recognized open posture can be used to determine the throwing speed of the user performing the grab and throw action, that is, to obtain the throwing speed of the hand. Furthermore, the throwing speed can be assigned to the initial speed of the virtual object thrown by the virtual hand in the application scene of the VR head display device, that is, the throwing speed is determined as the initial speed of the virtual object thrown by the virtual hand in the application scene of the VR head display device, so as to facilitate the subsequent related processing of the virtual object, for example, using the initial speed to predict the trajectory of the virtual object.

[0087] Based on this, the scheme provided by the embodiment of the present invention is applied to determine the throwing speed of the user's hand, and considering that the movement speed of the hand of the user will experience a process of first accelerating and then decelerating when performing the grabbing and throwing action, therefore, compared with directly using the position difference between the hand position when the user starts to perform the grabbing and throwing action and the hand position when the user completes the grabbing and throwing action, dividing it by the time difference between the moment when the user starts to perform the grabbing and throwing action and the moment when the user completes the grabbing and throwing action to obtain the throwing speed of the hand, in the embodiment of the present invention, the throwing speed is determined by using the recognition results of the hand position at each recognition moment in the process of the user performing the grabbing and throwing action, which can more fully consider the influence of the acceleration process and deceleration process of the hand on the throwing speed during the execution of the grabbing and throwing action, thereby making the determined throwing speed closer to the actual throwing speed of the user's real hand when performing the grabbing and throwing action.

[0088] That is to say, by applying the solution provided by the embodiment of the present invention, when the user's real hand performs the action of throwing an object, the determined throwing speed can be made closer to the real throwing speed of the user's real hand when performing the grab and throw action, thereby achieving a more natural and more user-expected interaction effect between the user and the VR head display device. In addition, for applications of VR head display devices such as games that originally wanted to use the grab and throw operation but were not developed smoothly or even stagnant due to the unstable throwing feedback, developers can adopt the solution of the embodiment of the present invention to solve the problem of unstable throwing feedback and achieve a more natural interaction between the user and the VR head display device.

[0089] A speed determination method provided by an embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0090] Figure 1 A flow chart of a speed determination method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method may include the following steps S101-S105.

[0091] S101: performing grasping gesture recognition on each frame of image captured by an image acquisition device.

[0092] Among them, the posture of each finger in the above grasping gesture conforms to the first preset posture.

[0093] Usually, when a user's hand performs a grabbing and throwing action, it will go through three stages: grabbing an object, moving quickly, and opening and throwing. That is, in the process of the user's hand performing the grabbing and throwing action, the user's hand first holds the object, and the user's hand begins to enter the grabbing gesture; then, the user's hand continues to hold the object and moves quickly. During this process, the user's hand is always in the grabbing gesture, and the movement speed of the user's hand is in an accelerated state; then, the user's hand opens to throw the object, and the user's hand enters the opening gesture. At this time, the grabbing and throwing action is completed; then, the user's hand enters the deceleration state until it stops moving or performs the next action. Obviously, during the execution of a grabbing and throwing action, the user's hand experiences a change from a grabbing gesture to an opening gesture, and it first accelerates and then decelerates. When the user's hand begins to be in the grabbing gesture, it can be regarded as starting to perform the grabbing and throwing action. When the user's hand changes from the grabbing gesture to the opening gesture, it can be regarded as stopping the grabbing and throwing action, that is, the grabbing and throwing action is completed at this moment.

[0094] For example, Figure 2 This is a schematic diagram of the movement trajectory of the user's hand when performing a grab and throw action while using a VR head display device. Figure 2As shown, user 201 is wearing a VR head display device 202; the dotted parabola 203 is the movement trajectory of the hand of user 201 during a grabbing and throwing action, and the movement direction of the hand of user 201 is from top to bottom, and the hand of user 201 is in the process of executing the grabbing and throwing action, and is in the open gesture after throwing the grasped object. Among them, the formation process of the dotted parabola 203 is the movement process of the hand of user 201 in executing the grabbing and throwing action, and the speed of each point on the dotted parabola 203 is the movement speed of the hand of user 201 when it moves to each point in the process of executing the grabbing and throwing action. Exemplarily, Figure 2 The speed V indicated by the arrow in is the moving speed of the user 201's hand when it moves to a certain point on the dotted parabola 203. Thus, from top to bottom, the speed of each point on the dotted parabola 203 increases first and then decreases. Figure 2 For example, in an embodiment of the present invention, by utilizing the frames of images of the hand of the user 201 captured by the image acquisition device carried by the VR head display device 202 during the formation process of the above-mentioned dotted parabola 203, a throwing speed that is as close as possible to the actual speed of the point on the dotted parabola 203 where the hand of the user 201 is when throwing the object is determined as the throwing speed of the hand.

[0095] That is to say, when determining the throwing speed of the user's hand performing a grabbing and throwing action, the process of the user's hand performing a grabbing and throwing action can be first identified. Among them, the grasping gesture recognition is first performed. When the grasping gesture is first recognized, it can be regarded as the beginning of the user's hand performing a grabbing and throwing action. Then, the opening gesture recognition continues. When the opening gesture is first recognized, it can be regarded as the end of the grabbing and throwing action performed by the user's hand this time. Therefore, the process from the first recognition of the grasping gesture to the first recognition of the opening gesture is the process of the user's hand performing a grabbing and throwing action.

[0096] Based on this, in an embodiment of the present invention, the VR head display device is equipped with an image acquisition device such as a camera. Therefore, during the interaction between the user and the VR head display device, the image acquisition device carried by the VR head display device can continuously acquire images, so that the VR head display device can obtain each frame of images acquired by the above-mentioned image acquisition device. During the above-mentioned interaction process, when the user's hand appears in the field of view of the above-mentioned image acquisition device, the user's hand can exist in the image acquired by the above-mentioned image acquisition device, so that the VR head display device can perform gesture recognition on the user's hand appearing in the above-mentioned image.

[0097] Among them, considering that the user's hand can assume various different hand postures when grasping, for example, making a fist, making a semi-fist, bending each finger with the fingertips aligned, etc. Therefore, in the embodiments of the present invention, according to factors such as the shape, type, and material of the virtual object that the user needs to grasp in the actual application scenario, the hand posture of the grasping gesture to be recognized above can be preset as the first preset posture. Therefore, in the embodiments of the present invention, the postures of each finger in the above grasping gesture conform to the first preset posture.

[0098] For example, the distance between the fingertips of each finger can be set as the posture of each finger in the above grasping gesture. Exemplarily, the above first preset posture can be: the distance between the fingertip of the index finger and the fingertip of the thumb of the same hand is less than a certain set distance. Thus, when the distance between the fingertip of the index finger and the fingertip of the thumb of the user's same hand in the image collected by the image acquisition device carried by the above VR headset device is less than this set distance, it can be determined that the above grasping gesture is recognized.

[0099] Correspondingly, corresponding to the hand postures of different grasping gestures, when the user's hand is open, it can also assume various different hand postures, for example, all fingers are straightened, the fingers are slightly bent but the palm is flattened, etc. Therefore, in the embodiments of the present invention, similarly, according to factors such as the shape, type, and material of the virtual object that the user needs to grasp in the actual application scenario, the hand posture of the opening gesture to be recognized above can be preset as the second preset posture. Therefore, in the embodiments of the present invention, the postures of each finger in the above opening gesture conform to the second preset posture.

[0100] For example, the distance between the fingertips of each finger can also be set as the posture of each finger in the above opening gesture. Exemplarily, the above second preset posture can be: the distance between the fingertip of the index finger and the fingertip of the thumb of the same hand is not less than a certain set distance. Thus, when the distance between the fingertip of the index finger and the fingertip of the thumb of the user's same hand in the image collected by the image acquisition device carried by the above VR headset device is not less than this set distance, it can be determined that the above opening gesture is recognized.

[0101] Generally, in the actual application scenario, it can be considered that the user's gesture is in a grasping posture or an opening posture. Therefore, after presetting the above first preset posture, the gestures of the user that do not conform to the above first preset posture can all be recognized as opening gestures.

[0102] Based on this, optionally, the first preset posture can be preset as: the distance between the fingertip of the index finger and the fingertip of the thumb of the same hand is less than the specified distance, and the second preset posture can be preset as: the distance between the fingertip of the index finger and the fingertip of the thumb of the same hand is not less than the above specified distance.

[0103] In this way, when performing gesture recognition on each frame of images captured by the image acquisition device carried by the VR head display device, the VR head display device can first perform grasping gesture recognition on each frame of images captured by the image acquisition device. Moreover, when the grasping gesture is first recognized, it can be regarded as the user's hand starting to grasp the object, and thus, it can be regarded as the user's hand starting to perform the grasping action.

[0104] Among them, the VR head display device recognizes the grasping gesture of each frame image in turn and in real time according to the acquisition order of the above-mentioned frame images from first to last, and for the sake of clarity of subsequent description, the image in which the grasping gesture is first recognized can be referred to as the first current image. For example, the VR head display device does not recognize the grasping gesture in the first frame image and the second frame image, but recognizes the grasping gesture in the third frame image, then the third frame image can be used as the first current image. In addition, the VR head display device can use various general image recognition methods to recognize the grasping gesture of the above-mentioned frame images, for example, using a neural network model to recognize the grasping gesture, etc., and the embodiments of the present invention do not make specific limitations on this.

[0105] S102: When the grasping gesture is recognized for the first time, an opening gesture recognition is performed on each frame image collected after the first current image.

[0106] The first current image is an image in which the grasping gesture is recognized for the first time, and the postures of the fingers in the opening gesture conform to the second preset posture.

[0107] As mentioned above, after the grasping gesture is first recognized, the VR headset device can continue to perform gesture recognition on subsequent images. Since the user's hand is in the process of grasping an object and moving quickly, the grasping gesture can continue to be recognized after the grasping gesture is first recognized. When the opening gesture is first recognized, it can be regarded as the user's hand changing from a grasping gesture to an opening gesture, throwing the grasped object, and thus, it can be regarded as the user's hand completing a grasping and throwing action.

[0108] Based on this, after the grasping gesture is recognized for the first time as mentioned above, VR recognition can perform opening gesture recognition on each frame image collected after the first current image.

[0109] Among them, similar to the grasping gesture described above, the VR head display device also recognizes the opening gesture for each frame image collected after the above-mentioned first current image in sequence and in real time according to the acquisition order of the above-mentioned frame images from first to last, and for the sake of clarity of subsequent description, the image in which the opening gesture is first recognized can be referred to as the second current image. For example, the VR head display device recognizes the grasping gesture for the first time in the third frame image, and recognizes the opening gesture for the first time in the twentieth frame image, then the twentieth frame image can be used as the second current image. In addition, the VR head display device can use various general image recognition methods to recognize the opening gesture for each of the above-mentioned frame images, for example, using a neural network model to recognize the opening gesture, etc., and the embodiments of the present invention do not make specific limitations on this.

[0110] Considering that the user can perform the grabbing and throwing action multiple times when using the VR head display device, when the above-mentioned opening gesture is recognized and it is considered that the user's hand has completed a grabbing and throwing action, the recognition process of the next grabbing and throwing action can be entered, that is, starting from the next frame image of the above-mentioned second current image, returning to the above-mentioned step S101, and performing the grasping gesture recognition on each frame image captured by the image acquisition device again until the user stops performing the grasping action. In this way, the throwing speed of the hand during each grabbing and throwing action performed by the user when using the VR head display device can be determined in a sequential cycle.

[0111] S103: When the open gesture is recognized for the first time, determine historical hand data within a specified time range.

[0112] Among them, the historical hand data indicates the various recognition moments of hand position recognition for each frame image according to the preset recognition frequency, and the recognition result of each recognition moment, the recognition result indicates whether the hand position is recognized, and the hand position is the image position of at least part of the hand area in each frame image; the starting time of the specified time range is not earlier than the acquisition time of the first current image, and the end time of the specified time range is not later than the acquisition time of the second current image; the second current image is: the image in which the opening gesture is recognized for the first time after the grasping gesture is recognized.

[0113] As mentioned above, during the process of the user using the VR head display device, the image acquisition device carried by the VR head display device can continuously acquire images, and during the above-mentioned use process, the VR head display device can perform hand position recognition on each frame image acquired by the above-mentioned image acquisition device according to the preset recognition frequency, and record the obtained hand data, and the above-mentioned hand data can record the recognition time and the recognition result obtained by performing hand position recognition on the image at the recognition time, that is, the above-mentioned hand data can record the corresponding relationship between the recognition time and the recognition result. Among them, the above-mentioned recognition result indicates whether the hand position is recognized, and the above-mentioned hand position is the image position of at least part of the hand area existing in each frame image, that is, when performing the above-mentioned hand position recognition, it is not limited to the image position of the entire complete hand of the user. When only part of the hand area such as part of the user's fingers and part of the palm exists in the image, the image position of the above-mentioned part of the hand area can be recognized as the recognized hand position.

[0114] That is to say, when the user uses the VR head display device, the VR head display device can identify whether there is at least part of the user's hand area in the image captured by the image acquisition device at the current moment at a preset time period that matches the above-mentioned preset recognition frequency, and when there is at least part of the user's hand area in the image, identify the image position of the above-mentioned hand area in the image as the recognition result. When the user's hand area does not exist in the image, the recognition result of this recognition can be recorded as empty. Furthermore, the VR head display device can use the above-mentioned current moment as the recognition moment, so that the above-mentioned current moment and the above-mentioned recognition result are recorded as a set of hand data in correspondence.

[0115] In this way, when viewing the hand data recorded by the VR head display device, if a group of hand data includes a recognition result that is an image position, it can be determined that the above recognition result indicates that the hand position is recognized, that is, at the recognition moment included in the group of hand data, at least part of the hand area of ​​the user is recognized in the image captured by the image acquisition device; and when the recognition result included in the group of hand data is empty, it can be determined that the above recognition result indicates that the hand position is not recognized, that is, at the recognition moment included in the group of hand data, the user's hand area does not exist in the image captured by the image acquisition device.

[0116] For example, assuming that the preset recognition frequency is 10 times / s (Second), the preset duration is 0.1s, and the starting time of the gesture interaction process is 0s. At 0s, the image acquisition device acquires the first frame image, and at the same time, the VR head display device acquires the first frame image and performs hand position recognition on the first frame image. At this time, 0s is the recognition time, and the recognition result of the hand position recognition of the first frame image is the recognition result of 0s. Then, the VR head display device can record the recognition results of the hand positions of the above 0s and the above first frame image as the first set of hand data. Afterwards, at 0.1s, the VR head display device acquires the image acquired by the image acquisition and recognition at 0.1s, and performs hand position recognition on the image. At this time, 0.1s is the recognition time, so the VR head display device can record the recognition results of 0.1s and 0.1s as the second set of hand data. By analogy, the VR head display device can record multiple sets of hand data.

[0117] Obviously, in an embodiment of the present invention, when a user uses a VR head display device, when the VR head display device starts to perform hand position recognition on the image captured by the image acquisition device at the current moment according to the preset recognition frequency, the starting moment is the first recognition moment, and thereafter, according to the above-mentioned preset recognition frequency, the moment reached at every preset duration that matches the above-mentioned preset recognition frequency is the next recognition moment. That is to say, after the above-mentioned hand position recognition starts, each recognition moment with the above-mentioned starting moment as the first recognition moment is determined, and the time difference between each adjacent two starting moments is the preset duration that matches the preset recognition frequency. As in the above example, the above-mentioned preset recognition frequency is 10 times / s, and the starting moment of the gesture position recognition is 0s, then the various recognition moments are: 0s, 0.1s, 0.2s, ..., and so on, and the time difference between each adjacent two recognition moments is 0.1s.

[0118] Among them, since the above-mentioned preset recognition frequency and the acquisition frequency of the above-mentioned image acquisition device can be the same or different, therefore, when the above-mentioned preset recognition frequency is different from the acquisition frequency of the above-mentioned image acquisition device, when a recognition moment is reached, the image acquisition device may not perform image acquisition at the recognition moment, that is, there is no image whose acquisition moment is the recognition moment. Therefore, among the images currently acquired by the image acquisition device, the image whose acquisition moment is closest to the recognition moment can be used as the image that the VR head display device needs to perform hand position recognition. Furthermore, the VR head display device can record the recognition results of the hand position recognition at the recognition moment and the image whose acquisition moment is closest to the recognition moment as a set of hand data.

[0119] For example, the recognition time is 0.5s, but there is no image captured and recognized at 0.5s, and at the current time (0.5s) there are images that have been captured by the image acquisition device and whose capture times are 0s, 0.2s and 0.4s, and the image acquisition device will capture the next frame of image at 0.6s. Then the image captured at the time of 0.4s is used as the image for which the VR head display device needs to perform hand position recognition. Furthermore, the VR head display device can record the recognition results of the hand position recognition of the above recognition time 0.5s and the above image captured at 0.4s as a set of hand data.

[0120] As mentioned above, the process from the first recognition of the grasping gesture to the first recognition of the opening gesture is the process of the user's hand performing a grasping and throwing action. Therefore, when the opening gesture is recognized for the first time, the VR headset device can determine the historical hand data recorded in the process from the first recognition of the grasping gesture to the first recognition of the opening gesture, that is, determine the groups of recognition moments and recognition results recorded by the VR headset device in the process of the user's hand performing this grasping and throwing action.

[0121] Based on this, when the opening gesture is recognized for the first time, the VR head display device can determine a specified time range in which the starting time is no earlier than the acquisition time of the first current image and the ending time is no later than the acquisition time of the second current image, and obtain the hand data recorded within the specified time range as historical hand data.

[0122] In addition, the above-mentioned identified hand position can usually be the coordinate position of at least part of the hand area existing in the image in a specified coordinate system, for example, the image coordinate system, the camera coordinate system of the above-mentioned image acquisition device, the world coordinate system, etc. Since the relative position relationship between the hand positions is used in the subsequent throwing speed determination process, therefore, in the embodiment of the present invention, the coordinate system where the above-mentioned hand position is located is not specifically limited, as long as the identified hand positions are located in the same coordinate system.

[0123] S104: Determine the distribution of recognition results indicating the recognized hand positions in the historical hand data.

[0124] S105: Determine the throwing speed of the hand using a determination method corresponding to a preset distribution condition.

[0125] Taking into account the hardware performance of the image acquisition device carried by the VR headset device, usually, due to the limitations of the above hardware performance, when the user's hand moves at a faster speed, the above image acquisition device may not be able to capture a clear hand image, or even be unable to capture a hand image, that is, there may not be a clear hand area in the image captured by the image acquisition device, or even no hand area, and then, when performing hand position recognition on the above images, the hand position cannot be recognized.

[0126] That is to say, according to the movement of the user's hand in the grabbing and throwing action, the VR head display device performs hand position recognition on the image captured by the image acquisition device, and there may be the following three situations:

[0127] Case 1: During the grabbing and throwing action, the user's hand moves slowly, so the image acquisition device can always acquire clear hand images, so that when the VR head display device recognizes the hand position of the image acquired by the image acquisition device, it can always recognize the hand position, that is, at each recognition moment during the grabbing and throwing action, the recognition result obtained by the VR head display device indicates that the hand position is recognized. Generally, the probability of Case 1 is generally less than 20%.

[0128] Obviously, in case 1, in the historical hand data within the specified time range, the recognition results at each recognition moment all indicate that the hand position is recognized. Furthermore, corresponding to case 1, the historical hand data can be used to calculate the maximum throwing speed within the specified time range as the throwing speed of the hand.

[0129] Case 2: Usually, in the process of performing the grabbing and throwing action, the user's hand first accelerates and then decelerates. During this period, when the user's hand moves faster, the above-mentioned image acquisition device cannot capture a clear hand image, or even cannot capture a hand image. Therefore, when the user's hand moves faster, for the image captured by the above-mentioned image acquisition device, the VR head display device cannot recognize the hand position and cannot perform gesture recognition. After the user's hand completes the action of opening and throwing, the user's hand movement speed will gradually slow down. When it slows down to a certain speed, the above-mentioned image acquisition device can capture a clear hand image again. Therefore, for the image captured by the above-mentioned image acquisition device, the VR head display device can recognize the hand position again, and recognize the opening gesture at the same time as recognizing the hand position again.

[0130] That is to say, in the above situation 2, the VR headset device recognizes the image of the opening gesture for the first time after recognizing the grasping gesture for the first time, that is, the above second current image is the first frame image of the hand position recognized again after the recognition result of the hand position recognition by the VR headset device is empty.

[0131] Obviously, in the above situation 2, within the above specified time range, the VR head display device experienced the process of identifying the hand position, failing to identify the hand position, and recognizing the hand position again. Therefore, in the historical hand data within the above specified time range, in the order of time from early to late, the recognition results of the first few recognition moments indicate that the hand position is recognized, the recognition results of the recognition moments after the first few and before the last are empty, indicating that the hand position is not recognized, and the recognition result of the recognition moment at the last position indicates that the hand position is recognized.

[0132] For example, corresponding to the above situation 2, assuming that in the process of the user's hand performing the grabbing and throwing action, starting from the 2nd second, the VR head display device cannot recognize the hand position of the user's hand because the speed of the user's hand is too fast, then the hand position of the hand at this time can be recorded as position a; when the 4th second, the speed of the user's hand slows down, the VR head display device can recognize the hand position of the user's hand again, then the hand position of the hand at this time can be recorded as position b. The average speed between the 2nd and 4th seconds can be obtained according to (ba) / ΔT, where ΔT is 2 seconds. Considering that the process of the user's hand performing the grabbing and throwing action is actually a process of first decelerating and then decelerating, an empirical coefficient can be multiplied on the above average speed to obtain a more reasonable result, that is, a throwing speed that is more in line with the user's expectations. For example, the above empirical coefficient can be 1.4, etc. Of course, the empirical coefficient can also be set to other values ​​according to the needs of the actual application scenario, and this embodiment of the present invention does not make specific limitations.

[0133] Case 3: The image acquisition device on the VR head display device has a limited field of view. If the accelerated movement of the user's hand is outside the field of view of the above image acquisition device during the grabbing and throwing action, the image acquisition device cannot capture the hand image during the acceleration of the user's hand. As a result, the VR head display device cannot identify the hand position and gesture recognition. Only after the user's hand enters the deceleration movement process and enters the field of view of the above image acquisition device, the image acquisition device can capture the hand image, so that the VR head display device can perform hand position recognition and gesture recognition.

[0134] For example, for a VR headset, when the user's hand starts to accelerate forward from behind the head to throw an object, the entire acceleration process of the user's hand from the beginning of movement to throwing the object may not be captured by the image acquisition device. As a result, during the above acceleration process, the VR headset cannot recognize the hand position or perform gesture recognition.

[0135] Obviously, in the above situation 3, in the historical hand data within the above specified time range, arranged in order from early to late, the recognition results at the recognition moments arranged in the first few indicate that the hand position is not recognized, while the recognition results at the recognition moments arranged after the above first few indicate that the hand position is recognized. Furthermore, corresponding to the above situation 3, the user's behavior can be judged based on experience to determine the throwing speed of the hand. Usually, when a user throws an object at an object very hard, there is a high probability that the user's eyes are looking directly at the target. Then, the direction of the user's line of sight can be used as the direction of the throwing speed, and the magnitude of the throwing speed can be determined by using the empirical value of the throwing speed of the user's hand performing the grabbing and throwing action, for example, using a larger value in the user's hand swinging speed range, exemplarily, 7m / s, etc.

[0136] As mentioned above, for the above different situations, when determining the throwing speed of the above hand, the determination method adopted can be different, so as to meet the requirement that in different situations, a throwing speed that is closer to the player's expectations can be determined. For the above different situations, the recognition results indicating the recognition of the hand position in the above historical hand data have different distributions, and the distribution can characterize the number of recognition results indicating the recognition of the hand position in the above historical hand data, and the arrangement position of the recognition results indicating the recognition of the hand position in the above historical hand data in the order of recognition time from early to late. For example, in the above situation 1, the above distribution is that in the above historical hand data, the recognition results at each recognition time all indicate that the hand position is recognized; in the above situation 3, the above distribution is that in the above historical hand data, the recognition results at each recognition time all indicate that the hand position is not recognized, etc.

[0137] Based on this, after obtaining the historical hand data within the above specified time range, the VR head display device can first determine the distribution of the recognition results indicating the hand position in the above historical hand data. The distribution can reflect the situation that the user's hand performs this grab and throw action. Then, the hand throwing speed can be determined using the preset determination method corresponding to the above distribution. In this way, by analyzing the recognition results of the hand position in the historical hand data, the hand throwing speed can be calculated according to the situation, thereby obtaining a throwing speed that is more in line with the user's expectations and improving the user's interactive experience with the VR head display device.

[0138] In addition, when a user uses a VR head-mounted display device, the user can perform multiple grabbing and throwing actions, and thus, it is necessary to determine the throwing speed of the user's hand when performing each grabbing and throwing action.

[0139] Based on this, optionally, after executing the above steps S101-S104 once and obtaining the throwing speed of the hand when the user performs a grabbing and throwing action, the VR head display device can start from the next frame image of the second current image and perform the grasping gesture recognition again on each frame image collected by the above image acquisition device, thereby repeating the above steps S101-S104 to determine the throwing speed of the hand when the user performs the next grabbing and throwing action. By looping in sequence, the throwing speed of the hand when the user performs each grabbing and throwing action during the use of the VR head display device can be determined to improve the interactive experience between the user and the VR head display device.

[0140] As can be seen from the above, by applying the solution provided in the embodiment of the present invention, the throwing speed of the user's hand can be determined, and considering that the hand movement speed of the user will experience a process of first accelerating and then decelerating when the user performs the grabbing and throwing action, therefore, compared with directly using the position difference between the hand position when the user starts to perform the grabbing and throwing action and the hand position when the user completes the grabbing and throwing action, dividing it by the time difference between the moment when the user starts to perform the grabbing and throwing action and the moment when the user completes the grabbing and throwing action to obtain the throwing speed of the hand, in the embodiment of the present invention, the throwing speed is determined by using the recognition results of the hand position at each recognition moment in the process of the user performing the grabbing and throwing action, which can more fully consider the influence of the acceleration process and deceleration process of the hand on the throwing speed during the execution of the grabbing and throwing action, thereby making the determined throwing speed closer to the actual throwing speed of the user's real hand when performing the grabbing and throwing action.

[0141] That is to say, by applying the solution provided by the embodiment of the present invention, when the user's real hand performs the action of throwing an object, the determined throwing speed can be made closer to the real throwing speed of the user's real hand when performing the grab and throw action, thereby achieving a more natural and more user-expected interaction effect between the user and the VR head display device. In addition, for applications of VR head display devices such as games that originally wanted to use the grab and throw operation but were not developed smoothly or even stagnant due to the unstable throwing feedback, developers can adopt the solution of the embodiment of the present invention to solve the problem of unstable throwing feedback and achieve a more natural interaction between the user and the VR head display device.

[0142] Optionally, in a specific implementation, the above distribution is: in the historical hand data, the recognition results at each recognition moment all indicate that the hand position is recognized; then Figure 3 As shown, the above step S105, using the determination method corresponding to the preset distribution situation to determine the throwing speed of the hand, may include the following steps S1051-S1052.

[0143] S1051: Calculate each candidate speed by using the position difference of the hand positions recognized at each two adjacent recognition moments in the historical hand data and the time difference between the two adjacent recognition moments.

[0144] S1052: Select the maximum speed among the candidate speeds as the throwing speed of the hand.

[0145] In this specific implementation, the above distribution is: in the above historical hand data, the recognition results at each recognition moment all indicate that the hand position is recognized, which means that the user's hand performs the grabbing and throwing action in accordance with the above situation 1. Then, the above historical hand data can be used to determine a maximum speed as the throwing speed of the hand.

[0146] Among them, for each two adjacent recognition moments in the above historical hand data, the position difference of the hand positions recognized at the two recognition moments and the time difference of the two adjacent recognition moments can be calculated, and the movement speed of the user's hand within the above time difference can be obtained by using the above position difference and time difference as a candidate speed. In this way, multiple candidate speeds can be obtained. Usually, the user's hand throws the object at the maximum movement speed, so the maximum speed among the candidate speeds can be selected as the throwing speed of the hand.

[0147] Optionally, for each two adjacent recognition moments in the historical hand data, since the hand position at the above recognition moment can be the position coordinate in the specified coordinate system, the above position difference can be a vector with magnitude and direction, referred to as a position vector, and the magnitude of the position vector is the distance the user's hand moves within the above time difference, and the direction of the position vector is the direction of movement of the user's hand within the above time difference. Furthermore, by dividing the above position difference by the above time difference, a vector with magnitude and direction can also be obtained as a velocity vector, and the velocity vector is the obtained candidate velocity, and the magnitude of the velocity vector is the rate of the candidate velocity, and the direction of the velocity vector is the direction of the candidate velocity. Furthermore, when selecting the throwing velocity of the hand, the candidate velocity with the largest rate is selected as the throwing velocity of the hand.

[0148] Optionally, for each two adjacent recognition moments in the historical hand data, since the hand position at the recognition moment can be the position coordinates in the specified coordinate system, the distance between the hand positions at the two recognition moments can be calculated using the position coordinates, and the distance can be divided by the time difference to obtain the rate of the candidate speed, and the offset direction between the two recognition moments can be calculated using the position coordinates to obtain the direction of the candidate speed, and then the candidate speed can be obtained by combining the rate and direction. Further, when selecting the throwing speed of the hand, the candidate speed with the largest rate is selected as the throwing speed of the hand.

[0149] Based on this, in this specific implementation, when the situation in which the user's hand performs a grabbing and throwing action meets the above-mentioned situation 1, and the above-mentioned image acquisition device can always capture clear hand images, the various recognition moments and recognition results in the above-mentioned historical hand data can be directly used to calculate the maximum movement speed of the user's hand as the throwing speed of the hand, so that the actual throwing speed that is closer to the user's real hand when performing the grabbing and throwing action can be obtained more quickly and accurately, thereby improving the user experience.

[0150] Optionally, in another specific implementation, the above distribution is: in the historical hand data, the content of the recognition results at each recognition moment does not uniformly indicate that the hand position is recognized, and in the historical hand data, the recognition results at the recognition moments of the first specified number of digits are arranged in order from early to late, including the recognition results indicating that the hand position is recognized; then Figure 4 As shown, the above step S105, using the determination method corresponding to the preset distribution situation to determine the throwing speed of the hand, may include the following steps S1053-S1054.

[0151] S1053: Calculate the movement speed of the hand as the first speed using the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment.

[0152] Among them, the first recognition moment is: the last recognition moment at which the hand position is recognized as indicated by the recognition result, and the second recognition moment is: the recognition moment at which the hand position is recognized as indicated by the recognition result, which is closest to the first recognition moment and earlier than the first recognition moment.

[0153] S1054: Based on the first speed, determine the throwing speed of the hand.

[0154] In this specific implementation, it is possible to first detect whether the recognition results of each recognition moment in the above historical hand data all indicate that the hand position is recognized. When it is detected that the recognition results of each recognition moment in the above historical hand data do not all indicate that the hand position is recognized, it is possible to further detect whether there is a recognition moment whose recognition result indicates that the hand position is recognized among the recognition moments arranged in the first specified number of digits in the order from early to late in the above historical hand data. In other words, it is possible to further detect whether the recognition results of the above historical hand data arranged in the first specified number of digits in the order from early to late in the above historical hand data include a recognition result indicating that the hand position is recognized. The above specified number is greater than 1, for example, the above specified number is 3. Of course, it can also be other values, and this is not specifically limited in the embodiments of the present invention.

[0155] As mentioned above, usually, in the above situation 2, in the historical hand data within the above specified time range, in order from early to late, the recognition results at the first few recognition moments indicate that the hand position is recognized, the recognition results at the recognition moments after the first few and before the last are empty, indicating that the hand position is not recognized, and the recognition result at the last recognition moment indicates that the hand position is recognized.

[0156] However, in some cases, due to unexpected circumstances such as shaking of the image acquisition device, in the above situation 2, at some of the recognition moments ranked in the first few positions, the image acquisition device may not be able to capture the hand image, and thus, the recognition results at these moments are empty. In these cases, considering that the distribution of the recognition results at each recognition moment in the historical hand data within the above specified time range is similar to the distribution of the recognition results in the above situation 2, these cases can also be regarded as meeting the above situation 2.

[0157] Therefore, the above distribution is as follows: in the above historical hand data, the contents of the recognition results at each recognition moment do not all indicate that the hand position has been recognized, and in the above historical hand data, the recognition results of the recognition moments arranged in the first specified number of digits in order from early to late include recognition results indicating that the hand position has been recognized, which indicates that the user's hand performing the grabbing and throwing action meets the above situation 2.

[0158] Among them, in the above historical hand data, in the order of time from early to late, the recognition results of the recognition times arranged in the first specified number of digits indicate that the recognized hand positions exist in two situations:

[0159] Case 1: In the above historical hand data, in the order of time from earliest to latest, the recognition results of the recognition times arranged in the first specified number of places indicate that the hand position is recognized.

[0160] Situation 2: In the above historical hand data, in the order from early to late, the recognition results of the recognition moments arranged in the first specified number of digits do not all indicate that the hand position is recognized, and in the above historical hand data, in the order from early to late, there are recognition moments whose recognition results indicate that the hand position is recognized.

[0161] In this way, the last recognition moment at which the above recognition result indicates the hand position can be taken as the first recognition moment, and the recognition moment at which the recognition result indicates the hand position, which is closest to the above first recognition moment and earlier than the above first recognition moment, can be taken as the second recognition moment. Thus, the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment can be used to calculate the hand movement speed as the first speed. Then, the hand throwing speed can be determined based on the above reference speed. For example, the above first speed can be determined as the hand throwing speed. The calculation method of the above first speed is the same as the above Figure 2 The calculation method of each candidate speed in the specific implementation shown is the same and will not be repeated here.

[0162] Based on this, in this specific implementation, when the user's hand performs a grabbing and throwing action that meets the above-mentioned situation 2, the above-mentioned image acquisition device can only capture partial images of the user's hand before it accelerates to a certain speed and after it decelerates to a certain speed. Thus, the recognition results indicating the hand position and the recognition moments corresponding to these recognition results in the above-mentioned historical hand data can be used to calculate the movement speed of the user's hand, and based on the calculated movement speed, the throwing speed of the hand can be further determined, so as to obtain as accurately as possible the actual throwing speed that is closer to the user's real hand when performing the grabbing and throwing action.

[0163] Optionally, in a specific implementation, such as Figure 5 As shown, the above step S1054: determining the throwing speed of the hand based on the first speed may include the following steps S1054A-S1054B.

[0164] S1054A: Calculate the product of the first speed and a preset empirical coefficient as the first reference speed.

[0165] S1054B: Based on the first reference speed, determine the throwing speed of the hand.

[0166] In this specific implementation, as mentioned above, in the above situation 2, since the user's hand moves faster than a certain speed, the above image acquisition device often cannot capture a clear hand image. Therefore, when the user's hand performs the grabbing and throwing action in accordance with the above situation 2, the first speed calculated above has a certain gap with the maximum moving speed of the user's hand during the grabbing and throwing action. Therefore, the first speed can be multiplied by a preset empirical coefficient to obtain a first reference speed, and then the throwing speed of the hand is determined based on the above first reference speed. For example, the above first reference speed is determined as the throwing speed of the hand.

[0167] Among them, the above-mentioned preset empirical coefficient can be determined according to the empirical value of the throwing speed of the user's hand, for example, it can be 1.4. Of course, it can also be set to other values ​​according to the needs of the actual application scenario. This embodiment of the present invention does not make specific limitations on this.

[0168] In this way, in this specific implementation, the final determined throwing speed of the hand can be closer to the actual throwing speed of the user's real hand when performing the grabbing and throwing action, which is more in line with the user's expectations.

[0169] Optionally, in another specific implementation, such as Figure 6 As shown in the above Figure 5 Based on the specific implementation shown, a speed determination method provided by an embodiment of the present invention may further include the following step S1054C.

[0170] S1054C: Utilize the recognition result to indicate the position difference of the hand between each two adjacent recognition moments and the time difference between the two adjacent recognition moments at each moment when the hand position is recognized, calculate each movement speed of the hand as each second speed, and select the largest second speed as the second reference speed.

[0171] Accordingly, in this specific implementation, the above step S1054B, based on the first reference speed, determines the throwing speed of the hand, and may include the following step S1054D:

[0172] S1054D: Select the maximum speed between the first reference speed and the second reference speed as the throwing speed of the hand.

[0173] In this specific implementation, the position difference of the hand position at each adjacent recognition moment and the time difference of the two adjacent recognition moments can be used to calculate the movement speeds of the hand as the second speeds, and then the maximum second speed is selected as the second reference speed. The calculation method of the second speeds and the selection method of the second reference speed are respectively the same as those in the above Figure 2 The calculation method of each candidate speed and the selection method of the throwing speed of the hand in the specific implementation method shown are the same and will not be repeated here.

[0174] Afterwards, the maximum speed between the first reference speed and the second reference speed may be selected as the throwing speed of the hand.

[0175] For example, as shown in Table 1 below, the historical hand data obtained in a specific example is shown. Assuming that the specified number is 3 and the preset empirical coefficient is 1.4, the process of determining the throwing speed of the hand includes the following steps 1-7.

[0176] Table 1

[0177]

[0178] Step 1: Use the position difference between L11 and L4, and the time difference between 150.30s and 105.12s to calculate the hand movement speed as the first speed V4, and calculate the product of the first speed V4 and the preset empirical coefficient 1.4 as the first reference speed, recorded as 1.4×V4.

[0179] Step 2: Calculate the first second speed V1 using the position difference between L1 and L2 and the time difference between 150.03s and 105.00s.

[0180] Step 3: Calculate the second speed V2 using the position difference between L2 and L3 and the time difference between 150.06s and 105.03s.

[0181] Step 4: Calculate the third second speed V3 using the position difference between L3 and L4 and the time difference between 150.09s and 105.06s.

[0182] Step 5: Record the above first speed V4 as the fourth second speed.

[0183] Step 6: Select the largest second speed among the above four second speeds as the second reference speed. Assume that the obtained second reference speed is V3.

[0184] Step 7: Select the maximum data of the first reference speed 1.4×V4 and the second reference speed V3 as the hand throwing speed. Assuming V3<1.4×V4, the obtained hand throwing speed is 1.4×V4.

[0185] For another example, as shown in Table 2 below, the historical hand data obtained in a specific example is shown, where it is assumed that the specified number is 3 and the preset empirical coefficient is 1.4. The process of determining the throwing speed of the hand includes the following steps 1-8.

[0186] Table 2

[0187]

[0188] Step 1: Use the position difference between L11 and L5, and the time difference between 150.30 and 150.15 to calculate the hand movement speed as the first speed v4, and calculate the product of the first speed v1 and the preset empirical coefficient 1.4 as the first reference speed, recorded as 1.4×v4.

[0189] Step 2: Calculate the first second velocity v1 using the position difference between L1 and L2 and the time difference between 150.03s and 105.00s.

[0190] Step 3: Calculate the second velocity v2 using the position difference between L2 and L4 and the time difference between 150.09s and 105.03s.

[0191] Step 4: Calculate the third second velocity v3 using the position difference between L4 and L5, and the time difference between 150.15s and 105.09s.

[0192] Step 5: Record the above first speed v4 as the fourth second speed.

[0193] Step 6: Among the above four second speeds, select the largest second speed as the second reference speed. Assume that the obtained second reference speed is v2.

[0194] Step 7: Select the maximum data between the second reference speed v2 and the first reference speed 1.4×v4 as the throwing speed of the hand. Assuming v2>1.4×v4, the obtained hand throwing speed is v2.

[0195] Thus, in this specific implementation, the hand throwing speed can be corrected using the above-mentioned preset empirical coefficients and the maximum speed among the calculated second speeds, so that the hand throwing speed finally determined can be closer to the actual throwing speed of the user's real hand when performing the grabbing and throwing action, and more in line with user expectations.

[0196] Optionally, in another specific implementation, the above distribution is: in the historical hand data, in the order of time from early to late, the recognition results of the recognition times arranged in the first specified number of digits all indicate that the hand position is not recognized; then Figure 7 As shown, the above step S105, using the determination method corresponding to the preset distribution situation to determine the throwing speed of the hand, may include the following step S1055.

[0197] S1055: Determine the preset rate as the rate of the hand throwing speed, and determine the user's line of sight direction as the direction of the hand throwing speed, to obtain the hand throwing speed.

[0198] In this specific implementation, it is possible to first detect whether the recognition results of each recognition moment in the above historical hand data all indicate that the hand position is recognized. When it is detected that the recognition results of each recognition moment in the above historical hand data do not all indicate that the hand position is recognized, it is possible to further detect whether the recognition results arranged in the first specified number of digits in the order from early to late in the above historical hand data indicate that the hand position is recognized. Among them, the above specified number is greater than 1, for example, the above specified number is 3, and of course, it can also be other values, and the embodiment of the present invention does not make specific limitations on this. Furthermore, when the detection result is: in the above historical hand data, in the order from early to late in the time, the recognition results of the recognition moments arranged in the first specified number of digits all indicate that the hand position is not recognized, it can be explained that the situation of the user's hand performing this grabbing and throwing action meets the above situation 3.

[0199] So far, the distribution of the recognition results indicating that the hand position is recognized in the above historical hand data is as follows: in the above historical hand data, in the order of time from early to late, the recognition results of the recognition times arranged in the first specified number of places all indicate that the hand position is not recognized.

[0200] As mentioned above, in the above situation 3, during the acceleration process of the user's hand, the image acquisition device cannot capture the hand image, so that the VR head display device cannot recognize the hand position and cannot perform gesture recognition. It is not until the user's hand enters the deceleration movement process and enters the field of view of the above image acquisition device that the image acquisition device can capture the hand image, so that the VR head display device can perform hand position recognition and gesture recognition. Therefore, combined with the above detection results, it can be explained that in this specific implementation method, it is impossible to use the above historical hand data to directly determine the throwing speed of the hand, so that the determined throwing speed is as close as possible to the actual throwing speed of the user's real hand when performing the grabbing and throwing action.

[0201] Based on this, the throwing speed of the hand when executing the grab and throw action can be directly determined according to the empirical value of the actual throwing speed of the user's hand. For example, it can be 7m / s. Of course, it can also be set to other values ​​according to the needs of the actual application scenario. This is not specifically limited in the embodiment of the present invention.

[0202] That is to say, in this specific implementation, the preset rate can be determined as the throwing speed of the hand. The preset rate can be determined based on the empirical value of the real throwing speed of the user's hand. Furthermore, considering that during the execution of the grabbing and throwing action, when the user throws the object, it is highly likely that the user's eyes will look directly at the object to which the object is thrown. Therefore, the user's line of sight can be determined as the direction of the throwing speed of the hand, thereby combining the above rate and direction to obtain the throwing speed of the hand.

[0203] Among them, optionally, the VR head display device as the execution subject can also be equipped with a positioning device, such as an IMU (Inertial Measurement Unit), a gyroscope, etc., and the positioning device can detect the position and angle of the VR head display device in space, and the angle of the VR head display device in space is usually used to indicate the orientation of the VR head display device, and the orientation of the VR head display device is usually consistent with the user's line of sight. Therefore, the angle obtained by the above positioning device can be used as the direction of the throwing speed of the above hand. For example, for a VR head display device, the positioning device it carries can detect the orientation of the VR head display device, and the orientation can be regarded as the direction of the user's line of sight, that is, the orientation of the VR head display device can be used as the user's line of sight, and then, as the direction of the throwing speed of the above hand.

[0204] Based on this, in this specific implementation, when the user's hand performs the grab and throw action in accordance with the above situation 3, the empirical value of the real throwing speed of the user's hand performing the grab and throw action can be used to directly assign a value to the rate of the hand throwing speed to be determined, and the user's line of sight direction is determined as the direction of the hand throwing speed, thereby obtaining the hand throwing speed. In this way, the real throwing speed that is closer to the user's real hand when performing the grab and throw action can be obtained more quickly and accurately, avoiding the problem of poor interaction between the user and the VR head display device caused by the large error between the hand throwing speed calculated by the hand position of the user's hand during the deceleration process in the above historical hand data and the real throwing speed of the user's real hand when performing the grab and throw action, thereby improving the user experience.

[0205] Optionally, in a specific implementation, in the above step S103, before determining the historical hand data within a specified time range, a speed determination method provided by an embodiment of the present invention further includes the following step 1.

[0206] Step 1: Determine a specified time range with the acquisition time of the first current image as the starting time and the acquisition time of the second current image as the ending time.

[0207] In this specific implementation, when the above-mentioned opening gesture is recognized for the first time, the time range between the acquisition time of the above-mentioned first current image and the acquisition time of the second current image can be used as the specified time range. Therefore, the specified time range with the acquisition time of the first current image as the starting time and the acquisition time of the second current image as the ending time can be determined. In this way, each recognition time in the specified time range and the recognition result of each recognition time can be obtained as historical hand data.

[0208] Optionally, in another specific implementation, in the above step S103, before determining the historical hand data within a specified time range, a speed determination method provided by an embodiment of the present invention further includes the following step 2.

[0209] Step 2: Determine a specified time range with the acquisition moment of the second current image as the end moment and a duration of a preset duration.

[0210] In this specific implementation, usually, according to the movement law of the human body, the duration of a user's hand performing a grabbing and throwing motion is relatively consistent. Therefore, the preset duration can be determined based on the empirical value of the duration of the user's hand performing a grabbing and throwing motion. The preset duration can be regarded as the duration of the user's hand performing a grabbing and throwing motion in the embodiment of the present invention. For example, 0.5s, 0.6s, etc. Of course, it can also be set to other durations according to the needs of the actual application scenario. This is not specifically limited in the embodiment of the present invention. Furthermore, in this way, when the opening gesture is first recognized, it can be determined that the acquisition moment of the second current image is the end moment, and the specified time range of the duration is the preset duration. In this way, each recognition moment in the specified time range and the recognition result of each recognition moment can be obtained as historical hand data.

[0211] Among them, since the above-mentioned preset duration can be an empirical value based on the duration of a throwing process performed by the user's hand (i.e., the process of the user quickly waving his arm and opening his hand to throw an object), and the above-mentioned throwing process is usually a part of the process of the user performing a grabbing and throwing action, therefore, usually, the above-mentioned preset duration will not exceed the duration of a user performing a grabbing and throwing action, that is, the above-mentioned preset duration will usually not exceed the duration between the first recognition of the grabbing hand gesture and the first recognition of the opening hand gesture, that is, the above-mentioned preset duration will usually not exceed the time difference between the acquisition time of the above-mentioned first current image and the second current image. Furthermore, the starting time of the above-mentioned specified time range is usually not earlier than the acquisition time of the above-mentioned first current image, and further, the above-mentioned specified time range meets the requirements that the starting time is not earlier than the acquisition time of the first current image, and the ending time is not later than the acquisition time of the second current image.

[0212] Optionally, in the specific implementation method shown in the above step 2, if the starting time of the above-determined specified time range is earlier than the acquisition time of the above-mentioned first current image, the starting time of the determined specified time range can be corrected to the acquisition time of the above-mentioned first current image, thereby obtaining the historical hand data within the specified time range after the start time is corrected.

[0213] In summary, Figure 8 A flowchart of a specific example provided by an embodiment of the present invention, such as Figure 8As shown, the specific example may include the following steps S801-S809:

[0214] S801: Performing grasping gesture recognition on each frame of image captured by the image acquisition device.

[0215] S802: When the grasping gesture is recognized for the first time, an open gesture recognition is performed on each frame image collected after the first current image.

[0216] Among them, the specific implementation methods of the above steps S801-S802 are the same as the above steps S101-S102, which will not be repeated here.

[0217] S803: Obtain hand position data.

[0218] When the open gesture is recognized for the first time, historical hand data within a specified time range is determined, and the determined historical hand data is the hand position data.

[0219] S804: Determine whether the position of the hand changes in each frame. If so, execute S805; otherwise, execute S806.

[0220] Generally, if the image acquisition device carried by the VR head display device can always capture clear hand images during the process of the user's hand performing the grabbing and throwing action, the VR head display device can always identify the hand position of the user's hand, and because the user's hand is always moving during the execution of the above grabbing and throwing action, the hand position recognized by the above VR head display device is changing. Among them, since the above hand position is obtained by identifying the image collected by the above image acquisition device, each identified hand position can be called a frame of hand position, and then, by detecting whether the hand position changes in each frame in the above acquired hand position data, it can be detected whether the recognition results at each recognition moment in the historical hand data all indicate that the hand position is recognized. If so, execute S805; otherwise, execute S806.

[0221] S805: Case 1: Calculate the hand speed of each frame and output the hand speed with the maximum rate.

[0222] If the detection result of the above step S804 is yes, it means that in the above acquired hand position data, the hand position changes in each frame, that is, it is detected that the recognition results at each recognition moment in the above historical hand data all indicate that the hand position is recognized, then it can be explained that the distribution of the recognition results indicating that the hand position is recognized in the above hand position data is: in the above hand position data, the recognition results at each recognition moment all indicate that the hand position is recognized. Therefore, the hand position recognition of this grab and throw action meets the above situation 1. Then, the hand speed of each frame can be calculated, that is, the hand movement speed when each hand position is recognized is calculated, and the hand speed with the maximum rate is output as the final determined hand throwing speed. Among them, the position difference of the hand position at each adjacent recognition moment in the historical hand data and the time difference of the two adjacent recognition moments can be used to calculate each candidate speed, and the maximum speed among each candidate speed (that is, the hand speed with the maximum rate) is selected as the hand throwing speed.

[0223] S806: Determine whether there is hand position data of the acceleration process, if so, execute S807; otherwise, execute S808.

[0224] If the detection result of the above step S804 is no, it means that during the process of this grabbing and throwing action, there is a part of time when the user's hand position data cannot be recognized. As mentioned above, when the hand position recognition of this grabbing and throwing action meets the above situation 2, during the accelerated movement of the user's hand, only the hand position data of the user's hand in the early acceleration process can be recognized, and in the later acceleration process, when the user's hand moves faster, the hand position data cannot be recognized. Based on this, if the detection result of the above step S804 is no, it can be further determined whether there is hand position data of the acceleration process in the above acquired hand position data. That is, detect whether the recognition results in the above historical hand data, arranged in the order of time from early to late, include recognition results indicating that the hand position is recognized. If so, execute S807; otherwise, execute S808.

[0225] S807: Case 2: (BA) / Δt×1.4.

[0226] If the detection result of the above step S806 is yes, it means that the acquired hand position data includes the hand position data of the acceleration process, that is, the distribution of the recognition results indicating the recognition of the hand position in the above hand position data is as follows: in the above hand position data, the recognition results at each recognition moment do not all indicate the recognition of the hand position, and in the above hand position data, the recognition results at the recognition moments arranged in the first specified number of digits in the order from early to late include the recognition results indicating the recognition of the hand position. Therefore, the hand position recognition of this grab and throw action meets the above situation 2, and therefore, the throwing speed of the hand can be directly calculated using the formula (BA) / Δt×1.4. That is, the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment are used to calculate the moving speed of the hand as the first speed, and the product of the first speed and the preset empirical coefficient is calculated as the throwing speed of the hand. Among them, B is the hand position at the first recognition moment, A is the hand position at the second recognition moment, Δt is the time difference between the first recognition moment and the second recognition moment, and 1.4 is a preset empirical coefficient.

[0227] S808: Case 3: The orientation of the VR head display device and the preset rate are used as the result.

[0228] If the detection result of the above step S806 is no, it means that during the process of this grabbing and throwing action, during the acceleration of the user's hand, the image acquisition device cannot capture the hand image, that is, in the above hand position data, the distribution of the recognition results indicating the recognition of the hand position is as follows: in the above hand position data, in the order of time from early to late, the recognition results of the recognition moments arranged in the first specified number of digits all indicate that the hand position is not recognized, so that the VR head display device cannot recognize the hand position and cannot perform gesture recognition. Only after the user's hand enters the deceleration movement process and enters the field of view of the above image acquisition device, the image acquisition device can capture the hand image, and then the VR head display device can perform hand position recognition and gesture recognition. Therefore, the hand position recognition of this grabbing and throwing action meets the above situation 3, and the orientation of the VR head display device and the preset rate can be used as the result, that is, the preset rate is determined as the rate of the hand throwing speed, and the user's line of sight direction is determined as the direction of the hand throwing speed to obtain the hand throwing speed.

[0229] S809: Get the hand speed at the time of throwing.

[0230] Through the process of steps S801-S808, the speed of the hand of the user at the throwing moment during the process of performing the grabbing and throwing action can be obtained, that is, the throwing speed of the hand can be obtained.

[0231] Corresponding to the speed determination method provided by the above embodiment of the present invention, the embodiment of the present invention further provides a speed determination device. The device is applied to a VR head display device, and the VR head display device is equipped with an image acquisition device.

[0232] Fig. 9 A schematic diagram of a speed determination device provided by an embodiment of the present invention is shown in FIG. Fig. 9 As shown, the speed determination device may include the following modules:

[0233] A first recognition module 910 is used to perform grasping gesture recognition on each frame of image captured by the image acquisition device; wherein the posture of each finger in the grasping gesture conforms to a first preset posture;

[0234] The second recognition module 920 is used to perform an open gesture recognition on each frame image collected after the first current image when the grasping gesture is recognized for the first time; wherein the first current image is: an image in which the grasping gesture is recognized for the first time, and the posture of each finger in the open gesture conforms to the second preset posture;

[0235] The data determination module 930 is used to determine the historical hand data within a specified time range when the opening gesture is first recognized; wherein the historical hand data indicates each recognition moment of the hand position recognition for each frame image according to a preset recognition frequency, and the recognition result of each recognition moment, the recognition result indicates whether the hand position is recognized, and the hand position is the image position of at least part of the hand area existing in each frame image; the starting time of the specified time range is not earlier than the acquisition time of the first current image and the ending time of the specified time range is not later than the acquisition time of the second current image, and the second current image is: the image in which the opening gesture is first recognized after the grasping gesture is recognized;

[0236] The speed determination module 940 is used to determine the distribution of the recognition results indicating the hand position in the historical hand data, and determine the throwing speed of the hand using a preset determination method corresponding to the distribution.

[0237] As can be seen from the above, by applying the solution provided by the embodiment of the present invention, in the process of users using various applications of VR head display devices, since the user's hands will experience the process of grasping objects, moving quickly and opening and throwing when performing grabbing and throwing actions, the process of the user performing a grabbing and throwing action can be identified by identifying the grabbing gesture and opening gesture in each frame image captured by the image acquisition device carried by the VR head display device. Among them, when the grabbing gesture is first recognized, it can be regarded as the user starting to perform the grabbing and throwing action, and after the grabbing gesture is first recognized, when the opening gesture is first recognized, it can be regarded as the user completing the grabbing and throwing action. In addition, in the above-mentioned gesture recognition process, the hand position recognition can be performed on each frame image captured by the above-mentioned image acquisition device according to the preset recognition frequency, that is, it can be identified whether there is at least a part of the user's hand area in the image and the image position of the existing hand area is identified. In this way, the method for determining the distribution of the hand positions in the recognition results recorded within the specified time range between the acquisition time of the image of the first recognized posture and the acquisition time of the image of the first recognized open posture can be used to determine the throwing speed of the user performing the grab and throw action, that is, to obtain the throwing speed of the hand. Furthermore, the throwing speed can be assigned to the initial speed of the virtual object thrown by the virtual hand in the application scene of the VR head display device, that is, the throwing speed is determined as the initial speed of the virtual object thrown by the virtual hand in the application scene of the VR head display device, so as to facilitate the subsequent related processing of the virtual object, for example, using the initial speed to predict the trajectory of the virtual object.

[0238] Based on this, the scheme provided by the embodiment of the present invention is applied to determine the throwing speed of the user's hand, and considering that the movement speed of the hand of the user will experience a process of first accelerating and then decelerating when performing the grabbing and throwing action, therefore, compared with directly using the position difference between the hand position when the user starts to perform the grabbing and throwing action and the hand position when the user completes the grabbing and throwing action, dividing it by the time difference between the moment when the user starts to perform the grabbing and throwing action and the moment when the user completes the grabbing and throwing action to obtain the throwing speed of the hand, in the embodiment of the present invention, the throwing speed is determined by using the recognition results of the hand position at each recognition moment in the process of the user performing the grabbing and throwing action, which can more fully consider the influence of the acceleration process and deceleration process of the hand on the throwing speed during the execution of the grabbing and throwing action, thereby making the determined throwing speed closer to the actual throwing speed of the user's real hand when performing the grabbing and throwing action.

[0239] That is to say, by applying the solution provided by the embodiment of the present invention, when the user's real hand performs the action of throwing an object, the determined throwing speed can be made closer to the real throwing speed of the user's real hand when performing the grab and throw action, thereby achieving a more natural and more user-expected interaction effect between the user and the VR head display device. In addition, for applications of VR head display devices such as games that originally wanted to use the grab and throw operation but were not developed smoothly or even stagnant due to the unstable throwing feedback, developers can adopt the solution of the embodiment of the present invention to solve the problem of unstable throwing feedback and achieve a more natural interaction between the user and the VR head display device.

[0240] Optionally, in a specific implementation, the data determination module 930 is further used to:

[0241] Before determining the historical hand data within the specified time range, determining a specified time range with the acquisition time of the first current image as the starting time and the acquisition time of the second current image as the ending time; or,

[0242] Before determining the historical hand data within the specified time range, a specified time range is determined with the acquisition time of the second current image as the end time and the duration being a preset duration.

[0243] Optionally, in a specific implementation, the distribution is: in the historical hand data, the recognition results at each recognition moment all indicate that the hand position is recognized; and the speed determination module 940 is specifically used to:

[0244] Calculate each candidate speed by using the position difference of the hand positions recognized at each two adjacent recognition moments in the historical hand data and the time difference between the two adjacent recognition moments;

[0245] The maximum speed among the candidate speeds is selected as the throwing speed of the hand.

[0246] Optionally, in a specific implementation, the distribution is as follows: in the historical hand data, the content of the recognition results at each recognition moment does not uniformly indicate that the hand position is recognized, and in the historical hand data, the recognition results at the recognition moments of the first specified number of digits are arranged in order from early to late, including the recognition results indicating that the hand position is recognized; the speed determination module 940 includes:

[0247] The speed calculation submodule calculates the moving speed of the hand as the first speed by using the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment; wherein the first recognition moment is: the last recognition moment at which the recognition result indicates that the hand position is recognized, and the second recognition moment is: the recognition result indicates that the hand position is recognized, the recognition moment closest to the first recognition moment and earlier than the first recognition moment;

[0248] The speed determination submodule is used to determine the throwing speed of the hand based on the first speed.

[0249] Optionally, in a specific implementation, the speed determination submodule includes:

[0250] a speed calculation unit, used to calculate the product of the first speed and a preset empirical coefficient as a first reference speed;

[0251] The speed determination unit is used to determine the throwing speed of the hand based on the first reference speed.

[0252] Optionally, in a specific implementation, the device further includes:

[0253] A speed selection module, used for calculating each moving speed of the hand as each second speed by using the position difference of the hand position between each two adjacent recognition moments at each recognition moment when the hand position is recognized indicated by the recognition result and the time difference between the two adjacent recognition moments before determining the throwing speed of the hand based on the first reference speed, and selecting the maximum second speed as the second reference speed;

[0254] The speed determination unit is specifically used for:

[0255] The maximum speed between the first reference speed and the second reference speed is selected as the throwing speed of the hand.

[0256] Optionally, in a specific implementation, the distribution is: in the historical hand data, in the order of time from early to late, the recognition results of the recognition moments arranged in the first specified number of digits all indicate that the hand position is not recognized; the speed determination module 940 is specifically used to:

[0257] The preset rate is determined as the rate of the hand throwing speed, and the user's line of sight direction is determined as the direction of the hand throwing speed, so as to obtain the hand throwing speed.

[0258] The embodiment of the present invention also provides a VR head display device, such as Fig.10As shown, it includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0259] Memory 1003, used for storing computer programs;

[0260] The processor 1001 is used to implement the steps of any speed determination method provided by the above-mentioned embodiment of the present invention when executing the program stored in the memory 1003.

[0261] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0262] The communication interface is used for communication between the above terminal and other devices.

[0263] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0264] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0265] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any speed determination method provided by the above-mentioned embodiment of the present invention are implemented.

[0266] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the steps of any speed determination method provided in the above-mentioned embodiments of the present invention.

[0267] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0268] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0269] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, VR head display device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0270] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A speed determination method, characterized in that: Applied to a VR head display device, the VR head display device is equipped with an image acquisition device; the method comprises: Performing grasping gesture recognition on each frame of image captured by the image acquisition device; wherein the posture of each finger in the grasping gesture conforms to a first preset posture; When the grasping gesture is recognized for the first time, an open gesture recognition is performed on each frame image collected after the first current image; wherein the first current image is: an image in which the grasping gesture is recognized for the first time, and the posture of each finger in the open gesture conforms to the second preset posture; When the opening gesture is recognized for the first time, historical hand data within a specified time range is determined; wherein the historical hand data indicates each recognition moment of hand position recognition for each frame image according to a preset recognition frequency, and a recognition result at each recognition moment, the recognition result indicating whether the hand position is recognized, the hand position being the image position of at least part of the hand area present in each frame image; the starting moment of the specified time range is not earlier than the acquisition moment of the first current image and the ending moment of the specified time range is not later than the acquisition moment of the second current image, the second current image being: the image at which the opening gesture is recognized for the first time after the grasping gesture is recognized; The distribution of the recognition results indicating the hand position in the historical hand data is determined, and the throwing speed of the hand is determined by using a preset determination method corresponding to the distribution.

2. The method according to claim 1, characterized in that Before determining the historical hand data within the specified time range, the method further includes: Determine a specified time range with the acquisition time of the first current image as the starting time and the acquisition time of the second current image as the ending time; or, A specified time range is determined, with the capture time of the second current image as the end time and the duration being a preset duration.

3. The method according to claim 1, characterized in that: The distribution is as follows: in the historical hand data, the recognition results at each recognition moment all indicate that the hand position is recognized; The method of determining the throwing speed of the hand by using the preset determination method corresponding to the distribution condition includes: Calculate each candidate speed by using the position difference of the hand positions recognized at each two adjacent recognition moments in the historical hand data and the time difference between the two adjacent recognition moments; The maximum speed among the candidate speeds is selected as the throwing speed of the hand.

4. The method according to claim 1, characterized in that: The distribution is as follows: in the historical hand data, the contents of the recognition results at each recognition moment do not all indicate that the hand position is recognized, and in the historical hand data, the recognition results at the recognition moments of the first specified number of digits are arranged in order from earliest to latest, including the recognition results indicating that the hand position is recognized; The method of determining the throwing speed of the hand by using the preset determination method corresponding to the distribution condition includes: The moving speed of the hand is calculated as the first speed by using the position difference between the hand position at the first recognition moment and the hand position at the second recognition moment, and the time difference between the first recognition moment and the second recognition moment; wherein the first recognition moment is: the last recognition moment at which the recognition result indicates that the hand position is recognized, and the second recognition moment is: the recognition result indicates that the hand position is recognized, the recognition moment that is closest to the first recognition moment and earlier than the first recognition moment; Based on the first speed, a throwing speed of the hand is determined.

5. The method according to claim 4, characterized in that The step of determining the throwing speed of the hand based on the first speed includes: Calculating the product of the first speed and a preset empirical coefficient as a first reference speed; Based on the first reference speed, the throwing speed of the hand is determined.

6. The method according to claim 5, characterized in that Before determining the throwing speed of the hand based on the first reference speed, the method further includes: Calculate the hand movement speeds as the second speeds using the position difference of the hand position between two adjacent recognition moments and the time difference between the two adjacent recognition moments at each recognition moment when the recognition result indicates the hand position, and select the maximum second speed as the second reference speed; The step of determining the throwing speed of the hand based on the first reference speed includes: The maximum speed between the first reference speed and the second reference speed is selected as the throwing speed of the hand.

7. The method according to claim 1, characterized in that The distribution is as follows: in the historical hand data, in the order of time from earliest to latest, the recognition results of the recognition times arranged in the first specified number of places all indicate that the hand position is not recognized; The method of determining the throwing speed of the hand by using the preset determination method corresponding to the distribution condition includes: The preset rate is determined as the rate of the hand throwing speed, and the user's line of sight direction is determined as the direction of the hand throwing speed, so as to obtain the hand throwing speed.

8. A speed determination device, characterized in that: Applied to a VR head display device, the VR head display device is equipped with an image acquisition device; the device comprises: A first recognition module, used for performing grasping gesture recognition on each frame of image captured by the image acquisition device; wherein the posture of each finger in the grasping gesture conforms to a first preset posture; A second recognition module is used to perform an open gesture recognition on each frame image collected after the first current image when the grasping gesture is recognized for the first time; wherein the first current image is: an image in which the grasping gesture is recognized for the first time, and the posture of each finger in the open gesture conforms to the second preset posture; A data determination module, for determining historical hand data within a specified time range when the opening gesture is first recognized; wherein the historical hand data indicates each recognition moment of hand position recognition for each frame image according to a preset recognition frequency, and a recognition result at each recognition moment, wherein the recognition result indicates whether the hand position is recognized, and the hand position is the image position of at least part of the hand area existing in each frame image; the starting moment of the specified time range is not earlier than the acquisition moment of the first current image and the ending moment of the specified time range is not later than the acquisition moment of the second current image, and the second current image is: the image at which the opening gesture is first recognized after the grasping gesture is recognized; The speed determination module is used to determine the distribution of the recognition results indicating the hand position in the historical hand data, and determine the throwing speed of the hand using a preset determination method corresponding to the distribution.

9. A VR head display device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.

Images