Interaction mode switching method and device, equipment, storage medium and program product

Abstract

The invention discloses an interaction mode switching method and device, equipment, a storage medium and a program product, and belongs to the technical field of computers. The method comprises the steps of determining a first position of a hand and a second position of a handle according to an image under the condition that the image is acquired; according to the first position and the second position, the distance between the hand and the handle is determined; according to the distance and a distance threshold value, the interaction mode is switched to a target mode, and the target mode comprises a gesture tracking mode or a handle tracking mode.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a method, device, equipment, storage medium and program product for switching an interactive mode. Background Art

[0002] At present, mainstream virtual reality (VR), augmented reality (AR) and mixed reality (MR) devices can support both handle tracking mode and gesture tracking mode. At present, the switching between the two interaction modes usually requires manual setting, or it is necessary to wait for a period of time to make sure that the handle is completely still before switching from handle tracking mode to gesture tracking mode. The first method requires tedious manual switching of the two interaction modes, and the second method requires waiting for a certain period of time as a basis for switching. Both methods have the problem of long switching time, resulting in interaction interruption and poor effect. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method, device, equipment, storage medium and program product for switching an interaction mode, which can solve the problem that the interaction mode switching takes a long time, resulting in interaction interruption and poor interaction effect.

[0004] In a first aspect, an embodiment of the present application provides a method for switching an interaction mode, the method comprising:

[0005] When the image is acquired, determining a first position of the hand and a second position of the handle according to the image;

[0006] determining a distance between the hand and the handle according to the first position and the second position;

[0007] According to the distance and the distance threshold, the interaction mode is switched to the target mode, which includes the gesture tracking mode or the handle tracking mode.

[0008] In a second aspect, an embodiment of the present application provides a switching device for an interactive mode, the device comprising:

[0009] A first determination module is used to determine a first position of the hand and a second position of the handle according to the image when the image is acquired;

[0010] A second determination module, used to determine the distance between the hand and the handle according to the first position and the second position;

[0011] The switching module is used to switch the interaction mode to the target mode according to the distance and the distance threshold. The target mode includes the gesture tracking mode or the handle tracking mode.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0013] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0014] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0015] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0016] In the embodiment of the present application, the first position of the hand and the second position of the handle can be determined based on the acquired image, and then the distance between the hand and the handle can be calculated. By comparing the distance with the distance threshold, it is determined whether the user intends to use the handle, so that the interaction mode can be switched to the gesture tracking mode or the handle tracking mode. In this way, the distance between the hand and the handle is used as the basis for switching the interaction mode, the waiting time during the switching process is reduced, and the interaction mode is quickly switched, achieving a better interaction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for switching an interactive mode provided in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of the distance in the method for switching the interactive mode provided in an embodiment of the present application;

[0019] Figure 3 is a working principle diagram of the method for switching the interactive mode provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of determining a first position in a method for switching an interactive mode provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of key points of the hand in the method for switching the interactive mode provided in an embodiment of the present application;

[0022] Figure 6 is a schematic diagram of determining a second position in the method for switching an interactive mode provided in an embodiment of the present application;

[0023] Figure 7 is a structural schematic diagram of an interactive mode switching device provided in an embodiment of the present application;

[0024] Figure 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0025] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] The following is a detailed description of the switching method of the interaction mode provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings. It can be understood that the application scenario of the embodiment of the present application can be in a scenario where a virtual VR, AR or MR device that can support both the handle tracking mode and the gesture tracking mode is used. For ease of understanding, this article will take the VR scenario as an example for explanation.

[0029] Figure 1 1 is a flow chart of a method for switching an interactive mode provided in an embodiment of the present application. The method for switching an interactive mode may include:

[0030] Step 101, when an image is acquired, determine a first position of the hand and a second position of the handle according to the image.

[0031] In step 101, an image including a hand and a handle may be captured by a camera capable of capturing and obtaining depth information, such as a fisheye camera and a depth camera.

[0032] The hand and the handle in the image may be identified based on the image. For example, a detection frame of the hand or a detection frame of the handle in the image may be identified by using a pre-trained target detection model.

[0033] The first position of the hand can be determined according to the detection frame of the hand. For example, the coordinates of the center point of the detection frame of the hand can be used as the first position of the hand, and the key points of the hand can be determined according to the hand detection frame, and the coordinates of the center point of the hand can be determined based on the coordinates of the key points, and the coordinates of the center point of the hand can be used as the first position of the hand. The specific setting can be based on actual needs and is not specifically limited here.

[0034] The second position of the handle can be determined based on the detection frame of the handle. For example, the coordinates of the center point of the detection frame of the handle can be used as the second position of the handle. The second position of the handle can also be determined based on the handle detection frame combined with the inertial measurement data of the handle. The specific setting can be based on actual needs and is not specifically limited here.

[0035] Step 102: Determine the distance between the hand and the handle according to the first position and the second position.

[0036] In step 102, the distance between the hand and the handle may be determined based on the first position and the second position determined in the image. Figure 2 As shown, the calculation formula for the distance between the hand 201 and the handle 202 can be shown as formula (1):

[0037]

[0038] Where, distance is the distance between the hand and the handle, position controller The second position of the handle, position palm This is the first position of the hand.

[0039] Step 103 : Switch the interaction mode to the target mode according to the distance and the distance threshold, where the target mode includes a gesture tracking mode or a handle tracking mode.

[0040] In step 103, the distance may be compared with a distance threshold, and whether the interaction mode is a gesture tracking mode or a handle tracking mode may be determined according to the comparison result.

[0041] The distance threshold may be a value or a range of values, which is not specifically limited here. For example, the distance threshold may be a value, such as Figure 3As shown in FIG. 1 , if the distance between the hand and the handle is less than the distance threshold, the interaction mode can be switched to the handle tracking mode. If the distance between the hand and the handle is greater than the distance threshold, the interaction mode can be switched to the gesture tracking mode. If the distance between the hand and the handle is equal to the distance threshold, the interaction mode of the previous frame can be kept unchanged.

[0042] In the embodiment of the present application, the first position of the hand and the second position of the handle can be determined based on the acquired image, and then the distance between the hand and the handle can be calculated. By comparing the distance with the distance threshold, it is determined whether the user intends to use the handle, so that the interaction mode can be switched to the gesture tracking mode or the handle tracking mode. In this way, the distance between the hand and the handle is used as the basis for switching the interaction mode, the waiting time during the switching process is reduced, and the interaction mode is quickly switched, achieving a better interaction effect.

[0043] In some embodiments, the distance threshold includes a first threshold and a second threshold, and the first threshold is less than the second threshold; switching the interaction mode to the target mode according to the distance and the distance threshold may include:

[0044] When the distance is greater than or equal to a second threshold, switching the interaction mode to a gesture tracking mode;

[0045] When the distance is less than or equal to the first threshold, the interaction mode is switched to the handle tracking mode;

[0046] When the distance is greater than the first threshold and less than the second threshold, switching of the interaction mode is prohibited.

[0047] In this embodiment, in order to avoid the situation where the interaction mode frequently and quickly switches between the gesture tracking mode and the handle tracking mode when the hand moves near the distance between the hand and the handle, thereby affecting the interaction effect, the distance threshold may include a first threshold and a second threshold, that is, the distance threshold may be a distance threshold range, and the distance threshold range may be expressed as [switch min , switch max ].

[0048] Among them, switch max It can be used as the second threshold to switch from controller tracking mode to gesture tracking mode. min It can be used as the first threshold for switching from gesture tracking mode to handle tracking mode.

[0049] For example, when the distance is greater than or equal to the second threshold, the interaction mode is switched to the gesture tracking mode. current ≥switch max , it will switch to gesture tracking mode immediately.

[0050] When the distance is less than or equal to the first threshold, the interaction mode can be switched to the handle tracking mode. current ≤switch max , it will immediately switch to controller tracking mode.

[0051] When the distance is greater than the first threshold and less than the second threshold, switching the interactive mode is prohibited. min <distance current <switch max , the current interaction mode is maintained, which can be controller tracking mode or gesture tracking mode.

[0052] In this way, by setting a buffer zone within the distance threshold range, the situation where the distance between the hand and the handle is within the distance threshold range is used as a situation where switching of the interaction mode is prohibited, thereby avoiding the phenomenon of frequent and rapid switching of the interaction mode due to the distance between the hand and the handle being near a certain critical point. While reducing the switching time of the interaction mode, the reliability of the interaction mode switching is improved to achieve better interaction effects.

[0053] In some embodiments, when the image is the i-th frame image, i is a positive integer, before switching the interaction mode to the target mode according to the distance and the distance threshold, the method may further include:

[0054] Determine the relative motion speed between the hand and the handle corresponding to the i-th frame according to the distance between the hand and the handle corresponding to the i-th frame and the distance between the hand and the handle corresponding to the i-1-th frame;

[0055] According to the relative motion speed and speed threshold corresponding to the i-th frame, the distance threshold corresponding to the i-1-th frame is adjusted to obtain the distance threshold corresponding to the i-th frame;

[0056] According to the distance and distance threshold, the interaction mode is switched to the target mode, which may include:

[0057] According to the distance between the hand and the handle corresponding to the i-th frame and the distance threshold corresponding to the i-th frame, the interaction mode is switched to the target mode.

[0058] In this embodiment, in order to facilitate understanding of the solution of the embodiment of the present application, the i-th frame image can be understood as the image of the current frame. In other words, the image can be acquired in real time to execute the interactive mode switching method provided by the embodiment of the present application in real time to determine whether the interactive mode needs to be switched at the current moment.

[0059] The relative motion speed between the hand and the handle corresponding to the i-th frame can be calculated based on the distance between the hand and the handle corresponding to the i-th frame and the distance between the hand and the handle corresponding to the i-1-th frame. For example, taking the i-th frame as the current frame and the i-1-th frame as the previous frame, the calculation formula for the relative motion speed can be shown as formula (2):

[0060]

[0061] Where velocity is the relative speed between the hand and the handle, and distance is current is the distance between the hand and the handle corresponding to the current frame, distance previous is the distance between the hand and the handle corresponding to the previous frame, and Δt is the time difference between the two frames.

[0062] The relative motion speed corresponding to the i-th frame may be compared with the speed threshold, and the distance threshold corresponding to the i-1-th frame may be adjusted according to the comparison result to obtain the distance threshold corresponding to the i-th frame.

[0063] The speed threshold may be a value or a range of values, which is not specifically limited here. For example, if the speed threshold may be a value, if the relative motion speed corresponding to the i-th frame is negative and the absolute value is greater than the speed threshold, it can be considered that the user's hand is approaching the handle quickly. At this time, it can be inferred that the user is more willing to pick up the handle and use the handle tracking mode for interaction. Based on this, the distance threshold can be increased so that the distance between the hand and the handle can be less than the distance threshold more quickly to switch to the handle tracking mode.

[0064] If the relative motion speed corresponding to the i-th frame is positive and its absolute value is greater than the speed threshold, it can be considered that the user's hand is moving away from the handle quickly. At this time, it can be guessed that the user wants to use gesture tracking mode to interact. Based on this, the distance threshold can be lowered so that the distance between the hand and the handle can reach a value greater than the distance threshold more quickly, thereby switching to gesture tracking mode.

[0065] The distance between the hand and the handle corresponding to the i-th frame may be compared with a distance threshold corresponding to the i-th frame, and whether the interaction mode is the gesture tracking mode or the handle tracking mode may be determined according to the comparison result.

[0066] In this way, the distance threshold can be adjusted according to the relative movement speed between the hand and the handle, so that the adjusted distance threshold can switch to the interaction mode that the user intends to use more quickly, thereby improving the flexibility of switching the interaction mode and further improving the interaction effect.

[0067] In some embodiments, when the distance threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and the speed threshold includes a third threshold and a fourth threshold, and the third threshold is less than the fourth threshold, the distance threshold corresponding to the i-1th frame is adjusted according to the relative motion speed and the speed threshold corresponding to the i-th frame to obtain the distance threshold corresponding to the i-th frame, which may include:

[0068] When the relative motion speed corresponding to the i-th frame is greater than or equal to the fourth threshold value and the fourth threshold value is a positive number, the second threshold value corresponding to the i-1-th frame is reduced to obtain the distance threshold value corresponding to the i-th frame;

[0069] When the relative motion speed corresponding to the i-th frame is less than or equal to the third threshold value and the third threshold value is a negative number, the first threshold value corresponding to the i-1-th frame is increased to obtain the distance threshold value corresponding to the i-th frame;

[0070] When the relative motion speed corresponding to the i-th frame is greater than the third threshold and less than the fourth threshold, the distance threshold corresponding to the (i-1)-th frame is determined as the distance threshold corresponding to the i-th frame.

[0071] In this embodiment, the distance threshold may include a first threshold and a second threshold, that is, the distance threshold may be a distance threshold range, and the distance threshold range may be expressed as [switch min , switch max ], correspondingly, the speed threshold may include a third threshold and a fourth threshold, that is, the speed threshold may be a speed threshold range, and the speed threshold range may be expressed as [velocity min , velocity max ].

[0072] Among them, velocity max Can be used as the second threshold switch in adjusting the distance threshold max The fourth threshold, velocity min Can be used as the first threshold switch in adjusting the distance threshold range min The third threshold value.

[0073] For example, when the relative motion speed corresponding to the i-th frame is greater than or equal to the fourth threshold and the fourth threshold is a positive number, the second threshold corresponding to the i-1-th frame can be reduced to obtain the distance threshold corresponding to the i-th frame. max is a positive number, velocity ≥ velocity max, it can be considered that the user's hand is moving away from the handle quickly. At this time, it can be guessed that the user wants to use the gesture tracking mode for interaction. Based on this, the second threshold corresponding to the i-1th frame can be lowered to obtain the distance threshold corresponding to the i-th frame, so that the distance between the hand and the handle can reach a value greater than the second threshold corresponding to the i-th frame more quickly, thereby switching to the gesture tracking mode.

[0074] When the relative motion speed corresponding to the i-th frame is less than the third threshold and the third threshold is a negative number, the first threshold corresponding to the i-1-th frame can be increased to obtain the distance threshold corresponding to the i-th frame. min is a negative number, velocity≤velocity min In fact, the absolute value of velocity is still greater than velocity min If the absolute value of is greater than , it can be considered that the user's hand is approaching the handle quickly. At this time, it can be guessed that the user is more willing to pick up the handle and use the handle tracking mode to interact. Based on this, the first threshold corresponding to the i-1th frame can be increased to obtain the distance threshold corresponding to the i-th frame, so that the distance between the hand and the handle can be smaller than the first threshold corresponding to the i-th frame more quickly, and the purpose of switching to the handle tracking mode is achieved.

[0075] When the relative motion speed corresponding to the i-th frame is greater than the third threshold and less than the fourth threshold, the distance threshold corresponding to the i-1-th frame may be determined as the distance threshold corresponding to the i-th frame. In this case, the distance threshold may not be adjusted, and whether the interaction mode needs to be switched and which interaction mode to switch to may be determined based on the distance threshold corresponding to the previous frame.

[0076] In this way, the relative movement speed between the hand and the handle can be compared with the speed threshold, and the distance threshold can be adjusted according to the comparison result, so that the adjusted distance threshold can switch to the interaction mode that the user intends to use more quickly, thereby improving the flexibility of switching the interaction mode and further improving the interaction effect.

[0077] In some embodiments, determining the first position of the hand according to the image may include:

[0078] Perform hand detection on the image to obtain a hand detection frame;

[0079] Identify the key points of the hand within the hand detection frame and obtain the coordinates of the key points of the hand;

[0080] According to the coordinates of the key points of the hand, determine the coordinates of the center point of the hand;

[0081] The coordinates of the center point of the hand are determined as the first position.

[0082] In this embodiment, hand detection can be performed on the image to obtain a hand detection frame. For example, Figure 4 As shown, taking an image taken by a fisheye camera as an example, the image taken by the fisheye camera can be used as input and input into the hand frame detection network. The number of fisheye cameras can be one, or multiple fisheye cameras, that is, the number is multiple. Taking two fisheye cameras as an example, the images 401 of the two fisheye cameras can be input into the hand frame detection network. The hand frame detection network can select the hand frame in the image to obtain a hand detection frame, and can cut out the hand detection frame according to the position and size information of the detection frame to obtain a hand detection frame 402. In some examples, the hand frame detection network can also classify the hands to determine the left and right hands.

[0083] The hand detection frame 402 can be input into the hand key point detection network, and the hand key point detection network can detect the key points 403 of the hand. Figure 5 As shown, for each palm 501, four joint points can be marked at the joints and fingertips, plus the wrist key point, the hand can have twenty-one key points 502. It can be understood that the position and number of the key points are only used as an example, and the position and number of the key points can be adjusted according to actual needs, and are not specifically limited here.

[0084] The hand key point detection network can output the horizontal and vertical coordinates of the key point 502 on the image, as well as the depth coordinate relative to the wrist key point. The three-dimensional posture 404 of the hand can be reconstructed based on the three-axis coordinates of the key point 502, and the coordinates of the center point of the hand can be calculated.

[0085] In some examples, taking the key point 502 including the wrist key point 5021, the index finger root key point 5022, the middle finger root key point 5023, the ring finger root key point 5024 and the little finger root key point 5025 as an example, the coordinates of the five key points can be calculated by averaging to obtain the coordinates of the center point of the hand, and the coordinates of the center point of the hand are used as the first position of the hand. The calculation formula of the first position is shown in formula (3):

[0086]

[0087] Among them, position palm is the first position of the hand, is the coordinate of the i-th key point.

[0088] In this way, hand detection can be performed on the image to obtain a hand detection frame, and the hand key points within the hand detection frame can be identified to obtain the coordinates of the hand key points, thereby calculating more accurate coordinates of the center point of the hand, thereby improving the accuracy of the first position and thereby improving the reliability of the basis for switching the interactive mode.

[0089] In some embodiments, determining the second position of the handle according to the image may include:

[0090] According to the image, determine the visual position information of the handle relative to the camera;

[0091] Get the inertial measurement unit data of the controller;

[0092] Determine the inertial position information of the handle according to the inertial measurement unit data;

[0093] The second position of the handle is determined according to the visual position information and the inertial posture information.

[0094] In this embodiment, the visual position information of the handle relative to the camera can be determined based on the image. For example, the image can be input into a target detection network to identify the position of the handle, thereby obtaining a handle detection frame, and the visual position information of the handle relative to the camera can be calculated based on the handle detection frame.

[0095] Take the light ring handle as an example. The light ring handle has a large ring on the outside of the handle, on which LED lights are installed at a certain density, so that the camera can capture the light information of the LED lights to help locate the position of the handle. At the same time, the light ring handle is generally equipped with an inertial measurement unit (IMU), which can also be used to calculate the position of the handle.

[0096] like Figure 6 As shown, the image 601 of the handle can be obtained by a fisheye camera, the position of the LED light on the image can be extracted from the image, and then the visual position information 602 of the handle relative to the camera can be calculated by a posture estimation (Perspective-n-Point, PnP) algorithm. Among them, the PnP algorithm infers the rotation matrix R and translation vector t of the camera through n known three-dimensional space points and their projection positions in the camera coordinate system. These three-dimensional space points are usually obtained in advance through other methods, such as triangulation, camera depth map, etc. The goal of the PnP algorithm is to find a preferred transformation so that the error between the projection position of these three-dimensional space points in the image coordinate system after transformation and the actual projection position is small enough.

[0097] The inertial posture information of the handle can also be determined based on the inertial measurement unit data. For example, the inertial posture information 604 can be calculated based on the IMU data 603 by the IMU integration algorithm. Among them, the IMU usually includes sensors such as a gyroscope and an accelerometer, so the IMU data can include acceleration and angular velocity, etc. The integral of acceleration is velocity, and the integral of velocity is displacement. Therefore, by integrating the linear acceleration, the velocity change of the handle in a certain time period can be obtained, and then the displacement can be obtained. The integral of the angular velocity is the angle change. By integrating the angular velocity, the rotation angle of the handle in a certain time period can be obtained. The rotation angle can be used to update the posture information of the handle, and the updated posture information is combined with the displacement information to obtain the inertial posture information of the handle.

[0098] According to the visual position information and the inertial posture information, an extended Kalman filter (EKF) fusion algorithm is used to fuse them to obtain a second position 605 of the handle.

[0099] In this way, the visual position information and inertial posture information of the handle can be combined to obtain more accurate handle position information, thereby improving the accuracy of the second position and further improving the reliability of the basis for switching the interactive mode.

[0100] The interactive mode switching method provided in the embodiment of the present application can be executed by an interactive mode switching device. In the embodiment of the present application, the interactive mode switching method performed by the interactive mode switching device is taken as an example to illustrate the interactive mode switching device provided in the embodiment of the present application.

[0101] like Figure 7 As shown, the embodiment of the present application provides a switching device 700 for an interactive mode, and the switching device 700 for an interactive mode may include:

[0102] A first determination module 701 is used to determine a first position of the hand and a second position of the handle according to the image when the image is acquired;

[0103] A second determination module 702, configured to determine the distance between the hand and the handle according to the first position and the second position;

[0104] The switching module 703 is used to switch the interaction mode to the target mode according to the distance and the distance threshold, and the target mode includes the gesture tracking mode or the handle tracking mode.

[0105] In the embodiment of the present application, the first position of the hand and the second position of the handle can be determined based on the acquired image, and then the distance between the hand and the handle can be calculated. By comparing the distance with the distance threshold, it is determined whether the user intends to use the handle, so that the interaction mode can be switched to the gesture tracking mode or the handle tracking mode. In this way, the distance between the hand and the handle is used as the basis for switching the interaction mode, the waiting time during the switching process is reduced, and the interaction mode is quickly switched, achieving a better interaction effect.

[0106] In some embodiments, the distance threshold includes a first threshold and a second threshold, and the first threshold is less than the second threshold. The switching module 703 may also be used to:

[0107] When the distance is greater than or equal to a second threshold, switching the interaction mode to a gesture tracking mode;

[0108] When the distance is less than or equal to the first threshold, the interaction mode is switched to the handle tracking mode;

[0109] When the distance is greater than the first threshold and less than the second threshold, switching of the interaction mode is prohibited.

[0110] In this way, by setting a buffer zone within the distance threshold range, the situation where the distance between the hand and the handle is within the distance threshold range is used as a situation where switching of the interaction mode is prohibited, thereby avoiding the phenomenon of frequent and rapid switching of the interaction mode due to the distance between the hand and the handle being near a certain critical point. While reducing the switching time of the interaction mode, the reliability of the interaction mode switching is improved to achieve better interaction effects.

[0111] In some embodiments, when the image is the i-th frame image and i is a positive integer, the interactive mode switching device 700 may further include:

[0112] A third determination module is used to determine the relative motion speed between the hand and the handle corresponding to the i-th frame according to the distance between the hand and the handle corresponding to the i-th frame and the distance between the hand and the handle corresponding to the i-1-th frame before switching the interaction mode to the target mode according to the distance and the distance threshold;

[0113] An adjustment module, used for adjusting the distance threshold corresponding to the i-1th frame according to the relative motion speed and the speed threshold corresponding to the i-th frame, to obtain the distance threshold corresponding to the i-th frame;

[0114] The switching module 703 is further used for:

[0115] According to the distance between the hand and the handle corresponding to the i-th frame and the distance threshold corresponding to the i-th frame, the interaction mode is switched to the target mode.

[0116] In this way, the distance threshold can be adjusted according to the relative movement speed between the hand and the handle, so that the adjusted distance threshold can switch to the interaction mode that the user intends to use more quickly, thereby improving the flexibility of switching the interaction mode and further improving the interaction effect.

[0117] In some embodiments, when the distance threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and the speed threshold includes a third threshold and a fourth threshold, and the third threshold is less than the fourth threshold, the adjustment module may also be used to:

[0118] When the relative motion speed corresponding to the i-th frame is greater than or equal to the fourth threshold value and the fourth threshold value is a positive number, the second threshold value corresponding to the i-1-th frame is reduced to obtain the distance threshold value corresponding to the i-th frame;

[0119] When the relative motion speed corresponding to the i-th frame is less than or equal to the third threshold value and the third threshold value is a negative number, the first threshold value corresponding to the i-1-th frame is increased to obtain the distance threshold value corresponding to the i-th frame;

[0120] When the relative motion speed corresponding to the i-th frame is greater than the third threshold and less than the fourth threshold, the distance threshold corresponding to the (i-1)-th frame is determined as the distance threshold corresponding to the i-th frame.

[0121] In this way, the relative movement speed between the hand and the handle can be compared with the speed threshold, and the distance threshold can be adjusted according to the comparison result, so that the adjusted distance threshold can switch to the interaction mode that the user intends to use more quickly, thereby improving the flexibility of switching the interaction mode and further improving the interaction effect.

[0122] In some embodiments, the first determining module 701 may also be used to:

[0123] Perform hand detection on the image to obtain a hand detection frame;

[0124] Identify the key points of the hand within the hand detection frame and obtain the coordinates of the key points of the hand;

[0125] According to the coordinates of the key points of the hand, determine the coordinates of the center point of the hand;

[0126] The coordinates of the center point of the hand are determined as the first position.

[0127] In this way, hand detection can be performed on the image to obtain a hand detection frame, and the hand key points within the hand detection frame can be identified to obtain the coordinates of the hand key points, thereby calculating more accurate coordinates of the center point of the hand, thereby improving the accuracy of the first position and thereby improving the reliability of the basis for switching the interactive mode.

[0128] In some embodiments, the first determining module 701 may also be used to:

[0129] According to the image, determine the visual position information of the handle relative to the camera;

[0130] Get the inertial measurement unit data of the controller;

[0131] Determine the inertial position information of the handle according to the inertial measurement unit data;

[0132] The second position of the handle is determined according to the visual position information and the inertial posture information.

[0133] In this way, the visual position information and inertial posture information of the handle can be combined to obtain more accurate handle position information, thereby improving the accuracy of the second position and further improving the reliability of the basis for switching the interactive mode.

[0134] The switching device of the interactive mode in the embodiment of the present application can be a device, or a component in the device, such as an integrated circuit or a chip. The device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0135] The interactive mode switching device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0136] The switching device of the interactive mode provided in the embodiment of the present application can realize Figures 1 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0137] Alternatively, if Figure 8As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned interactive mode switching method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0138] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0139] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0140] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0141] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0142] The processor 910 may be used for:

[0143] When the image is acquired, determining a first position of the hand and a second position of the handle according to the image;

[0144] determining a distance between the hand and the handle according to the first position and the second position;

[0145] According to the distance and the distance threshold, the interaction mode is switched to the target mode, which includes the gesture tracking mode or the handle tracking mode.

[0146] In the embodiment of the present application, the first position of the hand and the second position of the handle can be determined based on the acquired image, and then the distance between the hand and the handle can be calculated. By comparing the distance with the distance threshold, it is determined whether the user intends to use the handle, so that the interaction mode can be switched to the gesture tracking mode or the handle tracking mode. In this way, the distance between the hand and the handle is used as the basis for switching the interaction mode, the waiting time during the switching process is reduced, and the interaction mode is quickly switched, achieving a better interaction effect.

[0147] In some embodiments, the distance threshold includes a first threshold and a second threshold, and the first threshold is less than the second threshold. The processor 910 may also be configured to:

[0148] When the distance is greater than or equal to a second threshold, switching the interaction mode to a gesture tracking mode;

[0149] When the distance is less than or equal to the first threshold, the interaction mode is switched to the handle tracking mode;

[0150] When the distance is greater than the first threshold and less than the second threshold, switching of the interaction mode is prohibited.

[0151] In this way, by setting a buffer zone within the distance threshold range, the situation where the distance between the hand and the handle is within the distance threshold range is used as a situation where switching of the interaction mode is prohibited, thereby avoiding the phenomenon of frequent and rapid switching of the interaction mode due to the distance between the hand and the handle being near a certain critical point. While reducing the switching time of the interaction mode, the reliability of the interaction mode switching is improved to achieve better interaction effects.

[0152] In some embodiments, when the image is the i-th frame image and i is a positive integer, the processor 910 may further be configured to:

[0153] According to the distance and the distance threshold, before switching the interaction mode to the target mode, the relative motion speed between the hand and the handle corresponding to the i-th frame is determined according to the distance between the hand and the handle corresponding to the i-th frame and the distance between the hand and the handle corresponding to the i-1-th frame;

[0154] According to the relative motion speed and speed threshold corresponding to the i-th frame, the distance threshold corresponding to the i-1-th frame is adjusted to obtain the distance threshold corresponding to the i-th frame;

[0155] According to the distance between the hand and the handle corresponding to the i-th frame and the distance threshold corresponding to the i-th frame, the interaction mode is switched to the target mode.

[0156] In this way, the distance threshold can be adjusted according to the relative movement speed between the hand and the handle, so that the adjusted distance threshold can switch to the interaction mode that the user intends to use more quickly, thereby improving the flexibility of switching the interaction mode and further improving the interaction effect.

[0157] In some embodiments, when the distance threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and the speed threshold includes a third threshold and a fourth threshold, and the third threshold is less than the fourth threshold, the processor 910 may also be configured to:

[0158] When the relative motion speed corresponding to the i-th frame is greater than or equal to the fourth threshold value and the fourth threshold value is a positive number, the second threshold value corresponding to the i-1-th frame is reduced to obtain the distance threshold value corresponding to the i-th frame;

[0159] When the relative motion speed corresponding to the i-th frame is less than or equal to the third threshold value and the third threshold value is a negative number, the first threshold value corresponding to the i-1-th frame is increased to obtain the distance threshold value corresponding to the i-th frame;

[0160] When the relative motion speed corresponding to the i-th frame is greater than the third threshold and less than the fourth threshold, the distance threshold corresponding to the (i-1)-th frame is determined as the distance threshold corresponding to the i-th frame.

[0161] In this way, the relative movement speed between the hand and the handle can be compared with the speed threshold, and the distance threshold can be adjusted according to the comparison result, so that the adjusted distance threshold can switch to the interaction mode that the user intends to use more quickly, thereby improving the flexibility of switching the interaction mode and further improving the interaction effect.

[0162] In some embodiments, the processor 910 may also be configured to:

[0163] Perform hand detection on the image to obtain a hand detection frame;

[0164] Identify the key points of the hand within the hand detection frame and obtain the coordinates of the key points of the hand;

[0165] According to the coordinates of the key points of the hand, determine the coordinates of the center point of the hand;

[0166] The coordinates of the center point of the hand are determined as the first position.

[0167] In this way, hand detection can be performed on the image to obtain a hand detection frame, and the hand key points within the hand detection frame can be identified to obtain the coordinates of the hand key points, thereby calculating more accurate coordinates of the center point of the hand, thereby improving the accuracy of the first position and thereby improving the reliability of the basis for switching the interactive mode.

[0168] In some embodiments, the processor 910 may also be configured to:

[0169] According to the image, determine the visual position information of the handle relative to the camera;

[0170] Get the inertial measurement unit data of the controller;

[0171] Determine the inertial position information of the handle according to the inertial measurement unit data;

[0172] The second position of the handle is determined according to the visual position information and the inertial posture information.

[0173] In this way, the visual position information and inertial posture information of the handle can be combined to obtain more accurate handle position information, thereby improving the accuracy of the second position and further improving the reliability of the basis for switching the interactive mode.

[0174] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0175] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory 909 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0176] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0177] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned interactive mode switching method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0178] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0179] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned interactive mode switching method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0180] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0181] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned interactive mode switching method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0182] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0183] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0184] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for switching an interactive mode, characterized in that: The method comprises: When the image is acquired, determining a first position of the hand and a second position of the handle according to the image; determining a distance between the hand and the handle according to the first position and the second position; According to the distance and the distance threshold, the interaction mode is switched to a target mode, where the target mode includes a gesture tracking mode or a handle tracking mode.

2. The method according to claim 1, characterized in that The distance threshold includes a first threshold and a second threshold, and the first threshold is smaller than the second threshold; The step of switching the interaction mode to the target mode according to the distance and the distance threshold includes: When the distance is greater than or equal to the second threshold, switching the interaction mode to the gesture tracking mode; When the distance is less than or equal to the first threshold, switching the interaction mode to the handle tracking mode; When the distance is greater than the first threshold and less than the second threshold, switching of the interaction mode is prohibited.

3. The method according to claim 1, characterized in that When the image is the i-th frame image and i is a positive integer, before switching the interaction mode to the target mode according to the distance and the distance threshold, the method further includes: Determine the relative motion speed between the hand and the handle corresponding to the i-th frame according to the distance between the hand and the handle corresponding to the i-th frame and the distance between the hand and the handle corresponding to the (i-1)-th frame; According to the relative motion speed and speed threshold corresponding to the i-th frame, the distance threshold corresponding to the i-1-th frame is adjusted to obtain the distance threshold corresponding to the i-th frame; The step of switching the interaction mode to the target mode according to the distance and the distance threshold includes: The interaction mode is switched to the target mode according to the distance between the hand and the handle corresponding to the i-th frame and the distance threshold corresponding to the i-th frame.

4. The method according to claim 3, characterized in that In the case where the distance threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and the speed threshold includes a third threshold and a fourth threshold, and the third threshold is less than the fourth threshold, adjusting the distance threshold corresponding to the i-1th frame according to the relative motion speed and the speed threshold corresponding to the i-th frame to obtain the distance threshold corresponding to the i-th frame includes: When the relative motion speed corresponding to the i-th frame is greater than or equal to the fourth threshold and the fourth threshold is a positive number, lowering the second threshold corresponding to the i-1-th frame to obtain a distance threshold corresponding to the i-th frame; When the relative motion speed corresponding to the i-th frame is less than or equal to the third threshold and the third threshold is a negative number, increasing the first threshold corresponding to the (i-1)-th frame to obtain a distance threshold corresponding to the i-th frame; When the relative motion speed corresponding to the i-th frame is greater than the third threshold and less than the fourth threshold, the distance threshold corresponding to the (i-1)-th frame is determined as the distance threshold corresponding to the i-th frame.

5. The method according to claim 1, characterized in that Determining a first position of the hand according to the image includes: Performing hand detection on the image to obtain a hand detection frame; Identify the key points of the hand within the hand detection frame and obtain the coordinates of the key points of the hand; Determine the coordinates of the center point of the hand according to the coordinates of the key points of the hand; The coordinates of the center point of the hand are determined as the first position.

6. The method according to claim 1, characterized in that Determining a second position of the handle according to the image includes: Determine visual position information of the handle relative to the camera based on the image; Acquiring inertial measurement unit data of the handle; Determining inertial position information of the handle according to the inertial measurement unit data; A second position of the handle is determined according to the visual position information and the inertial posture information.

7. A switching device for an interactive mode, characterized in that: The device comprises: A first determination module is used to determine a first position of the hand and a second position of the handle according to the image when the image is acquired; A second determining module, configured to determine a distance between the hand and the handle according to the first position and the second position; The switching module is used to switch the interaction mode to a target mode according to the distance and the distance threshold, and the target mode includes a gesture tracking mode or a handle tracking mode.

8. The device according to claim 7, characterized in that The distance threshold includes a first threshold and a second threshold, and the first threshold is less than the second threshold; the switching module is further used for: When the distance is greater than or equal to the second threshold, switching the interaction mode to the gesture tracking mode; When the distance is less than or equal to the first threshold, switching the interaction mode to the handle tracking mode; When the distance is greater than the first threshold and less than the second threshold, switching of the interaction mode is prohibited.

9. The device according to claim 7, characterized in that In the case where the image is an i-th frame image, and i is a positive integer, the device further includes: A third determining module is used to determine the relative motion speed between the hand and the handle corresponding to the i-th frame according to the distance between the hand and the handle corresponding to the i-th frame and the distance between the hand and the handle corresponding to the (i-1)-th frame; An adjustment module, configured to adjust the distance threshold corresponding to the i-1th frame according to the relative motion speed and the speed threshold corresponding to the i-th frame, so as to obtain the distance threshold corresponding to the i-th frame; The switching module is further used for: The interaction mode is switched to the target mode according to the distance between the hand and the handle corresponding to the i-th frame and the distance threshold corresponding to the i-th frame.

10. The device according to claim 9, characterized in that In the case where the distance threshold includes a first threshold and a second threshold, the first threshold is less than the second threshold, and the speed threshold includes a third threshold and a fourth threshold, and the third threshold is less than the fourth threshold, the adjustment module is further configured to: When the relative motion speed corresponding to the i-th frame is greater than or equal to the fourth threshold value and the fourth threshold value is a positive number, the second threshold value corresponding to the i-1-th frame is reduced to obtain a distance threshold value corresponding to the i-th frame; When the relative motion speed corresponding to the i-th frame is less than or equal to the third threshold value and the third threshold value is a negative number, the first threshold value corresponding to the i-1-th frame is increased to obtain a distance threshold value corresponding to the i-th frame; When the relative motion speed corresponding to the i-th frame is greater than the third threshold and less than the fourth threshold, the distance threshold corresponding to the (i-1)-th frame is determined as the distance threshold corresponding to the i-th frame.

11. The device according to claim 7, characterized in that The first determining module is further used for: Performing hand detection on the image to obtain a hand detection frame; Identify the key points of the hand within the hand detection frame and obtain the coordinates of the key points of the hand; Determine the coordinates of the center point of the hand according to the coordinates of the key points of the hand; The coordinates of the center point of the hand are determined as the first position.

12. The device according to claim 7, characterized in that The first determining module is further used for: Determine visual position information of the handle relative to the camera based on the image; Acquiring inertial measurement unit data of the handle; Determining the inertial position information of the handle according to the inertial measurement unit data; A second position of the handle is determined according to the visual position information and the inertial posture information.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product, characterized in that The program product is stored in a storage medium, and the program product is executed by at least one processor to implement the steps of the method according to any one of claims 1 to 6.

Images