Interaction control method and apparatus, wearable device, and storage medium

Abstract

The application discloses an interactive control method and device, a wearable device and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: in response to a first preset gesture, detecting the relative position of a target virtual object; if an overlapping area is detected with the target virtual object, displaying a first preset halo in the overlapping area; if the first preset halo is detected in a target control area, superimposing and displaying a control component on the target virtual object; and if the input target gesture is a preset gesture in a preset control mode associated with a target cursor, controlling the target virtual object to perform a preset operation corresponding to the target gesture operation. In this way, the user performs corresponding control operations in different preset control areas through the cooperation of gestures and halos displayed on virtual objects, so that the control functions that can be realized by gesture operations are more abundant, the user experience is improved, and the precision of interactive control is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to an interaction control method and device, a wearable device and a storage medium. BACKGROUND

[0002] With the progress of intelligent technology, intelligent wearable devices gradually enter our life, and with the development of extended reality (XR) technology, an interaction feedback information loop is built between the virtual world, the real world and the user. When the user wears a head-mounted display, the user can use his / her hand to make a gesture and further trigger a specific function, which can be related to hardware or software control. It is easy for the user to use his / her hand to control the head-mounted display system.

[0003] In the prior art, when the user interacts with the virtual scene through gesture operation, the gesture setting is complex, which is not conducive to the recognition of the device, and the operation function that can be realized by the gesture set by the device is single, which is inconvenient to use, and the control object in the virtual scene cannot be accurately interacted, and the user experience is poor. SUMMARY

[0004] The present application provides an interaction control method and device, a wearable device and a storage medium.

[0005] In a first aspect, an interaction control method is provided, which is applied to a wearable device, and the method comprises: in response to a first preset gesture, detecting a relative position between the first preset gesture and a target virtual object; if it is detected that there is an overlapping area between the first preset gesture and the target virtual object, displaying a first preset halo in the overlapping area; if it is detected that the first preset halo is in a target control area in a plurality of preset control areas, superimposing and displaying a control component on the target virtual object, the control component comprising a preset cursor and a preset bounding box corresponding to each preset control area, and the preset cursor corresponding to each preset control area being associated with a preset control mode; if it is detected that an input target gesture is a preset gesture in a preset control mode associated with a target cursor, controlling the target virtual object to perform a preset operation corresponding to the target gesture operation, the target cursor being the preset cursor corresponding to any preset control area in the plurality of preset control areas.

[0006] In a second aspect, the embodiments of the present application provide an interaction control device, applied to a wearable device, the device comprising: a position detection module configured to detect a relative position between a first preset gesture and a target virtual object in response to the first preset gesture; a halo display module configured to display a first preset halo in an overlapping area between the first preset gesture and the target virtual object if the overlapping area is detected; a control component display module configured to display a control component on the target virtual object if the first preset halo is detected in a target control area in a plurality of preset control areas, the control component comprising a preset cursor and a preset boundary box corresponding to each preset control area, and each preset cursor corresponding to a preset control area being associated with a preset control mode; and a control module configured to control the target virtual object to perform a preset operation corresponding to a target gesture operation if the target gesture is detected as a preset gesture in the preset control mode associated with a target cursor, the target cursor being the preset cursor corresponding to any preset control area in the plurality of preset control areas.

[0007] In a third aspect, the embodiments of the present application provide a wearable device, comprising: one or more processors; a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the method described above.

[0008] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, wherein the computer readable storage medium stores program codes, and the program codes can be invoked by a processor to perform the method described above.

[0009] In the scheme provided in the application, in response to the first preset gesture, the relative position of the first preset gesture and the target virtual object is detected; if it is detected that there is an overlapping area between the first preset gesture and the target virtual object, a first preset halo is displayed in the overlapping area; if it is detected that the first preset halo is in a target control area in the plurality of preset control areas, a control component is superimposed and displayed on the target virtual object, the control component includes a preset cursor and a preset boundary box corresponding to each preset control area, and the preset cursor corresponding to each preset control area is associated with a preset control mode; if it is detected that the input target gesture is a preset gesture in the preset control mode associated with the target cursor, the target virtual object is controlled to perform a preset operation corresponding to the target gesture operation, and the target cursor is a preset cursor corresponding to any preset control area in the plurality of preset control areas. In this way, the user selects a virtual object in a virtual scene through a gesture operation, and through the cooperation between the user gesture and the halo superimposed and displayed on the virtual object, different control operations are performed in different preset control areas. The user can perform the corresponding preset operation only when the hand cooperates to make the preset gesture according to the prompt of the control component superimposed and displayed on the virtual object, so that the control function that can be realized by the user through the gesture operation is more rich, and the user experience is improved. At the same time, the limitation of the preset control area improves the precision and accuracy of the interactive control, and reduces the occurrence of problems such as miscontrol. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0011] Figure 1 A flowchart of an interactive control method provided by an embodiment of the application is shown.

[0012] Figure 2 A schematic diagram of a preset gesture provided by an embodiment of the application is shown.

[0013] Figure 3 A schematic diagram of a preset halo provided by an embodiment of the application is shown.

[0014] Figure 4 A schematic diagram of a preset control area provided by an embodiment of the application is shown.

[0015] Figure 5 A schematic diagram of a preset cursor provided by an embodiment of the application is shown.

[0016] Figure 6 A scene schematic diagram of a virtual scene in an embodiment is shown.

[0017] Figure 7 A scene schematic diagram of another virtual scene of the embodiment is shown.

[0018] Figure 8 A scene schematic diagram of another virtual scene of the embodiment is shown.

[0019] Figure 9 A scene schematic diagram of another virtual scene of the embodiment is shown.

[0020] Figure 10 A scene schematic diagram of another virtual scene of the embodiment is shown.

[0021] Figure 11 A flow schematic diagram of an interaction control method provided by another embodiment of the application is shown.

[0022] Figure 12 A flow schematic diagram of an interaction control method provided by an embodiment of the application is shown.

[0023] Figure 13 A structure block diagram of an interaction control device provided by an embodiment of the application is shown.

[0024] Figure 14 A structure block diagram of a wearable device provided by an embodiment of the application is shown.

[0025] Figure 15 A structure block diagram of a computer readable storage medium provided by an embodiment of the application is shown. DETAILED DESCRIPTION

[0026] In order to enable persons skilled in the art to better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative labor shall fall within the scope of protection of the present application.

[0027] It should be noted that in some of the flowcharts described in the specification, claims and drawings of the present application, multiple operations are described in a specific order. These operations can be performed in the order described or in parallel. The serial numbers of the operations, such as S110, S120, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these flowcharts can include more or fewer operations, and the operations can be executed in sequence or in parallel. In addition, the terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or sub-modules does not necessarily limit to those clearly listed steps or sub-modules, but can include other steps or sub-modules not clearly listed or inherent to these processes, methods, products or devices.

[0028] In the related art, when a user interacts with a virtual scene through a gesture operation, the conventional gesture operation is complex, has a large motion amplitude, the interaction process is complex, and it is difficult to perform high-precision interaction control. In addition, the operation function that can be achieved by the gesture set by the device is single, and the use is not convenient. The control object in the virtual scene cannot be accurately interacted, and the user experience is poor.

[0029] To solve the above problems, the inventors propose an interaction control method, device, wearable device and storage medium. By making a preset gesture when selecting a virtual object in a preset area, the difficulty of gesture interaction is simplified, the interaction operation function is diversified, and the interaction control precision is improved. The interaction control method provided by the embodiments of the present application will be described in detail below.

[0030] Please refer to Figure 1 , Figure 1 The flowchart of the interaction control method provided by an embodiment of the present application is shown, which is applied to a wearable device. The interaction control method provided by the embodiments of the present application will be described in detail below. Figure 1 The interaction control method provided by the embodiments of the present application can include the following steps:

[0031] Step S110: in response to a first preset gesture, detecting the relative position of the first preset gesture and a target virtual object.

[0032] In the embodiment, the interaction control device detects the relative position between the first preset gesture made by the user and the target virtual object in the virtual scene when it is confirmed that the gesture made by the user is the first preset gesture, to detect whether the target virtual object can be controlled by the user when the user makes the gesture.

[0033] In some embodiments, when the user starts the interaction control device and enters the virtual scene through the head-mounted device, a plurality of preset virtual objects are displayed in the virtual scene, and any one of the plurality of preset virtual objects can be selected by the user as a target virtual object available for interaction.

[0034] The wearable device includes, but is not limited to, a mixed reality (MR) head-mounted device, and can also be a virtual reality (VR) head-mounted device, an augmented reality (AR) head-mounted device, etc. The displayed virtual scene can include a 3DOF (Degree Of Freedom) scene and a 6DOF scene, where DOF is a degree of freedom. When the displayed virtual scene is a 3DOF scene, the virtual objects in the virtual scene have a 3-degree-of-freedom rotation angle, and when the displayed virtual scene is a 6DOF scene, the virtual objects in the virtual scene have 3 degrees of freedom related to position in addition to the 3-degree-of-freedom rotation angle. That is, when the user performs a preset operation on the target virtual object in the virtual scene through the interaction control device, the target virtual object can perform a two-dimensional rotation operation, a movement operation, a zoom-out operation, and a zoom-in operation when the virtual scene is a 3DOF scene, without involving spatial changes in up-down, front-back, and left-right directions.

[0035] Based on this, before detecting the relative position between the first preset gesture and the target virtual object, the wearable device needs to collect the gesture made by the user and input the collected gesture image into the interaction control device, the interaction control device takes the gesture currently made by the user as a target gesture, and compares the target gesture with a plurality of stored preset gestures.

[0036] Optionally, the plurality of preset gestures at least include an open hand gesture, a fist gesture, and a pinch gesture. Please refer to Figure 2 , Figure 2 A schematic diagram of a preset gesture provided by an embodiment of the application is shown.

[0037] In the embodiment, the wearable device can pre-store a plurality of preset gestures, and the interaction control apparatus can collect the gesture made by the user through an image collection module configured by the interaction control apparatus or an external image collection module in communication connection with the interaction control apparatus. The open hand gesture refers to a gesture in which five fingers are unfolded and the back of the hand appears. The first preset gesture can be the open hand gesture. When the current gesture made by the user is the open hand gesture, the wearable device collects the gesture made by the user and inputs the gesture into the interaction control apparatus. When the first preset gesture stored in the interaction control apparatus is successfully matched, it is determined that the current gesture made by the user is consistent with the first preset gesture. The fist gesture refers to a gesture in which the back of the hand is closed and five fingers are bent. The pinch gesture refers to a gesture in which the index finger and the thumb are pinched.

[0038] Step S120: If it is detected that the first preset gesture overlaps with the target virtual object, a first preset halo is displayed in the overlapping area.

[0039] In the embodiment, if the interaction control apparatus detects that the first preset gesture made by the user enters the interactive area range of the target virtual object, it is determined that the first preset gesture made by the user overlaps with the target virtual object. The interaction control apparatus displays the first preset halo in the overlapping area of the first preset gesture made by the user and the target virtual object.

[0040] Optionally, the area in the boundary range of the target virtual object is the interactive area between the user and the target virtual object. The interaction control apparatus detects the relative position of the first preset gesture and the target virtual object, that is, whether the gesture made by the user enters the interactive area range of the target virtual object. When the first preset gesture made by the user enters the interactive area range of the target virtual object, the user can control the target virtual object through gesture operation.

[0041] In some embodiments, the interaction control apparatus can pre-store a plurality of preset halos. Please refer to Figure 3 , Figure 3 A schematic diagram of the preset halo provided by the embodiment of the application is shown. As shown in Figure 3 , the preset halo pre-stored by the interaction control apparatus at least includes a first preset halo and a second preset halo.

[0042] The display effect of the preset halo is controlled and implemented by a particle system, which represents a technique for simulating some specific fuzzy phenomena in three-dimensional computer graphics, and the phenomena are difficult to achieve realistic physical motion rules by using other traditional rendering techniques, so that the display effect of the halo is dynamic. In the embodiment, the preset halo is set as a halo style of a sphere, the brightness between the sphere center and the sphere surface is changed from bright to dark, and the maximum radius and the minimum radius of the halo are set, the halo is displayed in the virtual scene from small to large, and then from large to small in a cycle, and the time for completing each cycle is set as a preset time, so that the breathing effect of the halo can be achieved.

[0043] In some embodiments, the sphere radius of the first preset halo can be set to be greater than the sphere radius of the second preset halo, that is, the ratio of the maximum radius to the minimum radius of the first preset halo is the same as the ratio of the maximum radius to the minimum radius of the second preset halo, the maximum radius of the first preset halo is greater than the maximum radius of the second preset halo, and the minimum radius of the first preset halo is also greater than the minimum radius of the second preset halo. Further, the colors of the first preset halo and the second preset halo are set to be different to further distinguish them.

[0044] Step S130: If it is detected that the first preset halo is in a target control region of the plurality of preset control regions, a control component is superimposed and displayed on the target virtual object, the control component includes a preset cursor and a preset bounding box corresponding to each preset control region, and the preset cursor corresponding to each preset control region is associated with a preset control mode.

[0045] In the embodiment, the interactive region of the target virtual object is divided into a plurality of preset control regions, and when it is detected that the first preset halo is in different control regions of the plurality of preset control regions, a control component corresponding to the control region is superimposed and displayed on the target virtual object, the control component is a preset cursor and a preset bounding box corresponding to each preset control region, and the preset cursor is associated with a preset control mode. By superimposing and displaying the preset bounding box and the preset cursor on the target virtual object, the user can be prompted to make a gesture corresponding to the preset cursor in the control region, so as to achieve a preset operation related thereto.

[0046] In the embodiment, the shape of the target virtual object can be a regular shape or an irregular shape, and when the user makes a related control operation on the target virtual object by a gesture, the preset bounding box superimposed on the target virtual object is always a regular rectangle or a cube, and when the target virtual object is an irregular shape, a rectangle or a cube that can wrap the target virtual object in the smallest way is used as the preset bounding box.

[0047] Optionally, please refer to Figure 4 ,Figure 4 A schematic diagram of the preset control region provided by the embodiment of the present application is shown, and the plurality of preset control regions at least include a movement control region, a scaling control region and a rotation control region. As shown in Figure 4 , Figure 4 When the target virtual object is a regular shape and the preset bounding box is a cube, the preset control region in the embodiment can include a preset control region 201 located in the area below the preset bounding box, and the preset control region 201 is a movement control region; the preset control region can also include a preset control region 202 located at each vertex of the preset bounding box, and the preset control region 202 is a scaling control region; the preset control region can also include a preset control region 203 located on the boundary of the preset bounding box parallel to the Y-axis area, and the preset control region 203 is a rotation control region. In other embodiments, the preset control region can also be set in different forms, such as being set on the boundary of the preset bounding box parallel to the Y-axis area.

[0048] Optionally, please refer to Figure 5 , Figure 5 A schematic diagram of the preset cursor provided by the embodiment of the present application is shown. As shown in Figure 5 , the movement control cursor is related to the fist-clenching gesture, and the movement control cursor is composed of the fist-clenching gesture pattern and the cross arrow above it; the scaling control cursor is related to the pinching gesture, and the scaling control cursor is composed of the pinching index finger and thumb gesture and the two arrows above and below the right side of it; the rotation control cursor is also related to the pinching gesture, and the rotation control cursor is composed of the pinching index finger and thumb gesture and the rotation arrow above it.

[0049] In the embodiment, when the user controls the first preset halo in different preset control regions in the plurality of preset control regions through the first preset gesture, the corresponding preset cursor is superimposed and displayed at the first preset halo on the target virtual object, and the gesture image contained in the displayed preset cursor is used to prompt the user to change the first preset gesture, i.e., the open gesture, to the corresponding preset gesture when the user performs the related operation, so as to control the target virtual object to realize the corresponding target control operation.

[0050] In some embodiments, after detecting that the first preset halo is in the target control region in the plurality of preset control regions, the interactive control method of the present application further includes switching the current display state of the target virtual object from the first display state to the second display state, and the appearance attribute of the target virtual object in the second display state is different from the appearance attribute of the target virtual object in the first display state.

[0051] Optionally, when the first preset halo is detected to have an overlapping area with the target virtual object and to be in the target control area of the plurality of preset control areas, it indicates that the user selects one of the plurality of preset virtual objects in the virtual scene as the target virtual object. The selected target virtual object is switched from the first display state to the second display state, indicating that the user completes the selection of the target virtual object. The target virtual object is switched in state to prompt the user to control it. The appearance attribute includes at least one of size information, color information, stroke information, and transparency information, wherein the size information is different in different display states of the virtual object; the color information is different in different display states of the virtual object; the stroke information is different in different display states of the virtual object; and the transparency information is different in different display states of the virtual object.

[0052] In some embodiments, the target virtual object switched from the first display state to the second display state can be enlarged, and the interactive area of the target virtual object is correspondingly enlarged, which is more convenient for the user to control the target virtual object by gestures. Optionally, the display state of the target virtual object can also be changed in color, or the color of the target virtual object can be changed while the target virtual object is enlarged. The display state of the target virtual object can be used to prompt the user to select the target virtual object. At the same time, the target virtual object is fixed at the current position in the virtual scene in the second display state, and no longer moves, so as to facilitate the user to control it by gestures.

[0053] Step S140: If the input target gesture is detected to be a preset gesture in a preset control mode associated with the target cursor, the target virtual object is controlled to perform a preset operation corresponding to the target gesture operation, and the target cursor is the preset cursor corresponding to any preset control area of the plurality of preset control areas.

[0054] In the embodiment, the user makes a corresponding target gesture according to the gesture image displayed by the target cursor in the control component superimposed on the target virtual object. The wearable device collects the gesture made by the user and inputs the collected gesture image into the interactive control device to take the gesture currently made by the user as the target gesture and compare it with the preset gesture in the preset control mode associated with the target cursor. The user controls the target virtual object to perform a preset operation corresponding to the target gesture operation by gesture operation.

[0055] For example, if the target cursor is a movement control cursor, the user is detected to make a gesture associated with a movement operation in the interactive area of the target virtual object. The user can control the movement control cursor superimposed on the target virtual object to control the target virtual object to move correspondingly while maintaining the gesture.

[0056] In some embodiments, after detecting that the input target gesture is a preset gesture in a preset control mode associated with the target cursor, the interactive control method can further include switching the current display state of the first preset halo from the third display state to a fourth display state, the appearance attribute of the first preset halo in the fourth display state being different from the appearance attribute of the first preset halo in the third display state.

[0057] Optionally, the first preset halo is switched from the third display state to the fourth display state, indicating that the input target gesture detected by the interactive control device matches the preset gesture in the preset control mode associated with the target cursor, and further prompting the user that the current gesture made by the user is the correct preset gesture in the preset control mode through the display state switching of the halo, and the user can control the target virtual object to perform a related control operation through the current gesture made by the user. The above-mentioned appearance attribute at least includes one of size information, color information, stroke information, and transparency information, wherein the size information is different in different display states of the halo; the color information is different in different display states of the halo; the stroke information is different in different display states of the halo; and the transparency information is different in different display states of the halo.

[0058] In the present embodiment, the display state switching effect of the first preset halo switched from the third display state to the fourth display state can include color change and size change of the first preset halo. It should be noted that the fourth display state of the first preset halo is different from the default display state of the second preset halo, for example, when the first preset halo is switched from the third display state to the fourth display state, color change and halo size reduction change occur at the same time, the maximum radius of the first preset halo in the fourth display state is greater than the maximum radius of the second preset halo in the default display state, the minimum radius of the first preset halo in the fourth display state is also greater than the minimum radius of the second preset halo in the default display state, and the color of the first preset halo in the fourth display state is different from the color of the second preset halo in the fourth display state, thereby distinguishing them.

[0059] In the present embodiment, the preset operation at least includes a moving operation, a zooming operation, and a rotating operation, the user can control the target virtual object to realize the moving operation in the moving control area, control the target virtual object to realize the zooming operation in the zooming control area, and control the target virtual object to realize the rotating operation in the rotating control area.

[0060] Optionally, a lower side region of the side of the bounding box facing the user is set as a movement control region. When the first preset halo is superimposed on the lower side region of the side of the bounding box facing the user, the bounding box superimposed on the target virtual object is switched to a movement control bounding box, that is, a control bar appears on the lower side of the side of the bounding box facing the user, and a movement control cursor is superimposed on the first preset halo to prompt the user that the target virtual object can be controlled to move through a movement operation gesture within the range.

[0061] Optionally, please refer to Figure 6 , Figure 6 A scene schematic diagram of a virtual scene of the embodiment is shown. As Figure 6 shown, when the virtual scene is a 3DOF scene, the bounding box superimposed on the target virtual object is a rectangle, and the target virtual object displays "rudder angle: specify rudder angle; rudder mode: vertex stationary; speed mode: constant speed; 2022-02-01; 16:40".

[0062] The preset control region 301 can be a lower side region of the bounding box. In the embodiment, the preset control region 301 can be a movement control region of the bounding box. When the first preset halo is superimposed on the lower side region of the bounding box, the current display state of the target virtual object is switched from the first display state to the second display state, and the control component superimposed on the target virtual object includes switching the bounding box to the movement control bounding box shown in the figure, the control bar appears on the lower side of the bounding box, and the movement control cursor is superimposed on the first preset halo.

[0063] The gesture pattern contained in the movement control cursor is a fist gesture used to prompt the target gesture of the user when performing a movement operation. When the fist gesture associated with the movement operation is detected, the movement control cursor moves in the interactive region following the movement of the user gesture. The interactive control device extracts the displacement value of the fist gesture made by the user in the X-Y axis direction, maps it on the virtual scene to control the target virtual object to realize the corresponding movement. If it is detected that the movement control cursor corresponding to the fist gesture made by the user leaves the interactive region, or it is detected that the user makes other preset gestures, the interactive control device moves the target virtual object to the position where the fist gesture is last detected in the interactive region, completing the movement operation.

[0064] Optionally, please refer to Figure 7 , Figure 7 A scene schematic diagram of another virtual scene of the embodiment is shown. As Figure 7As shown, in a 6DOF virtual scene, the bounding box superimposed on the target virtual object is a cube. The preset control area 302 can be the lower area of ​​the side of the bounding box facing the user. In this embodiment, the preset control area 302 can be the movement control area of ​​the bounding box. When the first preset halo is superimposed on the lower area of ​​the side of the bounding box facing the user, the current display state of the target virtual object switches from the first display state to the second display state. The control components superimposed on the target virtual object include switching the bounding box to the movement control bounding box shown in the figure, a control bar appearing at the lower boundary of the bounding box, and a movement control cursor superimposed on the first preset halo.

[0065] When a user makes a fist gesture associated with a movement operation, the movement control cursor follows the user's gesture and moves within the interactive area. The interactive control device extracts the displacement value of the user's fist gesture in the XYZ axis directions, maps it onto the virtual scene to control the target virtual object to achieve the corresponding movement, thereby realizing the movement operation of the target virtual object, which will not be elaborated further here.

[0066] Optionally, each vertex region of the bounding box is set as a scaling control region, and only the user-facing scaling control region on the bounding box is available for interaction. When the first preset halo is overlaid on the user-facing scaling control region within the bounding box, the bounding box overlaid on the target virtual object switches to a scaling control bounding box, that is, the lower boundary of the user-facing side of the bounding box displays a scaling style, and the scaling control cursor is overlaid on the first preset halo.

[0067] Please refer to Figure 8 , Figure 8 A schematic diagram of another virtual scene in this embodiment is shown. For example... Figure 8 As shown, when the virtual scene is a 3DOF scene, the bounding box superimposed on the target virtual object is a rectangle, and the first preset halo is superimposed on the preset control area 303. In this embodiment, the preset control area 303 can be the scaling control area of ​​the bounding box. When the first preset halo is superimposed on the scaling control area of ​​the bounding box, the bounding box switches to the scaling control bounding box shown in the figure, and a scaling style appears. The scaling control cursor is superimposed on the first preset halo.

[0068] The gesture pattern contained in the zoom control cursor is used to prompt the user to make a pinch gesture as the target gesture when performing a zoom operation. When a pinch gesture associated with the zoom operation is detected, the zoom control cursor moves within the interactive region following the movement of the user gesture. The interactive control device extracts the displacement value of the pinch gesture made by the user in the X-Y axis direction, maps it onto the virtual scene to control the target virtual object to realize corresponding zooming in or out. It should be noted that in the above operation, the vertices of the boundary box selected as a rectangle are used as the origin, and the vertices opposite the vertices triggering the zoom operation are used as the origin to perform isometric zooming on the target virtual object.

[0069] If it is detected that the zoom control cursor corresponding to the pinch gesture made by the user exits the interactive region, or if it is detected that the user makes another preset gesture, the interactive control device isometrically zooms the target virtual object to the position where the pinch gesture was last detected within the interactive region, completing the zoom operation.

[0070] Please refer to Figure 9 , Figure 9 A scene diagram of another virtual scene of the embodiment is shown. As Figure 9 shown, when the virtual scene is a 6DOF scene, the boundary box superimposed on the target virtual object is a cube, and the first preset halo is superimposed on the preset control region 304. In the embodiment, the preset control region 304 can be a zoom control region of the boundary box facing the user. When the first preset halo is superimposed on the zoom control region of the boundary box facing the user, the boundary box switches to the zoom control boundary box shown in the diagram, a zoom style appears, and a zoom control cursor is superimposed on the first preset halo.

[0071] When a pinch gesture associated with the zoom operation is detected, the zoom control cursor moves within the interactive region following the movement of the user gesture. The interactive control device extracts the displacement value of the pinch gesture made by the user in the X-Y-Z axis direction, maps it onto the virtual scene to control the target virtual object to realize corresponding zooming in or out, thereby realizing the zoom operation of the target virtual object, which will not be described again. It should be noted that in the above operation, if the vertex triggering the zoom operation is on the side of the boundary box selected as a cube facing the user, the opposite vertex of the side facing the user is selected as the origin; if the vertex triggering the zoom operation is on the side of the boundary box selected as a cube not directly facing the user, the opposite vertex of the side on the cube is selected as the origin to perform isometric zooming on the target virtual object.

[0072] Optionally, the area on the bounding box parallel to the Y-axis direction is set as a rotation control area, and only the rotation control area facing the user on the bounding box is available for interaction. When the first preset halo is superimposed on the rotation control area of the bounding box facing the user, the bounding box superimposed on the target virtual object is switched to a rotation control bounding box, that is, the lower boundary of the side of the bounding box facing the user appears a control bar, and a rotation control cursor is superimposed on the first preset halo.

[0073] Please refer to Figure 10 , Figure 10 A scene diagram of another virtual scene of the embodiment is shown. As shown in Figure 10 , the virtual scene is a 6DOF scene, and the rotation operation can be implemented. The bounding box superimposed on the target virtual object is a cube, and the first preset halo is superimposed on the preset control area 305. In the embodiment, the preset control area 305 can be the boundary line of the bounding box facing the user and parallel to the Y-axis direction, which is the rotation control area of the bounding box. When the first preset halo is superimposed on the rotation control area of the bounding box facing the user, the bounding box is switched to the rotation control bounding box shown in the figure, the rotation style appears on the rotation control area facing the user, and the rotation control cursor is superimposed on the first preset halo.

[0074] When it is detected that the user makes a pinch gesture associated with the rotation operation, the rotation control cursor moves in the interactive area following the movement of the user gesture. The interaction control device extracts the displacement value of the pinch gesture made by the user in the X-Y-Z axis direction, and maps it to the virtual scene to control the target virtual object to implement a corresponding rotation angle. It should be noted that in the above operation, the vertical center Y-axis of the bounding box selected as the cube is taken as the rotation center, and the target virtual object is rotated.

[0075] If it is detected that the rotation control cursor corresponding to the pinch gesture made by the user leaves the interactive area, or it is detected that the user makes other preset gestures, the interaction control device rotates the target virtual object to the position where the pinch gesture is last detected in the interactive area, and completes the rotation operation.

[0076] Optionally, if the first preset halo is superimposed on other areas of the interactive area of the target virtual object except the above-mentioned movement control area, zoom control area and rotation control area, no corresponding control component is superimposed on the target virtual object. The user controls the first preset halo superimposed on the target virtual object through gestures to implement default control operations. The above-mentioned default control operations at least include click operation, selection operation, activation operation and confirmation operation.

[0077] When the user makes a pinch gesture related to the default control operation is detected, the first preset halo moves within the interactive region following the movement of the user gesture. For example, when the default control operation is a drag control operation, such as volume size control, the interactive control device extracts the displacement value of the pinch gesture made by the user in the X-axis direction, maps it to the virtual scene to control the drag button in the target virtual object, and completes the drag operation based on the position where the pinch gesture is last detected within the interactive region of the target virtual object, thereby achieving corresponding volume adjustment.

[0078] In this embodiment, the method of the present application detects the relative position of the first preset gesture and the target virtual object in response to the first preset gesture, displays a first preset halo in the overlapping area if an overlapping area is detected between the first preset gesture and the target virtual object, displays a control component on the target virtual object if the first preset halo is detected in the target control area of the plurality of preset control areas, the control component includes a preset cursor and a preset boundary box corresponding to each preset control area, and each preset cursor corresponding to each preset control area is associated with a preset control mode. If the input target gesture is a preset gesture in the target cursor associated preset control mode, the target virtual object is controlled to perform a preset operation corresponding to the target gesture operation, and the target cursor is a preset cursor corresponding to any preset control area in the plurality of preset control areas. In this way, by coordinating the user gesture and the halo displayed on the virtual object, the user performs different control operations in different preset control areas through corresponding gestures. Then, the user can perform the corresponding preset operation when the hand gesture is made according to the prompt of the control component displayed on the virtual object, so that the user can achieve more control functions through gesture operation, improve the user experience, and at the same time, the limitation of the preset control area improves the accuracy and accuracy of the interactive control, reduces the occurrence of false control and other problems. And in the process of user gesture interacting with the control halo and the virtual object, the halo and the virtual object are switched to different display states when the interaction is confirmed, which provides feedback to the user, further improving the control accuracy in the interaction process.

[0079] Please refer to Figure 11 , Figure 11 The flowchart of the interactive control method provided by another embodiment of the present application is shown, which is applied to a wearable device. The interactive control method provided by the embodiment of the present application will be described in detail below. Figure 11 The interactive control method provided by the embodiment of the present application can include the following steps:

[0080] Step S410: In response to a first preset gesture, detecting the relative position of the first preset gesture and a target virtual object.

[0081] Step S420: If it is detected that the first preset gesture does not overlap with the target virtual object, a second preset halo is superimposed and displayed at the target hand position in the hand corresponding to the first preset gesture, the second preset halo having different appearance attributes from the first preset halo.

[0082] In the embodiment, if the interaction control device detects that the first preset gesture currently made by the user does not enter the interactive area range of the target virtual object, it is determined that the first preset gesture made by the user does not overlap with the target virtual object. At this time, the interaction control device superimposes and displays a second preset halo at the target hand position in the hand corresponding to the overlap area between the first preset gesture made by the user and the virtual scene.

[0083] Optionally, the second preset halo has different appearance attributes from the first preset halo, and the appearance attributes at least include one of size information, color information, stroke information, and transparency information. The size information is that the sizes of different preset halos are different; the color information is that the colors of different preset halos are different; the stroke information is that the solid and void of the boundaries of different preset halos are different; and the transparency information is that the brightness and rendering effects of different preset halos are different. Figure 3 As shown in the above embodiment, the maximum radius and the minimum radius of the second preset halo under the breathing effect are different from those of the first preset halo, and the color of the second preset halo is different from that of the first preset halo, so as to distinguish them.

[0084] In some embodiments, if it is detected that the first preset gesture currently made by the user is superimposed and displayed at the target hand position in the hand corresponding to the virtual scene, it indicates that the user does not select any preset virtual object in the plurality of preset virtual objects in the virtual scene as the target virtual object. At this time, the second preset halo is controlled by the gesture of the user. When the user controls the second preset halo to enter the interactive area of any preset virtual object in the plurality of preset virtual objects in the virtual scene through the gesture operation, the user selects the preset virtual object as the target virtual object, and the second preset halo is switched to display the first preset halo.

[0085] In some embodiments, when the interaction control device is activated but the user does not enter the virtual scene using the wearable device, a second preset halo can be displayed in a middle position of the virtual scene until the interaction control device detects that the user enters the virtual scene using the wearable device, the second preset halo is associated with the wearable device, so that the position of the second preset halo in the virtual scene is controlled by the gesture detected by the wearable device, and the second preset halo of the wearable device is displayed at a position 3 meters away from the wearable device. Alternatively, the initial display position of the second preset halo can also be set at the lower edge, upper edge, left edge or right edge of the virtual scene. The embodiments of the present application do not limit the initial display position of the second preset halo in the virtual scene, and the fixed display distance of the second preset halo relative to the wearable device can be adjusted according to the user's usage habits.

[0086] Step S430: If it is detected that the first preset gesture and the target virtual object have an overlapping area, display a first preset halo in the overlapping area.

[0087] Step S440: If it is detected that the first preset halo is in a target control area in the plurality of preset control areas, superimpose a control component on the target virtual object, the control component including a preset cursor and a preset boundary box corresponding to each of the preset control areas, and each of the preset cursors corresponding to the preset control areas is associated with a preset control mode.

[0088] Step S450: If it is detected that the input target gesture is a preset gesture in the preset control mode associated with the target cursor, control the target virtual object to perform a preset operation corresponding to the target gesture operation, and the target cursor is the preset cursor corresponding to any of the preset control areas in the plurality of preset control areas.

[0089] In the embodiments, the specific implementation of steps S410 and S430-S450 can refer to the content in the foregoing embodiments, which will not be described here.

[0090] In the embodiments, the method of the present application can be based on the detection that the first preset gesture and the target virtual object do not have an overlapping area, i.e., the user makes the first preset gesture but does not select the target virtual object, so that the halo superimposed at the target hand position is the second preset halo, which can prompt the user that the target virtual object cannot be controlled. In this way, when the relative positions of the gesture made by the user and the target virtual object are different, the second preset halo and the first preset halo superimposed in the virtual scene have different appearance properties, which can provide different feedback prompts to the user, and the user can control the target virtual object through the gesture operation only when the second preset halo is switched to the first preset halo, thereby reducing the miscontrol in the control process and improving the control accuracy in the interaction process.

[0091] Please refer to Figure 12 , Figure 12 The flowchart of the interaction control method provided by the embodiments of the present application is shown, which is applied to a wearable device. The interaction control method provided by the embodiments of the present application will be described in detail below. Figure 12 The interaction control method provided by the embodiments of the present application can include the following contents:

[0092] In the embodiment, when the user starts the interaction control device and enters the virtual scene through the head-mounted device, the interaction control device detects whether the opening gesture made by the user is superimposed on any virtual object in the virtual scene.

[0093] If yes, the interaction control device controls the first halo to appear at the superimposed position of the gesture made by the user and the virtual object. The first halo is the first preset halo described above, and the position of the first halo in the virtual scene follows the position of the user's hand. The user controls the movement of the first halo through the gesture. If no, the interaction control device controls the second halo to be displayed in the virtual scene. The second halo is the second preset halo described above. The position of the second halo in the virtual scene is fixed relative to the distance of the wearable device, and the position of the second halo in the virtual scene also follows the position of the user's hand.

[0094] The interaction control device detects whether the opening gesture made by the user is superimposed on the interactive range of the virtual object. If yes, the virtual object is switched to the hovering state when the gesture is superimposed on the interactive range of the virtual object, i.e., the object changes in size, color, etc., to prompt the user that the selection of the virtual object is completed.

[0095] Further, the interaction control device judges whether other control cursors appear on the interactive object. If yes, the control operation corresponding to the other control cursors can be performed. It can be understood that the gesture and the interactive range of the virtual object are any of the moving control region, the zooming control region and the rotating control region. Then, the other control cursors are superimposed and displayed on the virtual object, prompting the user to make the defining gesture associated with the cursor. The defining gesture includes the fist gesture associated with the moving operation and the pinching gesture associated with the rotating operation and the zooming operation. The gesture made by the user is judged. If the corresponding defining gesture is detected, it is taken as a confirmation gesture. The display state of the first halo changes. When the user controls the virtual object through the gesture operation, the virtual object presents the corresponding feedback and changes. Until other gestures except the confirmation gesture are detected, the process ends.

[0096] If it is detected that the interaction object does not appear on other control cursors, i.e. the gesture and the interaction region of the virtual object are not in any of the above-mentioned movement control region, zoom control region and rotation control region, the first halo does not change, the gesture made by the user is judged, if a pinch gesture is detected, it is regarded as a confirmation gesture, the first halo display state changes, when the user controls the virtual object through the gesture operation, the virtual object presents corresponding feedback and changes, until other gestures except the confirmation gesture are detected, the process ends.

[0097] Please refer to Figure 13 , Figure 13 The structure block diagram of the interaction control device 500 provided by an embodiment of the present application is shown, which is applied to a wearable device. The interaction control device 500 can include a position detection module 510, a halo display module 520, a control component display module 530 and a control module 540.

[0098] The position detection module 510 is configured to detect the relative position of the first preset gesture and a target virtual object in response to the first preset gesture.

[0099] The halo display module 520 is configured to display a first preset halo in an overlapping region of the first preset gesture and the target virtual object if it is detected that the first preset gesture and the target virtual object have the overlapping region.

[0100] The control component display module 530 is configured to display a control component on the target virtual object if it is detected that the first preset halo is in a target control region of a plurality of preset control regions, the control component including a preset cursor and a preset boundary box corresponding to each of the preset control regions, and each of the preset cursors corresponding to the preset control regions being associated with a preset control mode.

[0101] The control module 540 is configured to control the target virtual object to perform a preset operation corresponding to a target gesture operation if it is detected that the input target gesture is a preset gesture in a preset control mode associated with a target cursor, the target cursor being the preset cursor corresponding to any of the plurality of preset control regions.

[0102] In some other embodiments, the halo display module 520 is further configured to display a second preset halo on a target hand position in a hand corresponding to the first preset gesture if it is detected that the first preset gesture and the target virtual object do not have an overlapping region, the second preset halo having different appearance attributes from the first preset halo.

[0103] In some embodiments, the interaction control apparatus 500 further comprises a state switching module configured to switch a current display state of the target virtual object from a first display state to a second display state, wherein an appearance attribute of the target virtual object in the second display state is different from an appearance attribute of the target virtual object in the first display state.

[0104] In some embodiments, the state switching module is further configured to switch a current display state of the first preset halo from a third display state to a fourth display state, wherein an appearance attribute of the first preset halo in the fourth display state is different from an appearance attribute of the first preset halo in the third display state.

[0105] In some embodiments, the state switching module is further configured to switch a current display state of the first preset halo from a third display state to a fourth display state, wherein an appearance attribute of the first preset halo in the fourth display state is different from an appearance attribute of the first preset halo in the third display state.

[0106] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described apparatuses and modules can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0107] In several embodiments provided in the present application, the coupling between the modules can be electrical, mechanical or other forms of coupling.

[0108] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can exist physically independently, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0109] In summary, the interaction control apparatus comprises a position detection module configured to, in response to a first preset gesture, detect a relative position between the first preset gesture and a target virtual object; a halo display module configured to, if an overlapping area between the first preset gesture and the target virtual object is detected, display a first preset halo in the overlapping area; a control component display module configured to, if the first preset halo is detected to be in a target control area among a plurality of preset control areas, display a control component on the target virtual object, the control component comprising a preset cursor and a preset bounding box corresponding to each preset control area, and each preset cursor corresponding to a preset control area being associated with a preset control mode; and a control module configured to, if an input target gesture is detected to be a preset gesture in the preset control mode associated with the target cursor, control the target virtual object to perform a preset operation corresponding to the target gesture operation, the target cursor being a preset cursor corresponding to any preset control area among the plurality of preset control areas. In this way, a user selects a virtual object in a virtual scene through a gesture operation, and through cooperation between a user gesture and a halo displayed on the virtual object, different preset control areas are defined to perform different control operations, the user can perform a corresponding preset operation only when a preset gesture is made by hand cooperation according to a prompt of a control component displayed on the virtual object, so that the control function that can be achieved by the user through the gesture operation is more abundant, and the user experience is improved. Meanwhile, the definition of the preset control area improves the precision and accuracy of the interaction control, and reduces the occurrence of problems such as miscontrol.

[0110] The following will be described in combination with Figure 14 A wearable device 600 provided by the present application is described.

[0111] Referring to Figure 14 , Figure 14 A structural block diagram of a wearable device 600 provided by an embodiment of the present application is shown, and the above method provided by the embodiment of the present application can be executed by the wearable device 600.

[0112] The wearable device 600 in the embodiment of the present application can include one or more of the following components: a processor 601, a memory 602, and one or more application programs, wherein the one or more application programs can be stored in the memory 602 and configured to be executed by the one or more processors 601, and the one or more programs are configured to execute the method as described in the foregoing method embodiment.

[0113] The processor 601 can include one or more processing cores. The processor 601 connects various parts within the wearable device 600 by running or executing instructions, programs, code sets or instruction sets stored in the memory 602, and calling data stored in the memory 602, to perform various functions and process data of the wearable device 600. Optionally, the processor 601 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA). The processor 601 can integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used for processing wireless communication. It can be understood that the above-mentioned modem can also be integrated into the processor 601, and be realized separately by a communication chip.

[0114] The memory 602 can include a random access memory (RAM) and can also include a read-only memory (ROM). The memory 602 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 602 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing various method embodiments described below, etc. The data storage area can also store data created by the wearable device 600 in use (such as various corresponding relationships described above) and the like.

[0115] It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working process of the above-described devices and modules can refer to the corresponding process in the foregoing method embodiments, which will not be described herein.

[0116] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules displayed or discussed can be indirect coupling or communication connection between some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0117] In addition, each functional module in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software functional module.

[0118] Reference is made to Figure 15 , Figure 15 A structural block diagram of a computer readable storage medium provided by an embodiment of the present application is shown. The computer readable storage medium 700 stores program codes, which can be invoked by a processor to execute the methods described in the above method embodiments.

[0119] The computer readable storage medium 700 can be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer readable storage medium 700 includes a non-transitory computer readable medium. The computer readable storage medium 700 has a storage space for program codes 710 for executing any of the above methods. These program codes can be read from or written to one or more computer program products. The program codes 710 can be compressed in an appropriate form, for example.

[0120] In some embodiments, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of an electronic device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the steps in the above method embodiments.

[0121] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not drive the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An interactive control method, characterized in that, Applied to wearable devices, the method includes: In response to a first preset gesture, the relative position of the first preset gesture and the target virtual object is detected; If an overlapping area is detected between the first preset gesture and the target virtual object, a first preset halo is displayed in the overlapping area; If the first preset halo is detected to be in the target control area of ​​multiple preset control areas, a control component is superimposed and displayed on the target virtual object. The control component includes a preset cursor and a preset bounding box corresponding to each preset control area. The preset cursor corresponding to each preset control area is associated with a preset control mode. If the input target gesture is detected to be a preset gesture under a preset control mode associated with the target cursor, the target virtual object is controlled to perform a preset operation corresponding to the target gesture operation, and the target cursor is the preset cursor corresponding to any preset control area among the multiple preset control areas.

2. The method according to claim 1, characterized in that, After detecting the relative position of the first preset gesture and the target virtual object in response to the first preset gesture, the method further includes: If it is detected that the first preset gesture and the target virtual object do not overlap, a second preset halo is superimposed on the target hand position in the hand corresponding to the first preset gesture. The appearance attributes of the second preset halo are different from those of the first preset halo.

3. The method according to claim 1, characterized in that, After detecting that the first preset halo is located in a target control area within a plurality of preset control areas, the method further includes: The current display state of the target virtual object is switched from the first display state to the second display state, and the appearance attributes of the target virtual object in the second display state are different from those of the target virtual object in the first display state.

4. The method according to claim 1, characterized in that, After detecting that the input target gesture is a preset gesture under a preset control mode associated with the target cursor, the method further includes: The current display state of the first preset halo is switched from the third display state to the fourth display state. The appearance attributes of the first preset halo in the fourth display state are different from those of the first preset halo in the third display state.

5. The method according to any one of claims 2-4, characterized in that, The appearance attributes include at least one of the following: size information, color information, outline information, and transparency information.

6. The method according to any one of claims 1-4, characterized in that, The multiple preset control areas correspond one-to-one with the multiple preset control methods, and the multiple preset control methods correspond one-to-one with the multiple preset gestures, which include at least an open gesture, a clenched fist gesture, and a pinched gesture.

7. The method according to any one of claims 1-4, characterized in that, The plurality of preset control areas include at least a movement control area, a zoom control area, and a rotation control area.

8. An interactive control device, characterized in that, For use in wearable devices, the device includes: The position detection module is used to detect the relative position of the first preset gesture and the target virtual object in response to the first preset gesture; A halo display module is used to display a first preset halo in the overlapping area if an overlapping area is detected between the first preset gesture and the target virtual object. The control component display module is used to overlay and display a control component on the target virtual object if the first preset halo is detected to be in the target control area of ​​multiple preset control areas. The control component includes a preset cursor and a preset bounding box corresponding to each preset control area. The preset cursor corresponding to each preset control area is associated with a preset control mode. The control module is used to control the target virtual object to perform a preset operation corresponding to the target gesture operation if the input target gesture is detected to be a preset gesture under a preset control mode associated with the target cursor, wherein the target cursor is the preset cursor corresponding to any preset control area among the plurality of preset control areas.

9. A wearable device, characterized in that, The device includes: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to perform the method as described in any one of claims 1 to 7.

Images