Interaction method, system and device, electronic equipment and computer medium

Abstract

The invention discloses an interaction method, system and device, electronic equipment and a computer medium. The method comprises the following steps: displaying a scene picture; when the current working mode is the first working mode, if a first preset mode switching instruction is detected, the first working mode is adjusted to be a second working mode, and in the second working mode, a user interacts with the scene picture through a gesture of one hand, so that the scene picture can be switched. Interacting with the scene picture through a handheld device held by the other hand; wherein the first working mode is a gesture working mode or a handheld device working mode. According to the technical scheme provided by the embodiment of the invention, the first working mode is adjusted to the second working mode through the first preset mode switching instruction, and the power consumption of the handheld device is reduced in the switching process of gesture control and handheld device control.

Description

Technical Field

[0001] This disclosure belongs to the technical field of head-mounted display devices, and particularly relates to an interaction method, system, device, electronic device, and computer medium. Background Art

[0002] With the rapid development of VR technology, VR systems have gradually been applied in various fields such as games, videos, training, and education. VR systems currently support the positioning function of handheld devices. In existing practical applications, the input design of VR systems generally uses 3DoF and 6DoF handheld devices. In a game scenario, the handheld device can complete rapid interaction of users and obtain a relatively good gaming experience. However, in applications such as videos and education, using a handheld device for selection is relatively troublesome. For example, using a handheld device to control functions such as picture saving, picture zooming, and picture sharing is very unnatural and the operation is complex, while using gesture control to naturally operate for functions such as image saving and sharing during shooting is a more friendly and natural operation.

[0003] Currently, related products can only achieve either gesture control or handheld device control, resulting in extremely poor experience. Moreover, during the switching process between gesture control and handheld device control, the power consumption of the handheld device is too large, wasting power resources. Summary of the Invention

[0004] The disclosed embodiments provide an implementation solution different from the related art to solve the technical problems in the related art that related products can only achieve either gesture control or handheld device control, resulting in extremely poor experience, and during the switching process between gesture control and handheld device control, the power consumption of the handheld device is too large, wasting power resources.

[0005] In a first aspect, this disclosure provides an interaction method, and the method includes:

[0006] Display a scene picture;

[0007] When the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand;

[0008] wherein the first working mode is a gesture working mode or a handheld device working mode.

[0009] In a second aspect, this disclosure provides an interaction device, and the device includes:

[0010] A display module for displaying a scene picture;

[0011] An adjustment module, configured to, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand;

[0012] Wherein, the first working mode is a gesture working mode or a handheld device working mode.

[0013] In a third aspect, the present disclosure provides an electronic device, including:

[0014] A processor; and

[0015] A memory for storing executable instructions of the processor;

[0016] Wherein, the processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any method in the first aspect or any possible implementation manner of the first aspect.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements any method in the first aspect or any possible implementation manner of the first aspect.

[0019] The solution provided by the present disclosure, through the interaction method of the present disclosure, during the process of displaying a scene picture, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, the first working mode is adjusted to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand; wherein, the first working mode is a gesture working mode or a handheld device working mode. The solution of the present application reduces the power consumption of the handheld device during the switching process between gesture control and handheld device control. And in the second working mode, one hand of the user is used to interact with the scene picture through gestures, and the other hand interacts with the scene picture through a handheld device, adjusting the original gesture working mode with only gesture operations or the handheld device working mode with only handheld device operations to the user interacting with the scene picture through gestures with one hand and interacting with the scene picture through a handheld device with the other hand, thereby realizing the simultaneous progress of gesture work and handheld device work, and achieving the technical effect of making the interaction more natural and smooth. Description of the Drawings

[0020] To more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are some embodiments of the present disclosure. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings. In the drawings:

[0021] Figure 1 Schematic structural diagram of a system provided by an embodiment of the present disclosure;

[0022] Figure 2 Schematic flowchart of an interaction method provided by an embodiment of the present disclosure;

[0023] Figure 3 Schematic diagram of a scenario where a head-mounted display device interacts with a handheld device in the handheld device working mode provided by an embodiment of the present disclosure;

[0024] Figure 4 Schematic diagram of a scenario where a head-mounted display device interacts with a gesture in the gesture working mode provided by an embodiment of the present disclosure;

[0025] Figure 5 Schematic diagram of a scenario where a head-mounted display device interacts with a handheld device and a gesture in the second working mode provided by an embodiment of the present disclosure;

[0026] Figure 6 Schematic diagram of the exposure time of a shooting device of a head-mounted display device under different conditions provided by an embodiment of the present disclosure;

[0027] Figure 7 Schematic diagram of an information interaction method between a head-mounted display device and a handheld device provided by an embodiment of the present disclosure;

[0028] Figure 8 Schematic diagram of the relationship between the supplementary light range of an infrared lamp and the shooting range of a shooting device on a head-mounted display device provided by an embodiment of the present disclosure;

[0029] Figure 9 Schematic diagram of data transmission of a head-mounted display device in the handle working mode provided by an embodiment of the present disclosure;

[0030] Figure 10 Schematic structural diagram of an interaction device provided by an embodiment of the present disclosure;

[0031] Figure 11 Schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0032] Embodiments of the present disclosure are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, but should not be understood as limiting the present disclosure.

[0033] The terms "first" and "second" and the like in the specification, claims, and drawings of the disclosed embodiments are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the disclosed embodiments described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0034] With the rapid development of VR technology, VR systems are gradually being applied in various fields such as games, videos, training, and education. VR systems currently support the positioning function of handheld devices. In existing practical applications, the input of VR systems generally relies on 3DoF and 6DoF handheld devices. In game scenarios, users can interact with VR scene images through handheld devices to obtain a better gaming experience; however, in similar applications such as video and education, it is more troublesome to use handheld devices to interact with scene images. For example, it is very unnatural and complicated to use handheld devices to control image saving, image enlargement, and image sharing, while gestures can naturally operate images, take images, save images, and share images.

[0035] At present, related products can only realize gesture control or handheld device control, but in some special scenarios, handheld device control and gesture control need to be operated together. In the related technologies, only gesture control or only handheld device control cannot meet the needs of users, and the user experience is poor. In the process of switching between gesture control and handheld device control, the handheld device needs to be repeatedly turned off and on, and the handheld device consumes too much power, which wastes power resources.

[0036] The following specific embodiments are used to describe in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0037] Figure 1A schematic structural diagram of a system provided by an exemplary embodiment of the present disclosure. The structure includes: a head-mounted display device 11 and a control device 12.

[0038] Among them, the head-mounted display device 11 can be used to execute an interaction method. Specifically, it can be used to execute the following: display a scene picture; when the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gesture of one hand and interacts with the scene picture through a handheld device held by the other hand; wherein, the first working mode is a gesture working mode or a handheld device working mode.

[0039] In some embodiments, the aforementioned control device 12 can provide the content of the aforementioned scene picture for the aforementioned head-mounted display device 11 to display.

[0040] In some embodiments, the system may also only include the head-mounted display device 11, and this application does not make any limitations in this regard.

[0041] In some embodiments, the head-mounted display device 11 may include a display screen, a processor, and a housing. Among them, the display screen is used to display a scene picture; the processor is used to, when the current working mode of the head-mounted display device 11 is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gesture of one hand and interacts with the scene picture through a handheld device held by the other hand; wherein, the first working mode is a gesture working mode or a handheld device working mode.

[0042] In some embodiments, a plurality of shooting devices are provided on the housing, and the plurality of shooting devices can be a plurality of fisheye cameras.

[0043] In some embodiments, an infrared lamp is also provided on the housing for filling light when the aforementioned plurality of shooting devices are shooting images.

[0044] In some embodiments, an infrared lamp is also provided on the aforementioned handheld device, and the infrared lamp can facilitate the processor to determine the position of the handheld device according to the shooting content of the shooting device.

[0045] Among them, the aforementioned scene picture is determined based on the pose of the head-mounted display device, and the pose of the head-mounted display device is determined according to the environmental images around the head-mounted display device captured by the aforementioned plurality of shooting devices.

[0046] For the execution principles or interaction processes of the respective component units in the embodiments of this system, such as the head-mounted display device 11 and the control device 12, reference can be made to the descriptions of the following respective method embodiments.

[0047] Figure 2 A flowchart of an interaction method provided for an exemplary embodiment of the present disclosure. This method can be applied to a head-mounted display device, etc. The method includes S201 - S202:

[0048] S201, display a scene image.

[0049] In some embodiments, the scene image can be displayed through a display screen in the head-mounted display device.

[0050] Specifically, a plurality of shooting devices can be arranged on the head-mounted display device. The plurality of shooting devices can shoot environmental images around the head-mounted display device. The head-mounted display device can determine the pose information of the head-mounted display device according to the captured environmental images, and then determine the scene image to be displayed on the display screen according to the pose information of the head-mounted display device.

[0051] In some embodiments, the scene image can specifically be a 3D VR (Virtual Reality) image, an MR (Mixed Reality) image, or an AR (Augmented Reality) image. The type of the image can be a game image, an education and training image, etc.

[0052] S202, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene image through the gesture of one hand and interacts with the scene image through a handheld device held by the other hand;

[0053] In some embodiments, the first working mode is a gesture working mode or a handheld device working mode.

[0054] Among them, when the first working mode is the gesture working mode, the user interacts with the scene image displayed on the head-mounted display device through gestures. When the first working mode is the handheld device working mode, the user interacts with the scene image displayed on the head-mounted display device through the handheld device.

[0055] In some embodiments, in the second working mode, when the user interacts with the scene image through the gesture of one hand, the hand image of the virtual hand of this hand of the user is displayed in the scene image.

[0056] In some embodiments, when the head-mounted display device is under combined control of the handle and gestures, that is, when the head-mounted display device is in the second working mode, the user can control the image capture by operating the buttons of the handle, and can directly drag, save, delete, and share pictures through gestures. The combination of the two can enrich the interaction between the user and the head-mounted display device.

[0057] Figure 3 FIG. is a schematic diagram of a scenario where a head-mounted display device provided by an exemplary embodiment of the present disclosure interacts with a handheld device in the handheld device working mode. Figure 3 The handle 114 in FIG. is a handheld device. The user can operate the two handles 114 to interact with the scene displayed on the display screen of the head-mounted display device.

[0058] In other embodiments, the handheld device can also be gloves, etc., which can be specifically set according to actual situations.

[0059] Figure 4 FIG. is a schematic diagram of a scenario where a head-mounted display device provided by an exemplary embodiment of the present disclosure interacts with gestures in the gesture working mode. Figure 4 The 115 in FIG. is the user's hand. The user can interact with the scene displayed on the display screen of the head-mounted display device by making different gestures.

[0060] In some embodiments, the first preset mode switching instruction is an instruction triggered by at least one of the following information: first preset gesture information, first preset button information, first preset voice information, first preset handheld device information, where the first preset handheld device information is information sent by the handheld device for indicating mode switching. Specifically, the first preset handheld device information is an indication information sent by the handheld device for indicating that the head-mounted display device switches from the first working mode to the second working mode.

[0061] In some embodiments, the first preset gesture information mainly refers to the gesture information of gestures set by relevant personnel.

[0062] In some embodiments, the first preset button information mainly refers to the button information of buttons on the handheld device or the head-mounted display device set by relevant personnel.

[0063] In some embodiments, the first preset handheld device information can also be preset trajectory information of the handheld device set by relevant personnel. For example, the preset trajectory information can be trajectory information formed when the handheld device performs a circular motion.

[0064] In some other embodiments, when the first working mode is the handheld device working mode, the first preset handheld device information may also be information for indicating that one of the two handheld devices leaves a preset area or disconnects.

[0065] Optionally, the preset area described in this application may be the overlapping area of the coverage ranges of two cameras under the head-mounted display device.

[0066] Figure 5 FIG. is a schematic diagram of a scenario where a user interacts with a head-mounted display device through a handheld device and gestures in a second working mode provided by an exemplary embodiment of the present disclosure. Figure 5 In [the figure], the handle 114 is the handheld device, and 115 is the hand. The user can interact with the head-mounted display device through the handheld device and gestures at the same time, specifically referring to that the user can interact with the scene picture displayed on the head-mounted display device through the handheld device and gestures at the same time.

[0067] In some embodiments, in the second working mode, the method further includes S2021 - S2024:

[0068] S2021. Obtain a target environment image collected by a photographing device, where the target environment image includes first hand image information of the user and image information of the handheld device;

[0069] S2022. Obtain control information sent by the handheld device;

[0070] S2023. Determine a first interaction instruction of the user for the scene picture according to the target environment image and the control information;

[0071] S2024. Respond to the first interaction instruction.

[0072] Optionally, the control information sent by the handheld device may include operation information of a button arranged on the handheld device.

[0073] Optionally, the control information sent by the handheld device may further include trajectory information of the handheld device itself.

[0074] In some embodiments, in the foregoing 2023, determining the first interaction instruction of the user for the scene picture according to the target environment image and the control information may include the following S231 - S233:

[0075] S231. Determine gesture information of the user according to the first hand image information of the user included in the target environment image;

[0076] S232. Determine position information and / or attitude information of the handheld device according to the image information of the handheld device included in the target environment image;

[0077] S233. Determine a first interaction instruction of the user for the scene picture based on the gesture information, the position information, and / or the pose information, and the control information.

[0078] In some embodiments, determining a first interaction instruction of the user for the scene picture based on the gesture information, the position information, and / or the pose information, and the control information may include:

[0079] Determine a first instruction of the user for the scene picture based on the gesture information;

[0080] Determine a second instruction of the user for the scene picture based on the position information and / or the pose information, and the control information;

[0081] Wherein, the first instruction and the second instruction are included in the first interaction instruction.

[0082] Correspondingly, responding to the first interaction instruction includes: responding to the first instruction and the second instruction.

[0083] In some embodiments, the shooting device in the foregoing S2021 may be a plurality of shooting devices provided on the housing of the head-mounted display device.

[0084] Specifically, refer to Figures 3 - 5 As shown, a plurality of shooting devices 111 may be provided on the housing of the head-mounted display device. For example, the plurality of shooting devices 111 may be distributed on the edge of the housing for shooting environmental images. Among them, the number of the shooting devices 111 may be 4, and the specific installation positions of the shooting devices 111 may be referred to Figures 3 - 5 As shown, the shooting device 111 may specifically be a fisheye camera.

[0085] In some other embodiments, the shooting device may also be independently provided outside the head-mounted display device.

[0086] The interaction between the user and the head-mounted display device through gestures is specifically realized by tracking the user's gestures. The tracking of the user's gestures depends on the image containing the hand image information of the user captured by the shooting device 111 on the head-mounted display device.

[0087] In some embodiments, the exposure times of different shooting devices 111 may be the same or different.

[0088] Further, in some alternative embodiments, the method further includes the following S01 - S05:

[0089] S01. Control the shooting device to capture first image information based on the ambient light brightness, and the first image information includes the environmental information of the head-mounted display device;

[0090] Among them, the aforementioned S01 can be executed between the aforementioned S2021. The first image information in S01 is the image information used to determine the scene picture to be displayed on the display screen of the head-mounted display device.

[0091] Specifically, reference can be made to Figure 6 the situation shown in Case 1. Case 1 is the exposure time situation of each photographing device when the photographing device photographs the first image information based on the ambient light brightness. Among them, the photographing device in S01 refers to multiple photographing devices: Photographing Device 1, Photographing Device 2, Photographing Device 3, and Photographing Device 4. Among them, Photographing Device 1, Photographing Device 2, Photographing Device 3, and Photographing Device 4 can be Figure 3 the 4 photographing devices 111 in

[0092] During specific implementation, in combination with Figure 3 、 Figure 4 、 Figure 5 ,and Figure 8 as shown, at least one infrared lamp 112 is provided on the head-mounted display device 11, and the infrared lamp 112 can be provided on the outer shell of the head-mounted display device. The supplementary light range of the infrared lamp 112 can cover the overlapping area 30 of multiple photographing devices (such as the overlapping area 30 of the two photographing devices 111 below). Among them, this overlapping area 30 is the overlapping area of the horizontal and vertical FOVs of the two photographing devices 111 below the head-mounted display device. Since the user's gesture operations are basically in the front area of the head-mounted display device, the infrared lamp 112 needs to cover the overlapping area 30 of the coverage ranges of the two cameras below (such as Figure 8 the photographing device 111 in Figure 8 at a gesture operation distance (such as 1 m). Specifically, the supplementary light range of the infrared lamp 112 can be

[0093] the 40 in

[0094] In Case 1, since the orientations of the photographing devices are different and the brightness of the light near each photographing device is different, the exposure times of different photographing devices are different.

[0095] S02. Control the photographing device to photograph the second image information with a first preset exposure duration, and the second image information includes the second hand image information of the user;

[0096] In some embodiments, in the foregoing S2021, when the imaging device acquires the target environment image, it may capture the target environment image with a second preset exposure duration, where the second preset exposure duration may be less than a preset threshold, and the second preset exposure durations of the multiple imaging devices are the same.

[0097] Optionally, the second preset exposure duration may be less than 100 microseconds, for example, between 40 and 50 microseconds.

[0098] In some embodiments, the second preset exposure duration is the same as the first preset exposure duration.

[0099] In some embodiments, reference may be made to Figure 6 Case 2 shown in, where Case 2 is the exposure time (i.e., the second preset exposure duration) of each imaging device when the imaging device acquires the target environment image. Among them, the imaging device in S02 refers to multiple imaging devices: Imaging Device 1, Imaging Device 2, Imaging Device 3, and Imaging Device 4. The second preset exposure durations of the multiple imaging devices are the same.

[0100] Among them, when controlling the imaging device to acquire the target environment image, the infrared lamp on the head-mounted display device may be controlled to turn on for fill light. And the infrared lamp on the handheld device is controlled to turn on to facilitate the imaging device to capture the handheld device and determine the position of the handheld device. Correspondingly, the method further includes: controlling the infrared lamp disposed on the handheld device to light up; and controlling the imaging device to acquire the target environment image when the infrared lamp is lit.

[0101] In Case 2, the exposure times of each imaging device are the same and short, thereby reducing the influence of ambient light on the quality of the target environment image acquired by the imaging device.

[0102] Specifically, reference may be made to Figure 5 shown. In an environment with relatively dim ambient light, when multiple imaging devices 111 acquire the target environment image, the infrared lamp 112 may be controlled to turn on so that the multiple imaging devices 111 can acquire a target environment image with relatively good quality.

[0103] In some embodiments, reference may be made to Figure 6 Case 3 shown in, where Case 3 is the exposure time (i.e., the first preset exposure duration) of multiple imaging devices when the imaging device captures the second image information with the first preset exposure duration. Among them, the imaging device in S02 refers to multiple imaging devices: Imaging Device 1, Imaging Device 2, Imaging Device 3, and Imaging Device 4. The first preset exposure durations of the multiple imaging devices are the same. Among them, when controlling the imaging device to capture the second image information with the first preset exposure duration, the infrared lamp on the head-mounted display device may be controlled to turn on for fill light.

[0104] S03. Display the scene picture to be displayed based on the first image information;

[0105] S04. Determine the second interaction instruction of the user for the scene picture to be displayed based on the second image information;

[0106] S05. Respond to the second interaction instruction.

[0107] In some embodiments, in the foregoing S04, determining the second interaction instruction of the user for the scene picture to be displayed based on the second image information includes the following S041 - S042:

[0108] S041. Determine the gesture information of the user according to the second hand image information of the user included in the second image information;

[0109] S042. Determine the second interaction instruction of the user for the scene picture to be displayed according to the gesture information.

[0110] In this solution, both the head - mounted display device and the handheld device are provided with infrared lamps. Through this solution, clear and high - quality environmental images can be collected under the condition that the exposure times of multiple shooting devices are relatively small.

[0111] In some embodiments, when multiple shooting devices collect the target environmental image, in order to improve the exposure accuracy, in addition to obtaining the tracking image, an image information of a gesture can be collected at a relatively small exposure time, and the exposure points of three frames of images can be aligned simultaneously.

[0112] In summary, in Figure 6 In the first case described above, when multiple shooting devices shoot the external environment, due to the different brightnesses in the corresponding directions of multiple cameras, the exposure times are different. In the second and third cases, the purpose is not to obtain the environmental image, but to obtain the gesture and handle images (the darker the environmental image, the better), so the exposure time is set shorter, and the brightness of the gesture and the handle is brighter than the surrounding environment. This is achieved by using the infrared lamp set on the head - mounted display device for supplementary lighting and the active light emission on the handle. At this time, the exposure durations of different shooting devices can be the same.

[0113] In this application, in order to achieve high - frame - rate gesture tracking, the tracking performance can be improved by increasing the tracking frequency (the general gesture tracking frequency of existing VR all - in - one machines is generally 30Hz, and the tracking frequency can be increased to 60Hz, 90Hz), and the tracking accuracy can be improved by using the overlapping area of the camera and shortening the exposure time of the shooting device.

[0114] In some embodiments, when the handheld device is a handle, reference can be made to Figure 7As shown, taking the head-mounted display device 11 and the handle 114 as examples, after the user operates the buttons on the handle 114, control information is generated in the handle processor 1141 in the handle 114. Through the handle wireless chip 1142 provided in the handle 114, the control information is sent to the head-mounted wireless chip 116 in the head-mounted display device 11, and then the head-mounted processor 113 processes the control information. Among them, for the position of the head-mounted processor 113, reference can be made to Figures 3 - 5 As shown, specifically, the head-mounted processor 113 can be located inside the housing.

[0115] In some embodiments, the method further includes: obtaining the initial pose information sent by the handheld device.

[0116] In some other embodiments, in S2023, determining the first interaction instruction of the user for the scene picture according to the target environment image and the control information includes:

[0117] Determining the corresponding gesture information according to the first hand image information, and when the gesture information corresponds to a target operation instruction, taking the target operation instruction as the third instruction of the user for the scene picture;

[0118] Correcting the initial pose information based on the image information of the handheld device to obtain the corrected pose information;

[0119] Determining the fourth instruction of the user for the scene picture according to the corrected pose information and the control information;

[0120] Among them, the first interaction instruction includes the third instruction and the fourth instruction.

[0121] In some embodiments, determining the fourth instruction of the user for the scene picture according to the corrected pose information and the control information includes: determining the selected object selected by the handheld device in the scene picture according to the corrected pose information; determining the fifth instruction for the selected object based on the control information; and determining the fifth instruction for the selected object as the fourth instruction of the user for the scene picture.

[0122] In some embodiments, the handle in the present application is a 6DoF handle.

[0123] In some embodiments, when the current working mode is the second working mode, if a second preset mode switching instruction is detected, the second working mode is adjusted to the first working mode.

[0124] Among them, the second preset mode switching instruction is an instruction triggered by at least one of the following information: second preset gesture information, second preset button information, second preset voice information, second preset handheld device information, wherein the second preset handheld device information is information sent by the handheld device to instruct the head-mounted display device to switch from the second working mode to the first working mode.

[0125] In some embodiments, the second preset gesture information mainly refers to gesture information of a gesture set by a relevant person.

[0126] In some embodiments, the second preset button information mainly refers to the button information of the buttons on the handheld device or the head mounted display device set by relevant personnel.

[0127] In some embodiments, the second preset handheld device information may be preset track information of the handheld device set by relevant personnel, for example, track information formed when the handheld device performs a rectangular action.

[0128] In other embodiments, when the first working mode is the handheld device working mode, the second preset handheld device information may also be information sent by the handheld device indicating that another handheld device has entered a preset area or has been successfully connected.

[0129] In some embodiments, when the head-mounted display device is in the second working mode, one hand of the user is used to interact with the scene screen through gestures, and the other hand interacts with the scene screen through a handheld device. If a second preset mode switching instruction is detected, the second working mode of the head-mounted display device is adjusted to the first working mode. When the head-mounted display device is in the first working mode, the user interacts with the scene screen only through gestures, or the user interacts with the scene screen only through a handheld device.

[0130] In some embodiments, if the current working mode is the gesture working mode, if a third preset mode switching instruction is detected, the gesture working mode is adjusted to the handheld device working mode.

[0131] Among them, the third preset mode switching instruction is an instruction triggered by at least one of the following information: third preset gesture information, third preset button information, third preset voice information, third preset handheld device information, wherein the third preset handheld device information is information sent by the handheld device to indicate the switching mode.

[0132] In some embodiments, the third preset gesture information mainly refers to gesture information of a gesture set by relevant personnel.

[0133] In some embodiments, the third preset button information mainly refers to the button information of the buttons on the handheld device or the head mounted display device set by relevant personnel.

[0134] In some embodiments, the third preset handheld device information may be preset track information of the handheld device set by relevant personnel, for example, track information formed when the handheld device performs a square action.

[0135] In other embodiments, the third preset handheld device information may also be information indicating that both handheld devices have entered a preset area or have been successfully connected.

[0136] In a specific implementation, when the head-mounted display device is in the gesture working mode, the user interacts with the scene screen only through gestures. If a third preset mode switching instruction is detected, the gesture working mode of the head-mounted display device is adjusted to the handheld device working mode. When the head-mounted display device is in the handheld device working mode, the user interacts with the scene screen only through the handheld device.

[0137] In some embodiments, if the current working mode is the handheld device working mode, if a fourth preset mode switching instruction is detected, the gesture working mode is adjusted to the gesture working mode.

[0138] In a specific implementation, when the head-mounted display device is in the handheld device working mode, the user interacts with the scene screen only through the handheld device. If a fourth preset mode switching instruction is detected, the handheld device working mode of the head-mounted display device is adjusted to the gesture working mode. When the head-mounted display device is in the gesture working mode, the user interacts with the scene screen only through gestures.

[0139] Figure 9 A schematic diagram of data transmission of a head mounted display device in a handle working mode is provided as an exemplary embodiment of the present disclosure.

[0140] In some embodiments, in the handle working mode, such as Figure 9 As shown, the head mounted display device can receive data sent by the left handle, and can also receive data sent by the right handle, and each reception of data sent by the left handle and each reception of data sent by the right handle constitutes a cycle.

[0141] In the present disclosure, when the head mounted display device is in gesture working mode, the handle and the head mounted display device are connected at a low working frequency (such as 10Hz), and the IMU data is not transmitted at this time. The handle detects the IMU sensor and button information, and maintains the existing working frequency and gesture working mode before detecting the user action. After the head mounted display device receives the data, the algorithm side does not need to transmit data, nor does it need to calculate the handle positioning information, saving resources.

[0142] In some embodiments, the user can actively switch the current handle working mode of the head-mounted display device to the gesture working mode, or switch the current gesture working mode of the head-mounted display device to the handle working mode by operating the application displayed on the head-mounted display device or tapping the head-mounted display device.

[0143] According to the solution provided by the present disclosure, through the interaction method of the present disclosure, during the process of displaying a scene picture, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, the first working mode is adjusted to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand; wherein, the first working mode is a gesture working mode or a handheld device working mode. The solution of the present application reduces the power consumption of the handheld device during the switching process between gesture control and handheld device control. And in the second working mode, one hand of the user is used to interact with the scene picture through gestures, and the other hand interacts with the scene picture through a handheld device, adjusting the original gesture working mode with only gesture operations or the handheld device working mode with only handheld devices to a mode where one hand of the user interacts with the scene picture through gestures and the other hand interacts with the scene picture through a handheld device, thereby realizing the simultaneous operation of gesture work and handheld device work, and achieving the technical effect of more natural and smooth interaction.

[0144] Figure 10 FIG. 7 is a schematic structural diagram of an interaction device provided for an exemplary embodiment of the present disclosure. The structure includes: a display unit 401 and an adjustment unit 402.

[0145] The display unit 401 is used to display a scene picture;

[0146] The adjustment unit 402 is used to, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand;

[0147] Wherein, the first working mode is a gesture working mode or a handheld device working mode.

[0148] Optionally, the first preset mode switching instruction is an instruction triggered by at least one of the following information: first preset gesture information, first preset key information, first preset voice information, first preset handheld device information, wherein the first preset handheld device information is information sent by the handheld device for instructing mode switching.

[0149] Optionally, in the second working mode, the device is further used for:

[0150] Obtain a target environment image collected by a photographing device, where the target environment image includes first hand image information of a user and image information of the handheld device;

[0151] Obtain control information sent by the handheld device;

[0152] Determine a first interaction instruction of the user for the scene picture according to the target environment image and the control information;

[0153] Respond to the first interaction instruction.

[0154] Optionally, the foregoing device is further configured to:

[0155] Control the infrared lamp disposed on the handheld device to light up;

[0156] When the infrared lamp is lit, control the photographing device to collect the target environment image.

[0157] Optionally, the foregoing device is further configured to:

[0158] Control first image information photographed by the photographing device based on ambient light brightness, where the first image information includes environment information of a head-mounted display device;

[0159] Control the photographing device to photograph second image information with a first preset exposure duration, where the second image information includes second hand image information of the user;

[0160] Display a scene picture to be displayed based on the first image information;

[0161] Determine a second interaction instruction of the user for the scene picture to be displayed based on the second image information;

[0162] Respond to the second interaction instruction.

[0163] Optionally, when the current working mode is the second working mode, if a second preset mode switching instruction is detected, adjust the second working mode to the first working mode.

[0164] Optionally, the foregoing device is further configured to: when the current working mode is the gesture working mode, if a third preset mode switching instruction is detected, adjust the gesture working mode to the handheld device working mode.

[0165] It should be understood that the device embodiments and the method embodiments can correspond to each other, and similar descriptions can refer to the method embodiments. To avoid repetition, they will not be elaborated here. Specifically, the device can execute the above method embodiments, and the foregoing and other operations and / or functions of each module in the device respectively correspond to the corresponding processes in each method in the above method embodiments. For the sake of brevity, they will not be elaborated here.

[0166] The device of the embodiments of the present disclosure has been described above from the perspective of functional modules in combination with the drawings. It should be understood that the functional modules can be implemented in the form of hardware, or in the form of instructions in software, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in the present disclosure can be completed by the integrated logic circuit in hardware in the processor and / or instructions in software. The steps of the method disclosed in combination with the embodiments of the present disclosure can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps in the above method embodiments.

[0167] Figure 11 is a schematic block diagram of an electronic device provided by the embodiments of the present disclosure. The electronic device may include:

[0168] A memory 501 and a processor 502. The memory 501 is used to store a computer program and transmit the program code to the processor 502. In other words, the processor 502 can call and run the computer program from the memory 501 to implement the method in the embodiments of the present disclosure.

[0169] For example, the processor 502 can be used to execute the above method embodiments according to the instructions in the computer program.

[0170] In some embodiments of the present disclosure, the processor 502 may include, but is not limited to:

[0171] A general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and so on.

[0172] In some embodiments of the present disclosure, the memory 501 includes, but is not limited to:

[0173] A volatile memory and / or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0174] In some embodiments of the present disclosure, the computer program may be divided into one or more modules, and the one or more modules are stored in the memory 501 and executed by the processor 502 to complete the method provided by the present disclosure. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0175] As Figure 11 shown, the electronic device may further include:

[0176] A transceiver 503, and the transceiver 503 may be connected to the processor 502 or the memory 501.

[0177] Among them, the processor 502 may control the transceiver 503 to communicate with other devices. Specifically, it may send information or data to other devices, or receive information or data sent by other devices. The transceiver 503 may include a transmitter and a receiver. The transceiver 503 may further include an antenna, and the number of antennas may be one or more.

[0178] It should be understood that the various components in the electronic device are connected through a bus system. Among them, the bus system includes, in addition to a data bus, a power bus, a control bus, and a status signal bus.

[0179] The present disclosure also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a computer, the computer can execute the methods in the above method embodiments. Or, the embodiments of the present disclosure also provide a computer program product including instructions. When the instructions are executed by a computer, the computer executes the methods in the above method embodiments.

[0180] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present disclosure are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a digital video disc (DVD)), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0181] According to one or more embodiments of the present disclosure, an interaction method is provided. The method includes:

[0182] Display a scene picture;

[0183] When the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjust the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand;

[0184] Wherein, the first working mode is a gesture working mode or a handheld device working mode.

[0185] According to one or more embodiments of the present disclosure, the first preset mode switching instruction is an instruction triggered by at least one of the following information: first preset gesture information, first preset key information, first preset voice information, first preset handheld device information, wherein the first preset handheld device information is information sent by the handheld device for indicating mode switching.

[0186] According to one or more embodiments of the present disclosure, in the second working mode, the method further includes:

[0187] Obtaining a target environment image collected by a photographing device, wherein the target environment image includes first hand image information of a user and image information of the handheld device;

[0188] Obtaining control information sent by the handheld device;

[0189] Determining a first interaction instruction of the user for the scene picture according to the target environment image and the control information;

[0190] Responding to the first interaction instruction.

[0191] According to one or more embodiments of the present disclosure, the method further includes:

[0192] Controlling an infrared lamp disposed on the handheld device to be lit;

[0193] When the infrared lamp is lit, controlling the photographing device to collect the target environment image.

[0194] According to one or more embodiments of the present disclosure, the method further includes:

[0195] Controlling first image information photographed by the photographing device based on ambient light brightness, the first image information including environmental information of a head-mounted display device;

[0196] Controlling the photographing device to photograph second image information with a first preset exposure duration, the second image information including second hand image information of the user;

[0197] Displaying a scene picture to be displayed based on the first image information;

[0198] Determining a second interaction instruction of the user for the scene picture to be displayed based on the second image information;

[0199] Responding to the second interaction instruction.

[0200] According to one or more embodiments of the present disclosure, when the current working mode is the second working mode, if a second preset mode switching instruction is detected, the second working mode is adjusted to the first working mode.

[0201] According to one or more embodiments of the present disclosure, the method includes: when the current working mode is the gesture working mode, if a third preset mode switching instruction is detected, adjusting the gesture working mode to the handheld device working mode.

[0202] According to one or more embodiments of the present disclosure, an interaction device is provided, the device includes:

[0203] A display unit for displaying a scene picture;

[0204] An adjustment unit for, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjusting the first working mode to the second working mode, in the second working mode, the user interacts with the scene picture through the gesture of one hand and interacts with the scene picture through a handheld device held by the other hand;

[0205] Wherein, the first working mode is a gesture working mode or a handheld device working mode.

[0206] According to one or more embodiments of the present disclosure, in the second working mode, the device is further configured to:

[0207] Obtain a target environment image collected by a photographing device, wherein the target environment image includes first hand image information of a user and image information of the handheld device;

[0208] Obtain control information sent by the handheld device;

[0209] Determine a first interaction instruction of the user for the scene picture according to the target environment image and the control information;

[0210] Respond to the first interaction instruction.

[0211] According to one or more embodiments of the present disclosure, the device is further configured to:

[0212] Control the infrared lamp arranged on the handheld device to be lit;

[0213] When the infrared lamp is lit, control the photographing device to collect the target environment image.

[0214] According to one or more embodiments of the present disclosure, the device is further configured to:

[0215] Control the first image information captured by the photographing device based on the ambient light brightness, the first image information includes the ambient information of the head-mounted display device;

[0216] Control the photographing device to photograph second image information with a first preset exposure duration, where the second image information includes second hand image information of the user;

[0217] Display a scene picture to be displayed based on the first image information;

[0218] Determine a second interaction instruction of the user for the scene picture to be displayed based on the second image information;

[0219] Respond to the second interaction instruction.

[0220] According to one or more embodiments of the present disclosure, in the case where the current working mode is the second working mode, if a second preset mode switching instruction is detected, adjust the second working mode to the first working mode.

[0221] According to one or more embodiments of the present disclosure, the device is further configured to: when the current working mode is the gesture working mode, if a third preset mode switching instruction is detected, adjust the gesture working mode to the handheld device working mode.

[0222] According to one or more embodiments of the present disclosure, there is provided an electronic device, including:

[0223] A processor; and

[0224] A memory for storing executable instructions of the processor;

[0225] Wherein, the processor is configured to execute the foregoing methods by executing the executable instructions.

[0226] According to one or more embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the foregoing methods are implemented.

[0227] Those of ordinary skill in the art can realize that the modules and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present disclosure.

[0228] In several embodiments provided by the present disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of the devices or modules can be in electrical, mechanical, or other forms.

[0229] The modules described as separate components may or may not be physically separated. The components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, in each embodiment of the present disclosure, the various functional modules can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module.

[0230] The above is only the specific implementation manner of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present disclosure can easily think of changes or substitutions, which should all be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An interaction method, characterized in that, The method includes: Displaying a scene picture; When the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjusting the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand; Wherein, the first working mode is a gesture working mode or a handheld device working mode.

2. The method according to claim 1, characterized in that, The first preset mode switching instruction is an instruction triggered by at least one of the following information: first preset gesture information, first preset key information, first preset voice information, first preset handheld device information. Wherein, the first preset handheld device information is information sent by the handheld device for indicating mode switching.

3. The method according to claim 1, wherein In the second working mode, the method further includes: Obtaining a target environment image collected by a photographing device, wherein the target environment image includes first hand image information of the user and image information of the handheld device; Obtaining control information sent by the handheld device; Determining a first interaction instruction of the user for the scene picture according to the target environment image and the control information; Responding to the first interaction instruction.

4. The method according to claim 3, characterized in that, The method further includes: Controlling an infrared lamp disposed on the handheld device to be lit; When the infrared lamp is lit, controlling the photographing device to collect the target environment image.

5. The method according to claim 1, wherein The method further includes: Controlling the photographing device to shoot first image information based on the ambient light brightness, and the first image information includes environmental information of a head-mounted display device; Controlling the photographing device to shoot second image information with a first preset exposure duration, and the second image information includes second hand image information of the user; Displaying a scene picture to be displayed based on the first image information; Determining a second interaction instruction of the user for the scene picture to be displayed based on the second image information; Responding to the second interaction instruction.

6. The method according to claim 1, wherein When the current working mode is the second working mode, if a second preset mode switching instruction is detected, adjusting the second working mode to the first working mode.

7. The method according to claim 1, wherein The method includes: when the current working mode is the gesture working mode, if a third preset mode switching instruction is detected, adjusting the gesture working mode to the handheld device working mode.

8. An interactive device, characterized in that, The device includes: A display unit for displaying a scene picture; An adjustment unit for, when the current working mode is the first working mode, if a first preset mode switching instruction is detected, adjusting the first working mode to the second working mode. In the second working mode, the user interacts with the scene picture through the gestures of one hand and interacts with the scene picture through a handheld device held by the other hand; Wherein, the first working mode is a gesture working mode or a handheld device working mode.

9. An electronic device, characterized in that, Includes: A processor; And A memory for storing executable instructions of the processor; Wherein, the processor is configured to execute the method according to any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when executed by a processor, implements the method according to any one of 1-7.

Images