Information interaction method and device, head-mounted display, wearable equipment and medium

Abstract

The embodiment of the invention provides an information interaction method and device, a head-mounted display, wearable equipment and a medium, and relates to the technical field of augmented reality. The method comprises the following steps: acquiring first exposure image information based on an initial exposure mode, and acquiring first space displacement information of the head-mounted display; determining a head-mounted display pose of the head-mounted display according to the first exposure image information and the first spatial displacement information; when the exposure signal of the image collector is determined, sending the exposure signal to the handle so as to control a plurality of light-emitting devices of the handle to be switched to target brightness within a target exposure duration; acquiring second exposure image information based on the target exposure mode within the target exposure duration, and acquiring second space displacement information of the handle; determining a handle pose of the handle based on the second exposure image information and the second space displacement information; and determining virtual interaction content output by the head-mounted display based on the head-mounted display pose and the handle pose. According to the embodiment of the invention, image acquisition is carried out alternately through the initial exposure mode and the target exposure mode, and meanwhile, the target exposure mode is synchronous with the target brightness state of the light-emitting device, so that more accurate pose positioning is realized.

Description

Technical Field

[0001] The present application relates to the field of extended reality technology, and in particular to an information interaction method, apparatus, head-mounted display, wearable device and medium. Background Art

[0002] With the continuous advancement of intelligent technology, the application of extended reality (XR) technology in various fields has become more extensive and in-depth. XR technology is expected to play a greater role in entertainment, education, medical care, construction and other fields, changing the way people live and work.

[0003] Wearable devices based on XR technology generally include a head-mounted display and a handle. The head-mounted display provides users with visual virtual content, and the handle can provide stable and efficient input for the head-mounted display. It can also provide users with tactile feedback when interacting with virtual content. Therefore, stable communication between the head-mounted display and the handle, as well as precise positioning of the handle and the head-mounted display are very important.

[0004] In the related technology, ultrasound or gyroscopes can be used to locate and track the spatial posture of the handle and head-mounted display, and provide virtual content to the user based on the spatial posture of the handle and head-mounted display. However, there is a problem of low posture positioning accuracy, which affects the user's virtual interaction experience. Summary of the invention

[0005] The present application provides an information interaction method, device, head-mounted display, wearable device and medium, which are used to solve the problem of low posture positioning accuracy in the prior art.

[0006] According to one aspect of the present application, an information interaction method is provided, which is applied to a head-mounted display of a smart wearable device, wherein the smart wearable device further comprises a handle configured with a plurality of light-emitting devices, and the head-mounted display is configured with an image collector, and the method comprises:

[0007] Based on the initial exposure mode, first exposure image information is collected, and first spatial displacement information of the head mounted display is obtained; according to the first exposure image information and the first spatial displacement information, a head mounted display posture of the head mounted display is determined; wherein the plurality of light emitting devices maintain an initial brightness in the initial exposure mode;

[0008] When the exposure signal of the image collector is determined, an exposure signal is sent to the handle to control the multiple light-emitting devices of the handle to switch to the target brightness within the target exposure time; wherein the target brightness is greater than the initial brightness; and the exposure time of the initial exposure mode is greater than the target exposure time;

[0009] Within the target exposure time, collecting second exposure image information based on the target exposure mode, and obtaining second spatial displacement information of the handle; determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information;

[0010] Based on the head display posture and the handle posture, the virtual interactive content output by the head mounted display is determined.

[0011] In a possible implementation, the determining of the exposure signal of the image collector includes:

[0012] The exposure signal of the image collector is determined at every preset time interval; wherein the preset time interval is determined based on the frame rate of the target exposure mode.

[0013] In yet another possible implementation, the step of determining the exposure signal of the image collector includes:

[0014] An exposure signal is determined based on the exposure state of the image collector; wherein the sending time of the exposure signal is placed before the start time of the target exposure mode.

[0015] In yet another possible implementation, after acquiring the second exposure image information based on the target exposure mode, the method includes:

[0016] Storing the second exposure image information into a preset exposure image information set;

[0017] Before determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information, the method further includes:

[0018] From the exposure image information set, second exposure image information is extracted.

[0019] In another possible implementation, the second exposure image information is assigned an exposure tag;

[0020] Extracting second exposure image information from the exposure image information set includes:

[0021] According to the exposure tag, the second exposure image information is obtained by screening from the exposure image information set.

[0022] In another possible implementation, the extracting the second exposure image information from the exposure image information set includes:

[0023] For each exposure image information in the exposure image information set, sampling is performed to obtain a plurality of pixel points of each exposure image information;

[0024] Determine the brightness of each exposure image information based on the grayscale values ​​of the plurality of pixels;

[0025] When the brightness is less than a preset brightness threshold, the exposure image information corresponding to the brightness is determined as the second exposure image information.

[0026] In another possible implementation, the determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information includes:

[0027] Performing posture prediction according to the second exposure image information and the second spatial displacement information to obtain initial posture information;

[0028] Acquire first spatial displacement information of the head mounted display;

[0029] The initial displacement information is corrected based on the first spatial displacement information to obtain the handle posture of the handle.

[0030] In another possible implementation, performing posture prediction according to the second exposure image information and the second spatial displacement information to obtain initial posture information includes:

[0031] Determining relative position information of at least two light emitting devices in the second exposure image information;

[0032] Determining serial numbers of at least two light emitting devices based on the relative position information;

[0033] Position prediction is performed according to the serial numbers of at least two light-emitting devices and the second spatial displacement information to obtain initial position information.

[0034] According to another aspect of an embodiment of the present application, an information interaction device is provided, which is applied to a head-mounted display of a smart wearable device, wherein the smart wearable device further includes a handle configured with a plurality of light-emitting devices, and the head-mounted display is configured with an image collector, and the device includes:

[0035] A first determination module is used to collect first exposure image information based on an initial exposure mode and obtain first spatial displacement information of a head mounted display; determine a head mounted display posture of the head mounted display according to the first exposure image information and the first spatial displacement information; wherein the plurality of light emitting devices maintain an initial brightness in the initial exposure mode;

[0036] A switching module is used to determine the exposure signal of the image collector and send an exposure signal to the handle to control the multiple light-emitting devices of the handle to switch to the target brightness within the target exposure time; wherein the target brightness is greater than the initial brightness; and the exposure time of the initial exposure mode is greater than the target exposure time;

[0037] A second determination module is used to collect second exposure image information based on the target exposure mode within the target exposure time, and obtain second spatial displacement information of the handle; based on the second exposure image information and the second spatial displacement information, determine the handle posture of the handle;

[0038] The interaction module is used to determine the virtual interactive content output by the head-mounted display based on the head display posture and the handle posture.

[0039] According to another aspect of the present application, a head-mounted display is provided, which includes: a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the method shown in the first aspect of the present application.

[0040] According to another aspect of the present application, a wearable device is provided, comprising a handle and a head-mounted display as shown in the third aspect of the present application, wherein the handle is used to cooperate with the head-mounted display to perform the steps of the method shown in the first aspect of the present application.

[0041] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method shown in the first aspect of the present application are implemented.

[0042] According to another aspect of the present application, a computer program product is provided, including a computer program, which implements the steps of the method shown in the first aspect of the present application when executed by a processor.

[0043] The beneficial effects of the technical solution provided by this application are:

[0044] The information interaction method, apparatus, head-mounted display, wearable device and medium provided in the present application can collect first exposure image information through an initial exposure mode, and obtain first spatial displacement information of the head-mounted display; determine the head display posture of the head-mounted display according to the first exposure image information and the first spatial displacement information; when the exposure signal of the image collector is determined, send an exposure signal to the handle to control multiple light-emitting devices of the handle to switch to the target brightness within the target exposure time; within the target exposure time, collect second exposure image information based on the target exposure mode, and obtain second spatial displacement information of the handle; determine the handle posture of the handle based on the second exposure image information and the second spatial displacement information, and determine the virtual interactive content output by the head-mounted display based on the head display posture and the handle posture. The target brightness in the present application is greater than the initial brightness, and the target exposure time can be in the microsecond level, which is much lower than the exposure time of the initial exposure mode. Image acquisition can be performed alternately in the initial exposure mode and the target exposure mode: on the one hand, image acquisition can be performed for light-emitting devices of multiple target brightness in the target exposure mode. Since the target exposure time can be in the microsecond level, interfering stray light in the environment can be filtered out in the target exposure mode, and the position and posture of the handle can be located more accurately; on the other hand, the initial brightness state of multiple light-emitting devices in the initial exposure mode is maintained, and images of the surrounding environment are acquired. This avoids misrecognition when the handle is blocked while reducing the energy consumption of the light-emitting devices, thereby improving the accuracy of the head-mounted display position information of the head-mounted display.

[0045] At the same time, by accurately positioning the obtained handle posture and head display posture, this application can make the virtual interactive content output by the head-mounted display more in line with the user's perception, thereby improving the user's virtual interactive experience. At the same time, the target exposure time in this application is short, which can reduce the heat caused by the high brightness state of the light-emitting device, further improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A flowchart of an information interaction method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of an application scenario of an information interaction method provided in an embodiment of the present application;

[0049] Figure 3A schematic diagram of short exposure and LED highlight synchronization in an information interaction method provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the structure of an XR device in an information interaction method provided in an embodiment of the present application;

[0051] Figure 5 A timing diagram of an exemplary information interaction method provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of an information interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0054] Various structural schematic diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may further design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0055] In the context of the present disclosure, when a layer / element is referred to as being "on" another layer / element, the layer / element may be directly on the other layer / element or an intervening layer / element may exist between them. In addition, if a layer / element is "on" another layer / element in one orientation, the layer / element may be "below" the other layer / element when the orientation is reversed.

[0056] First, the terms involved in this application are explained:

[0057] Extended Reality (XR) refers to the combination of reality and virtuality through computers to create a virtual environment for human-computer interaction. It is also a general term for multiple technologies such as AR (Augmented Reality), VR (Virtual Reality), and MR (Mixed Reality). By integrating the visual interaction technologies of the three, it brings the experiencer an "immersive feeling" of seamless transition between the virtual world and the real world.

[0058] The core interaction method of XR devices is still the handle. The handle usually contains a trigger button, a home button (start button), an OK button and other custom buttons for interacting with HMD (Head Mount Display) devices. Compared with bare hands, the handle can provide more stable and efficient input. The linear motor on the handle can also simulate touch and provide users with tactile feedback when interacting with virtual content. The core difficulty of handle interaction lies in the real-time tracking of the handle posture. In the existing technology, the handle can be positioned and tracked based on ultrasonic or electromagnetic solutions, but there are problems of low accuracy and susceptibility to environmental interference.

[0059] Based on the above technical problems, the present application performs image acquisition alternately through an initial exposure mode and a target exposure mode: on the one hand, in the target exposure mode, image acquisition is performed for multiple light-emitting devices of target brightness. Since the target exposure time can be in the microsecond level, the interfering stray light in the environment can be filtered out in the target exposure mode, and the position and posture of the handle can be located more accurately; on the other hand, the initial brightness of multiple light-emitting devices in the initial exposure mode is maintained, and images of the surrounding environment are acquired. This avoids misidentification when the handle is blocked while reducing the energy consumption of the light-emitting devices, thereby improving the accuracy of the position and posture information of the head-mounted display and enhancing the user's virtual interactive experience.

[0060] The following is a detailed description of the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0061] The present application provides an information interaction method, such as Figure 1 As shown, a head mounted display that can be applied to a smart wearable device, the head mounted display is configured with an image collector, the smart wearable device further includes a handle configured with a plurality of light emitting devices, and the method includes:

[0062] S101, collecting first exposure image information based on an initial exposure mode, and obtaining first spatial displacement information of a head mounted display; and determining a head mounted display posture of the head mounted display according to the first exposure image information and the first spatial displacement information.

[0063] Wherein, a plurality of light emitting devices maintain initial brightness in an initial exposure mode.

[0064] Optionally, the number of handles of the smart wearable device may be two, and the two handles may be used as inputs of a head-mounted display, and the head-mounted display may output virtual interactive content based on the inputs; Figure 2As shown, the head-mounted display can be in the form of wearable glasses 202. The user can wear the wearable glasses 202 to obtain virtual visual information. At the same time, the user's two hands can hold the handle 201 to perform corresponding interactive operations, so that the user can immersively experience virtual interactive processes such as virtual games, private theaters, and online live broadcasts. A plurality of light-emitting devices 203 are arranged on the handle 201. The wearable glasses 202 can collect the light spots or surrounding environment of the light-emitting devices 203 based on the image collector to determine the posture information of the handle 201 or the head-mounted display.

[0065] Optionally, the head mounted display may be equipped with a camera, multiple cameras, and an inertial measurement unit (IMU); for example, 12 cameras may be arranged at different positions of the head mounted display, and at least one IMU may be arranged on the head mounted display. IMUs are divided into six-axis and nine-axis. The six-axis IMU includes a gyroscope and an accelerometer, and the nine-axis IMU includes a gyroscope, an accelerometer, and a magnetometer.

[0066] Specifically, the head-mounted display can collect first exposure image information of the surrounding environment based on the camera in the initial exposure mode, then obtain first spatial displacement information based on the IMU unit, and determine the head-mounted display posture information of the head-mounted display itself according to the first exposure image information and the first spatial displacement information.

[0067] S102, when the exposure signal of the image collector is determined, an exposure signal is sent to the handle to control the multiple light-emitting devices of the handle to switch to the target brightness within the target exposure time.

[0068] Among them, the target brightness is greater than the initial brightness; and the exposure time of the initial exposure mode is greater than the target exposure time.

[0069] Among them, multiple light-emitting devices maintain a high-brightness state within a target exposure time; the high-brightness state is triggered based on an exposure signal sent by the head-mounted display to the handle.

[0070] Specifically, the head mounted display sends an exposure signal to the handle to control the multiple light emitting devices of the handle to switch from an initial brightness to a target brightness and enter a target exposure mode with a target exposure duration.

[0071] Optionally, the head-mounted display can send an exposure signal to the handle based on low-latency 2.4G private protocol wireless transmission technology. After the handle receives the exposure signal, the multiple light-emitting devices on the handle remain in a high-brightness state within the target exposure time, and the camera on the head-mounted display enters a target exposure mode for the target exposure time.

[0072] The embodiment of the present application uses wireless transmission technology to send the exposure signal of the camera to the handle, so that when the camera is in the target exposure mode, the multiple light-emitting devices on the handle maintain the target brightness, so that the short exposure of the camera is synchronized with the high-brightness state of the light-emitting devices. Since the camera can filter out interfering stray light in the environment in the short exposure mode, capturing the handle image in the short exposure mode will be more robust, effectively enhancing the accuracy of subsequent handle posture positioning.

[0073] S103, within the target exposure time, collecting second exposure image information based on the target exposure mode, and obtaining second spatial displacement information of the handle; and determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information.

[0074] Specifically, the head mounted display can collect the second exposure image information of the handle within the target exposure time based on the camera, and then obtain the second spatial displacement information of the handle based on the IMU unit. The second spatial displacement information may include inertial parameters such as angular velocity, angular acceleration, and magnetic field strength.

[0075] Specifically, the head-mounted display can obtain the spot combination information of multiple light-emitting devices in the second exposure image information, and then calculate the spot combination information using the multi-view geometry principle to obtain the six degrees of freedom (6Dof) posture of each light-emitting device in the coordinate system of the head-mounted display; then, based on the above 6Dof posture and the second space displacement information, determine the handle posture information of the handle.

[0076] Optionally, the light-emitting device can be an infrared light-emitting diode (LED), and the camera on the head-mounted display can sample the environment based on a fixed frame rate. When the LED on the handle is highlighted, it can be photographed by the camera and appear as a light spot in the sampled image. The multi-view geometry knowledge in computer vision can be used to solve the 6Dof position of the light spot in the camera coordinate system, and then obtain the 6Dof posture of the light spot in the head-mounted display system coordinate system.

[0077] Optionally, the camera of the head-mounted display can capture environmental images in the form of alternating long and short exposures. The exposure modes of alternating long and short exposures can be short-long, long-short, short-short-long, short-long-long and other similar combinations, which are not specifically limited in the embodiments of the present application. Taking the 60fps frame rate as an example, the camera can be set to an alternating working mode of short exposure with a target exposure time of 30 frames and long exposure with an initial exposure time of 30 frames. The exposure time of the above-mentioned long exposure, i.e., the initial exposure mode, is in the millisecond level, and the short exposure and target exposure modes are specially designed for infrared LED recognition. The target exposure time is in the us level. Except for the LED and ambient light, other environments are photographed as black pictures. At the same time, the exposure time of the short exposure matches the high-brightness state of the LED, and the infrared LED flickers with the short exposure time of the camera. Figure 3 As shown, the camera is in a mode of alternating long and short exposure, that is, short exposure of the first exposure time and long exposure of the second exposure time are cycled alternately, the infrared LED is in a high-brightness state during the first exposure time corresponding to the short exposure, and is in a low-brightness state at other times, and the short exposure is synchronized with the high-brightness state.

[0078] In the embodiment of the present application, the long exposure and short exposure alternating working modes of the multiple cameras of the head-mounted display can be realized by configuring the camera register parameters. The head-mounted display can send an exposure signal to the handle based on the exposure state of the camera, so that the short exposure of the target exposure time is synchronized with the high brightness state of the light-emitting device. The light-emitting device is in an alternating mode of high brightness state and low brightness state based on the exposure state.

[0079] S104: Determine virtual interactive content output by the head mounted display based on the head mounted display posture and the handle posture.

[0080] Among them, the above-mentioned virtual interactive content may include light and shadow virtual visual content output by the head-mounted display, and may also include simulated touch provided by the handle; the above-mentioned simulated touch can be achieved by the linear motor on the handle.

[0081] The embodiment of the present application performs image acquisition alternately through the initial exposure mode and the target exposure mode: on the one hand, image acquisition can be performed for multiple light-emitting devices of target brightness in the target exposure mode. Since the target exposure time can be in the microsecond level, the interfering stray light in the environment can be filtered out in the target exposure mode, and the position of the handle can be located more accurately; on the other hand, the initial brightness state of multiple light-emitting devices in the initial exposure mode is maintained, and the image of the surrounding environment is acquired. Under the premise of reducing the energy consumption of the light-emitting device, the misidentification when the handle is blocked is avoided, and the accuracy of the head display position information of the head-mounted display is improved. At the same time, by accurately locating the obtained handle position and head display position, the present application can make the virtual interactive content output by the head-mounted display more in line with the user's perception and enhance the user's virtual interactive experience. At the same time, the short target exposure time in the present application can reduce the heat caused by the high brightness state of the light-emitting device, and further enhance the user experience.

[0082] In an embodiment of the present application, a possible implementation method is provided, wherein the above-mentioned determination of the exposure signal of the image collector includes:

[0083] The exposure signal of the image collector is determined at every preset time interval; wherein the preset time interval is determined based on the frame rate of the target exposure mode.

[0084] In some embodiments, the frame rate of the image collector, i.e., the camera, may be 60 fps, and one frame lasts for The target exposure mode, i.e., short exposure, and the initial exposure mode, i.e., long exposure, can be performed alternately at intervals. Then, the interval between short exposures is That is, the above preset time interval can be set as

[0085] Optionally, the time interval may be calibrated after each acquisition of a preset number of frames of image information to ensure accurate sending of the exposure signal.

[0086] In an embodiment of the present application, the time interval can be determined by the frame rate of the target exposure mode, and the head-mounted display sends an exposure signal to the handle based on the time interval, which can achieve synchronization between the initial exposure mode and the target brightness state of the light-emitting device, laying a good foundation for subsequent precise positioning of the handle.

[0087] In an embodiment of the present application, a possible implementation method is provided, wherein the above-mentioned determination of the exposure signal of the image collector includes:

[0088] An exposure signal is determined based on the exposure state of the image collector; wherein the sending time of the exposure signal is placed before the start time of the target exposure mode.

[0089] Optionally, the timing of sending the exposure signal can be determined according to the start time of the target exposure mode of the camera, and the timing of sending the exposure signal can be advanced relative to the target exposure mode, i.e., short exposure, according to the delay size, until the target exposure mode is synchronized with the target brightness of the light-emitting device. After receiving the exposure signal, the handle end switches the LED to a high-brightness state, and after the target exposure time, the LED turns to a low-brightness state, waiting for the next exposure signal to arrive, and so on.

[0090] Optionally, the camera frame rate can be 90fps, so one frame lasts It can be set that after every 1 frame of long exposure, 2 frames of short exposure are performed, that is, long and short exposures are cycled in sequence; after the camera is started, the long exposure can be set by default, and the head-mounted display can send exposure signals to the handle in the second frame, third frame, fifth frame, sixth frame, etc.

[0091] For example, the exposure signal can be sent to the handle within 1ms before the camera starts the target exposure mode, i.e., short exposure. Due to the delay in the signal being transmitted to the handle, when the exposure signal is placed before the start-up moment of the target exposure mode, the handle controls the LED to be highlighted when receiving the exposure signal, and the camera is in the target exposure mode.

[0092] The present application ensures the synchronization of the target exposure mode with the target brightness state of the light-emitting device by placing the sending time of the exposure signal before the start time of the target exposure mode, and sending the exposure signal to the handle before the exposure state of the image acquisition device is the target exposure mode, thereby further improving the positioning accuracy of the handle.

[0093] A possible implementation method is provided in the embodiment of the present application.

[0094] After acquiring the second exposure image information based on the target exposure mode, the method includes:

[0095] Storing the second exposure image information into a preset exposure image information set;

[0096] Before determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information, the method further includes:

[0097] From the exposure image information set, second exposure image information is extracted.

[0098] In the embodiment of the present application, by storing the images output by the image collector, i.e., the camera, the above image information can be sorted from front to back based on the acquisition time sequence, so as to facilitate the subsequent distinction between the first exposure image information and the second exposure image information. The step of selecting the second exposure image information from the exposure image information set will be described in detail below.

[0099] A possible implementation method is provided in an embodiment of the present application, where the second exposure image information is assigned an exposure tag;

[0100] Extracting second exposure image information from the exposure image information set includes:

[0101] According to the exposure tag, the second exposure image information is obtained by screening from the exposure image information set.

[0102] Optionally, the image collector can add an exposure tag to the collected image during short exposure, and the exposure tag can be a timestamp, which is not specifically limited in the embodiment of the present application. Then, the second exposure image information can be searched according to the exposure tag.

[0103] In an embodiment of the present application, a possible implementation method is provided, wherein the extracting the second exposure image information from the exposure image information set includes:

[0104] S201 , for each exposure image information in the exposure image information set, sampling is performed to obtain a plurality of pixel points of each exposure image information.

[0105] S202, determining the brightness of each exposure image information based on the grayscale values ​​of multiple pixels.

[0106] S203: When the brightness is less than a preset brightness threshold, the exposure image information corresponding to the brightness is determined as the second exposure image information.

[0107] Taking the light-emitting device as LED as an example, since the target exposure time is designed for the identification of the light-emitting device, which is in the microsecond level, in the target exposure mode, except for the LED and the ambient light, other environments are photographed as black pictures, that is, the second exposure image information is almost a completely black image; therefore, random sampling can be performed on each second exposure image information, and the brightness of the exposure image information can be predicted based on the grayscale values ​​of multiple pixel points obtained by sampling. When the brightness is less than the preset brightness threshold, the second exposure image information is found and obtained.

[0108] The embodiment of the present application distinguishes the first exposure image information from the second exposure image information through the exposure mark or image brightness of the image information; the second exposure image information can be used to identify the light-emitting device for tracking the handle, and the first exposure image information can be used to identify the surrounding environment of the head-mounted display for tracking the head-mounted display, which effectively improves the posture robustness of the handle and the head-mounted display.

[0109] In an embodiment of the present application, a possible implementation method is provided, wherein the determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information includes:

[0110] S301, performing posture prediction according to the second exposure image information and the second spatial displacement information to obtain initial posture information.

[0111] Specifically, the posture prediction can be performed based on a pre-trained deep learning model, and the second exposure image information and the second spatial displacement information are input into the deep learning model to output the initial posture of the handle.

[0112] S302: Acquire first spatial displacement information of a head mounted display.

[0113] Specifically, the first spatial displacement information of the head mounted display itself may be acquired based on an inertial measurement unit configured by the head mounted display.

[0114] S303: Correct the initial displacement information based on the first spatial displacement information to obtain handle posture information of the handle.

[0115] Optionally, since the initial displacement information is based on the coordinate system of the handle, and the first spatial displacement information is based on the coordinate system of the head mounted display, the initial displacement information can be converted into a coordinate system according to the coordinate system of the first spatial displacement information to obtain the handle posture information of the handle.

[0116] The embodiment of the present application predicts the posture of the handle through the second exposure image information and the second spatial displacement information of the handle, and then unifies the first spatial displacement information and the initial posture of the head-mounted display in the coordinate system. The present application integrates image information and spatial displacement information, effectively enhancing the accuracy of positioning.

[0117] A possible implementation method is provided in an embodiment of the present application. The pose prediction is performed according to the second exposure image information and the second spatial displacement information to obtain the initial pose information, including:

[0118] S401, determining relative position information of at least two light-emitting devices in second exposure image information.

[0119] Optionally, the light-emitting device is displayed in the second exposure image information in the form of a light spot, and the relative position information of the light-emitting device can be determined according to the geometric features of the light spot.

[0120] In the embodiment of the present application, each handle can be equipped with multiple light-emitting devices, and the distribution structure of the light-emitting devices can be specially designed; taking the light-emitting devices as LEDs as an example, the distribution structure feature can be: in each rotation posture of the handle, the relative positions between the LEDs are unique. The above distribution structure feature can be achieved based on the structural design and the size and shape design of the LEDs.

[0121] S402: Determine serial numbers of at least two light-emitting devices based on relative position information.

[0122] In the embodiment of the present application, since the distribution structure of the light-emitting devices is specially designed, the serial numbers of at least two light-emitting devices are derived according to the relative position information.

[0123] S403, performing posture prediction according to the serial numbers of at least two light-emitting devices and the second spatial displacement information to obtain initial posture information.

[0124] Specifically, the initial position information may be obtained by calculating based on the serial numbers of at least two light-emitting devices and the second spatial displacement information through the principle of multi-view geometry.

[0125] The embodiment of the present application can determine the numbers of at least two light-emitting devices by determining the relative position information of at least two light-emitting devices in the second exposure image information and combining the special design of the distribution structure of the light-emitting devices, and then perform posture prediction based on the numbers of the at least two light-emitting devices and the second spatial displacement information to obtain initial posture information; the present application integrates image information and spatial displacement information, which can make the prediction of initial posture information more accurate.

[0126] In order to better understand the above information interaction method, Figure 4 and Figure 5 An example of the information interaction method of the present application is described in detail, such as Figure 4 As shown, it is a schematic diagram of the structure of an XR device, including a handle 4001 and a head-mounted display 4002, wherein the handle 4001 includes a first communication device 401, a microprocessor 402, a power supply 403, a first inertial measurement unit 404, and an infrared LED 405, and the head-mounted display 4002 includes a second communication device 406, a 6Dof camera 407, a second inertial measurement unit 408, and a mainboard 409. The handle 4001 and the head-mounted display 4002 perform information exchange based on the first communication device 401 and the second communication device 406, respectively, and the method includes the following steps:

[0127] S501 , the head mounted display 4002 determines an exposure signal based on the exposure state of the 6Dof camera 407 , and sends the exposure signal to the handle 4001 .

[0128] The sending time of the exposure signal is placed before the start time of the short exposure.

[0129] Optionally, the strobe signal of the 6Dof camera 407 may be used as an exposure signal, and the exposure signal is sent out with a relatively short exposure advance timing according to the delay size, until the first exposure is synchronized with the high brightness state of the infrared LED 405 .

[0130] Optionally, the alternating working mode of long exposure and short exposure of multiple cameras of the head mounted display 4002 can be realized by configuring the camera register parameters. Taking the frame rate of 60fps as an example, the alternating working mode of the camera can be set to a short exposure of 30 frames for the first exposure time and a long exposure of 30 frames for the second exposure time. The first exposure time can be 100μ, and the second exposure time can be 1ms-5ms.

[0131] S502, after receiving the exposure information, the handle 4001 keeps the multiple infrared LEDs 405 in a high brightness state within the first exposure duration, and at the same time the 6Dof camera 407 performs a short exposure of the first exposure duration.

[0132] S503 , the head mounted display 4002 collects first exposure image information within a first exposure time through the 6Dof camera 407 , and obtains first spatial displacement information of the handle 4004 collected by the first inertial measurement unit 404 .

[0133] The first spatial displacement information may include inertial parameters such as angular velocity, angular acceleration, and magnetic field strength.

[0134] S504, the head mounted display 4002 collects second exposure image information within a second exposure time, obtains second spatial displacement information through the second inertial measurement unit 408, and determines second posture information of the head mounted display 4002 according to the second exposure image information and the second spatial displacement information.

[0135] S505, the head-mounted display 4002 obtains the light spot combination information of multiple infrared LEDs 405 in the first exposure image information, and then calculates the light spot combination information using the multi-view geometry principle to obtain the 6Dof posture of each infrared LED 405 in the coordinate system of the head-mounted display 4002; then, based on the above 6Dof posture and the first spatial displacement information, the first posture information of the handle 4001 is determined.

[0136] S506: Determine the virtual interactive content output by the head mounted display 4002 based on the first posture and the second posture.

[0137] Among them, the above-mentioned virtual interactive content may include light and shadow virtual visual content output by the head-mounted display, and may also include simulated touch provided by the handle; the above-mentioned simulated touch can be achieved by the linear motor on the handle.

[0138] The embodiment of the present application provides an information interaction device, which is applied to a head-mounted display of an intelligent wearable device. The intelligent wearable device also includes a handle configured with multiple light-emitting devices, and the head-mounted display is configured with an image collector, such as Figure 6As shown, the information interaction device 60 may include: a first determination module 601, a switching module 602, a second determination module 603 and an interaction module 604;

[0139] The first determination module 601 is used to collect first exposure image information based on the initial exposure mode and obtain first spatial displacement information of the head mounted display; determine the head mounted display posture of the head mounted display according to the first exposure image information and the first spatial displacement information; wherein the multiple light emitting devices maintain initial brightness in the initial exposure mode;

[0140] The switching module 602 is used to determine the exposure signal of the image collector and send an exposure signal to the handle to control the multiple light-emitting devices of the handle to switch to the target brightness within the target exposure time; wherein the target brightness is greater than the initial brightness; and the exposure time of the initial exposure mode is greater than the target exposure time;

[0141] The second determination module 603 is used to collect second exposure image information based on the target exposure mode within the target exposure time, and obtain second spatial displacement information of the handle; based on the second exposure image information and the second spatial displacement information, determine the handle posture of the handle;

[0142] The interaction module 604 is used to determine the virtual interactive content output by the head mounted display based on the head mounted display posture and the handle posture.

[0143] In an embodiment of the present application, a possible implementation is provided. When determining the exposure signal of the image collector, the switching module 602 is used to:

[0144] The exposure signal of the image collector is determined at every preset time interval; wherein the preset time interval is determined based on the frame rate of the target exposure mode.

[0145] In an embodiment of the present application, a possible implementation is provided. When determining the exposure signal of the image collector, the switching module 602 is used to:

[0146] An exposure signal is determined based on the exposure state of the image collector; wherein the sending time of the exposure signal is placed before the start time of the target exposure mode.

[0147] A possible implementation manner is provided in an embodiment of the present application. After collecting the second exposure image information based on the target exposure mode, the second determination module 603 is used to:

[0148] Storing the second exposure image information into a preset exposure image information set;

[0149] Before determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information, the second determination module 603 is used to:

[0150] From the exposure image information set, second exposure image information is extracted.

[0151] A possible implementation method is provided in an embodiment of the present application, where the second exposure image information is assigned an exposure tag;

[0152] When the second determination module 603 extracts the second exposure image information from the exposure image information set, it is used to:

[0153] According to the exposure tag, the second exposure image information is obtained by screening from the exposure image information set.

[0154] A possible implementation manner is provided in an embodiment of the present application. When the second determination module 603 extracts the second exposure image information from the exposure image information set, it is used to:

[0155] For each exposure image information in the exposure image information set, sampling is performed to obtain a plurality of pixel points of each exposure image information;

[0156] Determine the brightness of each exposure image information based on the grayscale values ​​of the plurality of pixels;

[0157] When the brightness is less than a preset brightness threshold, the exposure image information corresponding to the brightness is determined as the second exposure image information.

[0158] In an embodiment of the present application, a possible implementation manner is provided, in which the second determination module 603 is used to:

[0159] Performing posture prediction according to the second exposure image information and the second spatial displacement information to obtain initial posture information;

[0160] Acquire first spatial displacement information of the head mounted display;

[0161] The initial displacement information is corrected based on the first spatial displacement information to obtain the handle posture of the handle.

[0162] A possible implementation method is provided in an embodiment of the present application. When the second determination module 603 performs posture prediction according to the second exposure image information and the second spatial displacement information to obtain the initial posture information, it is used to:

[0163] Determining relative position information of at least two light emitting devices in the second exposure image information;

[0164] Determining serial numbers of at least two light emitting devices based on the relative position information;

[0165] Position prediction is performed according to the serial numbers of at least two light-emitting devices and the second spatial displacement information to obtain initial position information.

[0166] The device of the embodiments of the present application can execute the method provided by the embodiments of the present application, and the implementation principles are similar. The actions performed by each module in the device of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For the detailed functional description of each module of the device, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0167] The beneficial effects brought about by the technical solution provided by the device of the embodiment of the present application are the same as the beneficial effects brought about by the corresponding method shown in the previous text, and will not be repeated here.

[0168] In an embodiment of the present application, a head-mounted display is provided, including a memory, a processor, and a computer program stored in the memory, and the processor executes the above-mentioned computer program to implement the steps of the information interaction method. The beneficial effects brought by the technical solution provided by the head-mounted display in the embodiment of the present application are the same as the beneficial effects brought by the corresponding method shown in the above text, and will not be repeated here.

[0169] A wearable device is provided in an embodiment of the present application, including a handle and the head-mounted display mentioned above in the present application, wherein the handle is used to cooperate with the head-mounted display to execute the steps of the information interaction method of the present application.

[0170] In an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the information interaction method in the embodiment of the present application are implemented.

[0171] A computer program product is provided in an embodiment of the present application, including a computer program, which, when executed by a processor, implements the steps of the information interaction method in the embodiment of the present application.

[0172] In the above description, the technical details such as the patterning of each layer are not described in detail. However, those skilled in the art should understand that various technical means can be used to form layers, regions, etc. of desired shapes. In addition, in order to form the same structure, those skilled in the art can also design methods that are not completely the same as the methods described above. In addition, although the various embodiments are described above separately, this does not mean that the measures in the various embodiments cannot be used in combination to advantage.

[0173] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0174] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An information interaction method is applied to a head-mounted display of an intelligent wearable device, wherein the intelligent wearable device further comprises a handle configured with a plurality of light-emitting devices, and the head-mounted display is configured with an image collector. It is characterized in that The method comprises: Based on the initial exposure mode, first exposure image information is collected, and first spatial displacement information of the head mounted display is obtained; according to the first exposure image information and the first spatial displacement information, a head mounted display posture of the head mounted display is determined; wherein the plurality of light emitting devices maintain initial brightness in the initial exposure mode; When the exposure signal of the image collector is determined, an exposure signal is sent to the handle to control the multiple light-emitting devices of the handle to switch to a target brightness within a target exposure duration; wherein the target brightness is greater than the initial brightness; and the exposure duration of the initial exposure mode is greater than the target exposure duration; Within the target exposure time, collecting second exposure image information based on the target exposure mode, and obtaining second spatial displacement information of the handle; determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information; Based on the head mounted display posture and the handle posture, virtual interactive content output by the head mounted display is determined.

2. The method according to claim 1, It is characterized in that The step of determining the exposure signal of the image collector comprises: The exposure signal of the image collector is determined at every preset time interval; wherein the preset time interval is determined based on the frame rate of the target exposure mode.

3. The method according to claim 1, It is characterized in that The step of determining the exposure signal of the image collector comprises: An exposure signal is determined based on the exposure state of the image collector; wherein the sending time of the exposure signal is placed before the start time of the target exposure mode.

4. The method according to claim 1, It is characterized in that After acquiring the second exposure image information based on the target exposure mode, the method includes: storing the second exposure image information into a preset exposure image information set; Before determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information, the method further includes: The second exposure image information is extracted from the exposure image information set.

5. The method according to claim 4, It is characterized in that The second exposure image information is assigned an exposure label; The extracting the second exposure image information from the exposure image information set includes: The second exposure image information is obtained by screening from the exposure image information set according to the exposure label.

6. The method according to claim 4, It is characterized in that The extracting the second exposure image information from the exposure image information set includes: For each exposure image information in the exposure image information set, sampling to obtain a plurality of pixel points of each exposure image information; Determining the brightness of each of the exposure image information based on the grayscale values ​​of the plurality of pixels; When the brightness is less than a preset brightness threshold, the exposure image information corresponding to the brightness is determined as the second exposure image information.

7. The method according to claim 1, It is characterized in that The determining the handle posture of the handle based on the second exposure image information and the second spatial displacement information includes: Performing posture prediction according to the second exposure image information and the second spatial displacement information to obtain initial posture information; Acquiring first spatial displacement information of the head mounted display; The initial displacement information is corrected based on the first spatial displacement information to obtain the handle posture.

8. The method according to claim 7, It is characterized in that The performing posture prediction according to the second exposure image information and the second spatial displacement information to obtain initial posture information includes: Determining relative position information of at least two light-emitting devices in the second exposure image information; Determining serial numbers of the at least two light-emitting devices based on the relative position information; Perform posture prediction according to the serial numbers of the at least two light-emitting devices and the second spatial displacement information to obtain initial posture information.

9. An information interaction device, applied to a head-mounted display of an intelligent wearable device, wherein the intelligent wearable device further comprises a handle equipped with a plurality of light-emitting devices, and the head-mounted display is equipped with an image collector. It is characterized in that The device comprises: A first determination module is configured to collect first exposure image information based on an initial exposure mode, and obtain first spatial displacement information of the head mounted display; determine a head mounted display posture of the head mounted display according to the first exposure image information and the first spatial displacement information; wherein the plurality of light emitting devices maintain an initial brightness in the initial exposure mode; A switching module, configured to, when determining an exposure signal of the image collector, send an exposure signal to the handle to control the multiple light-emitting devices of the handle to switch to a target brightness within a target exposure duration; wherein the target brightness is greater than the initial brightness; and the exposure duration of the initial exposure mode is greater than the target exposure duration; a second determination module, configured to collect second exposure image information based on a target exposure mode within the target exposure time, and obtain second spatial displacement information of the handle; and determine a handle posture of the handle based on the second exposure image information and the second spatial displacement information; An interaction module is used to determine the virtual interactive content output by the head mounted display based on the head mounted display posture and the handle posture.

10. A head mounted display, It is characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

11. A wearable device comprising a handle and the head mounted display according to claim 10, in, The handle is used to cooperate with the head-mounted display to execute the method described in claims 1-8.

12. A storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

13. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images