Interaction mode switching method, head-mounted display device, and computer-readable storage medium

Abstract

The application discloses an interactive mode switching method, a head-mounted display device and a computer readable storage medium. The interactive mode switching method comprises the following steps: acquiring sound change information of a handle; determining whether the use state of the handle is changed according to the sound change information; if yes, acquiring a holding part posture signal and a holding part optical signal of the handle; and switching the current interactive mode of the head-mounted display device according to the holding part posture signal and the holding part optical signal. The application realizes automatic switching of the hand interactive mode, and avoids the phenomenon that the user has to operate complicatedly when switching the interactive mode of the head-mounted display device.

Description

Technical Field

[0001] This application relates to the field of head-mounted display technology, and more particularly to an interaction mode switching method, a head-mounted display device, and a computer-readable storage medium. Background Technology

[0002] VR (Virtual Reality) hand interaction modes are mainly divided into controller interaction mode and bare-hand interaction mode. The controller device detects hand movements, while the bare-hand gesture interaction, independent haptic controllers, haptic gloves, and myoelectric bracelets analyze hand movements. Since it is difficult to implement the bare-hand interaction mode alone, VR hand interaction modes are usually used in combination with controller interaction mode. Therefore, users need to switch the interaction mode according to the actual situation. For example, when a user pulls the trigger in a shooting game, they need to switch to controller interaction mode.

[0003] The current method of switching interaction modes is for users to select the desired mode by clicking on the UI (User Interface). However, according to user data, the gamepad interaction mode is used more frequently than the bare-handed interaction mode. Users need to manually switch modes frequently, which is cumbersome and affects the user experience. Summary of the Invention

[0004] The main purpose of this application is to provide a method for switching hand interaction modes, a smart controller, and a computer storage medium, aiming to solve the technical problem of cumbersome operation when users switch hand interaction modes.

[0005] To achieve the above objectives, this application provides an interaction mode switching method, which is applied to a head-mounted display device. The interaction modes of the head-mounted display device include a switchable non-handheld interaction mode and a handheld interaction mode. The method includes: Acquire sound change information of the handle, wherein the sound change information is collected by a sound sensor located at the grip part of the handle; Based on the sound change information, determine whether the usage status of the handle has changed; If so, the grip position posture signal and grip optical signal of the handle are acquired, wherein the grip position posture signal is acquired by an inertial sensor located at the grip position of the handle, and the grip optical signal is acquired by an optical sensor located at the grip position of the handle. The current interaction mode of the head-mounted display device is switched based on the posture signal of the gripping part and the optical signal of the gripping part.

[0006] Optionally, the step of determining whether the usage status of the handle has changed based on the sound change information includes: Based on the sound change information, determine the sound fluctuation amplitude and sound fluctuation phase characteristics of the handle within a first preset duration; Based on the amplitude and phase characteristics of the sound fluctuations, determine whether the usage status of the handle has changed.

[0007] Optionally, the step of determining whether the usage state of the handle has changed based on the sound wave amplitude and the sound wave phase characteristics includes: Obtain a pre-calibrated acoustic phase feature template of the vibration state of the gripping part; If the amplitude of the sound fluctuation is greater than or equal to the first preset amplitude value, and the matching degree between the sound fluctuation phase feature and the sound phase feature template is greater than or equal to the preset ratio threshold, then it is determined that the usage state of the handle has changed. If the amplitude of the sound fluctuation is less than the first preset amplitude value, and / or the matching degree between the sound fluctuation phase feature and the sound phase feature template is less than the preset ratio threshold, then it is determined that the usage state of the handle has not changed.

[0008] Optionally, the step of switching the current interaction mode of the head-mounted display device based on the posture signal of the gripping part and the optical signal of the gripping part includes: Based on the posture signal of the gripping part, determine the position and posture change information of the handle within a second preset time period, wherein the position and posture change information includes angular rotation value and displacement value; Based on the optical signal of the grip portion, the light intensity fluctuation information of the handle within a third preset time period is determined, wherein the light intensity fluctuation information includes the light intensity fluctuation direction and the light intensity fluctuation amplitude value; The handle status change information is determined based on the angle rotation value, the displacement value, the light intensity fluctuation direction, and the light intensity fluctuation amplitude value. Based on the handle status change information, the current interaction mode of the head-mounted display device is switched.

[0009] Optionally, the step of determining the handle state change information based on the angle rotation value, the displacement value, the light intensity fluctuation direction, and the light intensity fluctuation amplitude value includes: If the angle rotation value is greater than or equal to a preset angle threshold, the displacement value is greater than or equal to a preset displacement threshold, the light intensity fluctuation direction is downward, and the light intensity fluctuation amplitude value is greater than a first preset amplitude value, then the handle state change information is determined to be the first change information, wherein the first change information is the change information of the handle state changing from the placed state to the picked-up state; If the angle rotation value is less than a preset angle threshold, the displacement value is less than a preset displacement threshold, the light intensity fluctuation direction is upward, and the light intensity fluctuation amplitude value is greater than a second preset amplitude value, the handle state change information is determined to be the second change information, wherein the second change information is the change information of the handle state from the picked-up state to the placed state.

[0010] Optionally, the step of switching the current interaction mode of the head-mounted display device based on the handle state change information includes: If the controller status change information is the first change information, then the current interaction mode of the head-mounted display device is switched from the controller interaction mode to the non-controller interaction mode; If the controller status change information is the second change information, then the current interaction mode of the head-mounted display device is switched from the non-controller interaction mode to the controller interaction mode.

[0011] Optionally, before the step of switching the current interaction mode of the head-mounted display device from the controller interaction mode to the non-controller interaction mode, the method further includes: Disconnect the communication link established between the handle and the head-mounted display device; Prior to the step of switching the current interaction mode of the head-mounted display device from the non-handheld interaction mode to the handheld interaction mode, the method further includes: Establish a communication link between the handle and the head-mounted display device.

[0012] Optionally, the non-handle interaction mode is a bare-hand tracking interaction mode, an eye-tracking interaction mode, or a head-tracking interaction mode.

[0013] This application also provides a head-mounted display device, which is a physical device. The head-mounted display device includes: a memory, a processor, and a program for the interaction mode switching method stored in the memory and executable on the processor. When the program for the interaction mode switching method is executed by the processor, it can implement the steps of the interaction mode switching method as described above.

[0014] This application also provides a computer-readable storage medium storing a program implementing an interactive mode switching method, wherein the program implementing the interactive mode switching method is executed by a processor to implement the steps of the interactive mode switching method as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the interaction mode switching method described above.

[0016] This application provides an interaction mode switching method, a head-mounted display device, and a computer-readable storage medium. The method involves acquiring sound change information from a handle, wherein this sound change information is collected by a sound sensor located at the grip portion of the handle. Based on this sound change information, it is determined whether the handle's usage state has changed. If so, the method acquires the grip portion posture signal and grip portion optical signal, wherein the grip portion posture signal is collected by an inertial sensor located at the grip portion of the handle, and the grip portion optical signal is collected by an optical sensor located at the grip portion of the handle. Based on the posture signal and optical signal of the gripping part, the current interaction mode of the head-mounted display device is quickly switched, thereby realizing automatic switching of hand interaction mode and avoiding the cumbersome operation when users switch the interaction mode of the head-mounted display device. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in this embodiment or the prior art, the accompanying drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the first embodiment of the interaction mode switching method of this application; Figure 2 This is an overall schematic diagram of the smart handle of this application; Figure 3 This is a schematic diagram of the control module of the smart handle of this application; Figure 4 This is a schematic diagram of the module structure of the interaction mode switching device according to an embodiment of this application; Figure 5 This is a schematic diagram of the hardware operating environment involved in the head-mounted display device in this embodiment.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1 Please refer to Figure 1 This embodiment provides an interaction mode switching method, which is applied to a head-mounted display device. The interaction modes of the head-mounted display device include a switchable non-handheld interaction mode and a handheld interaction mode. The method includes: Step S10: Obtain sound change information of the handle, wherein the sound change information is collected by a sound sensor located at the grip part of the handle; The execution subject of the interaction mode switching method in this application embodiment is a head-mounted display device, which includes, but is not limited to, Mixed Reality (MR) devices (such as MR glasses or MR helmets), Augmented Reality (AR) devices (such as AR glasses or AR helmets), Virtual Reality (VR) devices (such as VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof.

[0023] In this embodiment, when the interaction mode is a controller interaction mode, the controller establishes a connection with the head-mounted display device, and the user manipulates the controller to detect hand movements. When the interaction mode is a non-controller interaction mode, the controller disconnects from the head-mounted device and switches to a non-controller interaction mode other than the controller interaction mode. For example, the non-controller interaction mode can be a bare-hand tracking interaction mode, an eye-tracking interaction mode, or a head-tracking interaction mode. In the bare-hand tracking interaction mode, the head-mounted device can analyze hand movements or gestures through a camera or photograph to input user operation commands corresponding to the hand movements or gestures. In the eye-tracking interaction mode, a camera can track the user's gaze direction or eye movements, and user operation commands can be input based on this gaze direction or eye movements. In the head-tracking interaction mode, an IMU (Inertial Measurement Unit) sensor can detect head posture and movements, and user operation commands can be input based on this head posture and movements. In this embodiment, the bare-hand tracking interaction mode, eye-tracking interaction mode, or head-tracking interaction mode have been extensively studied by those skilled in the art, and the sensors will not be described in detail here.

[0024] Step S20: Determine whether the usage status of the handle has changed based on the sound change information; Step S30: If yes, then acquire the grip position posture signal and grip optical signal of the handle, wherein the grip position posture signal is acquired by an inertial sensor located at the grip position of the handle, and the grip optical signal is acquired by an optical sensor located at the grip position of the handle. As will be known to those skilled in the art, the inertial sensor in this embodiment may include a six-axis accelerometer and gyroscope sensor.

[0025] Step S40: Switch the current interaction mode of the head-mounted display device according to the posture signal of the gripping part and the optical signal of the gripping part.

[0026] Because the sound of the handle's grip shell changes when the handle is held or released, the sound sensor detects an increase in the volume of the handle's grip shell, thus determining that the handle's usage state has changed. The next step is to simultaneously detect the handle's grip posture and light intensity. When the handle is held and used, the ambient light brightness decreases, and the handle will undergo a certain posture change after a preset time. Therefore, when the optical sensor detects a decrease in light intensity, and the handle's posture information (including rotation angle and / or position) changes within a preset time, the change in the handle's usage state is determined to be a change from a placed state to a picked-up state. Conversely, when the handle is released, the ambient light brightness increases, and the handle will not undergo a posture change within a preset time. Therefore, when the optical sensor detects an increase in light intensity, and the handle's posture information (including rotation angle and / or position) does not change within a preset time (or the posture change is less than a preset posture change value), the change in the handle's usage state is determined to be a change from a picked-up state to a placed state.

[0027] Reference Figure 2 An acoustic microphone (i.e., the aforementioned sound sensor), an inertial sensor, and an optical sensor are fixed to the grip portion of the handle. The optical sensor collects light intensity data (i.e., optical signals of the grip portion) of the outer shell in real time. The inertial sensor collects the pose data (including the current position and angle in real space) of the outer shell of the grip portion in real time. The sound sensor collects the volume data of the outer shell of the grip portion in real time. The handle collects light intensity data, pose data, and volume data every preset time interval. In this way, the pose change information, light intensity fluctuation information, and sound change information of the grip portion of the handle within the preset time interval are obtained. This allows the head-mounted display device to accurately determine whether the handle's state change information is a change from a picked-up state to a placed state or a change from a placed state to a picked-up state based on the pose change information, light intensity fluctuation information, and sound change information. This enables the head-mounted display device to accurately switch the current interaction mode of the head-mounted display device to the interaction mode that the user intends to use, thereby improving the user's immersive experience. The user only needs to focus on their own experience without being interrupted by frequent switching of interaction modes, avoiding the cumbersome operation when switching the interaction mode of the head-mounted display device.

[0028] like Figure 3As shown, in this embodiment, the main control chip, optical sensor, inertial sensor, and acoustic microphone (i.e., the aforementioned sound sensor) are fixed to the handle grip by heat fusion. The main control chip in the handle processes the environmental data of the handle grip detected by the optical sensor, inertial sensor, and acoustic microphone (specifically including sound change information, grip posture signal, and grip optical signal) to determine the handle state change information (i.e., handle usage state change information). This can be understood as the user picking up or placing the handle causing changes in the aforementioned environmental data, such as the volume, light intensity, and posture of the handle grip.

[0029] In this embodiment, the interaction mode switching of the head-mounted display device is initiated by the handle actively sending sound change information, grip posture signals, and grip optical signals to the head-mounted display device. Upon receiving these signals, the head-mounted display device first determines whether the handle's usage state has changed based on the received sound change information. If so, the head-mounted display device switches its current interaction mode based on the received grip posture signals and optical signals. For example, if the grip posture signals and optical signals indicate a change in handle state from a placed state to a picked-up state, the current interaction mode of the head-mounted display device is switched from a non-handle interaction mode to a handle interaction mode. Conversely, if the grip posture signals and optical signals indicate a change in handle state from a picked-up state to a placed state, the current interaction mode of the head-mounted display device is switched from a handle interaction mode to a non-handle interaction mode.

[0030] This application provides an interaction mode switching method, a head-mounted display device, and a computer-readable storage medium. By acquiring sound change information from a handle, which is collected by a sound sensor located at the grip portion of the handle, the method determines whether the handle's usage state has changed. If so, it acquires the grip portion posture signal and grip portion optical signal. The grip portion posture signal is collected by an inertial sensor located at the grip portion of the handle, and the grip portion optical signal is collected by an optical sensor located at the grip portion of the handle. Based on these grip portion posture and optical signals, the current interaction mode of the head-mounted display device is quickly switched, thereby achieving automatic switching of hand interaction modes and avoiding the cumbersome operation required for users to switch interaction modes of the head-mounted display device.

[0031] In one implementable manner, the step of determining whether the usage state of the handle has changed based on the sound change information includes: Step A10: Based on the sound change information, determine the sound fluctuation amplitude and sound fluctuation phase characteristics of the handle within a first preset duration; Step A20: Determine whether the usage status of the handle has changed based on the amplitude and phase characteristics of the sound fluctuation.

[0032] In this embodiment, since the sound wave phase characteristics of the vibration of the grip shell caused by the handle being placed or picked up (i.e., when the handle's usage state changes) conform to a certain pattern, a sound phase feature template can be obtained by pre-calibrating the sound wave phase characteristics of the handle's usage state change. By comparing the real-time detected sound wave phase characteristics with the sound phase feature template, it can be basically determined whether the handle's usage state has changed. In addition, since the handle's grip shell will vibrate with a certain amplitude when the handle's usage state changes, the magnitude of the sound wave amplitude can be used to further assist in determining whether the handle's usage state has changed.

[0033] Specifically, the step of determining whether the usage state of the handle has changed based on the sound wave amplitude and the sound wave phase characteristics includes: Step B10: Obtain a pre-calibrated acoustic phase feature template for the vibration state of the gripping part; It is understandable that by repeatedly detecting the sound wave phase characteristics of the grip when the handle is placed or picked up (i.e., when the use state of the handle changes), multiple sound wave phase characteristic samples corresponding to the grip when the use state of the handle changes can be obtained, and the sound phase characteristic template can be pre-calibrated based on each sound wave phase characteristic sample.

[0034] Step B20: If the amplitude of the sound fluctuation is greater than or equal to the first preset amplitude value, and the matching degree between the sound fluctuation phase feature and the sound phase feature template is greater than or equal to the preset ratio threshold, then it is determined that the usage state of the handle has changed. Step B30: If the amplitude of the sound fluctuation is less than the first preset amplitude value, and / or the matching degree between the sound fluctuation phase feature and the sound phase feature template is less than the preset ratio threshold, then it is determined that the usage state of the handle has not changed.

[0035] In this embodiment, the first preset amplitude value is not specifically limited, but is based on the amplitude of sound fluctuation caused by the vibration of the grip caused by the change in the use state of the handle.

[0036] This embodiment does not impose specific limitations on the sound phase feature template, but aims to more realistically and accurately reflect the sound phase features generated by the vibration of the grip caused by changes in the handle's usage state.

[0037] This embodiment utilizes the principle that the sound wave phase characteristics that cause the grip shell to vibrate when the handle's usage state changes conform to a certain pattern. It pre-calibrates the vibration phase characteristic pattern (i.e., sound phase characteristic template) that causes the grip to vibrate when the handle's usage state changes. If the sound wave amplitude is greater than or equal to a first preset amplitude value, and the matching degree between the sound wave phase characteristic and the sound phase characteristic template is greater than or equal to a preset ratio threshold, then it is determined that the handle's usage state has changed. Conversely, if the sound wave amplitude is less than the first preset amplitude value, and / or the matching degree between the sound wave phase characteristic and the sound phase characteristic template is less than the preset ratio threshold, then it is determined that the handle's usage state has not changed. Therefore, compared to judging whether the hand's usage state has changed solely based on the sound wave phase characteristic or sound wave amplitude, this embodiment improves the accuracy of determining whether the handle's usage state has changed.

[0038] Example 2 Based on the first embodiment of this application, in another embodiment of this application, the same or similar content as in the first embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, the step of switching the current interaction mode of the head-mounted display device according to the posture signal of the gripping part and the optical signal of the gripping part includes: Step C10: Based on the posture signal of the gripping part, determine the position and posture change information of the handle within a second preset time period, wherein the position and posture change information includes angle rotation value and displacement value; Step C20: Based on the optical signal of the grip, determine the light intensity fluctuation information of the handle within a third preset time period, wherein the light intensity fluctuation information includes the light intensity fluctuation direction and the light intensity fluctuation amplitude value. Step C30: Determine the handle status change information based on the angle rotation value, the displacement value, the light intensity fluctuation direction, and the light intensity fluctuation amplitude value; Step C40: Switch the current interaction mode of the head-mounted display device according to the handle status change information.

[0039] Since the change in volume of the handle grip shell is determined by sound change information, the user may have picked up or put down the handle at this time. However, the use status of the handle cannot be determined solely by sound change information. Therefore, it is necessary to determine the changes in light intensity and position of the handle grip shell by using light intensity fluctuation information and position change information respectively, and then combine the light intensity fluctuation information and position change information to determine the handle state change information.

[0040] In this embodiment, the posture signal of the gripping part can be posture data collected at two time points with a preset time interval, used to characterize the posture changes (including angle rotation values ​​and displacement values) within the preset time interval. Similarly, the optical fluctuation information can be light intensity data collected at two time points with a preset time interval, used to characterize the light intensity fluctuation within the preset time interval, wherein the preset time interval can be 1 millisecond, 2 milliseconds, or 3 milliseconds, etc.

[0041] It is understandable that determining the change in the controller's state is based on the detection of environmental data of the controller's grip area by sensors (including optical sensors, sound sensors, and inertial sensors). This can be understood as the user picking up or placing the controller causing changes in the aforementioned environmental data, such as sound, light intensity, and posture.

[0042] In this embodiment, the light intensity fluctuation direction is used to characterize the light intensity fluctuation trend within a third preset time period, and the light intensity fluctuation amplitude value is used to characterize the change in light intensity data within the third preset time period. For example, the light intensity fluctuation information includes light intensity data at two time points with a preset time interval, wherein the difference between the larger and smaller light intensity data is the light intensity fluctuation amplitude value. If the light intensity data at the later time point is greater than the light intensity data at the earlier time point, then the light intensity fluctuation direction is upward; conversely, if the light intensity data at the later time point is less than the light intensity data at the earlier time point, then the light intensity fluctuation direction is downward.

[0043] For example, if the time interval between the previous time point p and the next time point q is a preset time, and the light intensity data collected by the main control chip at time point p is 1000 candela, and the light intensity data collected at time point q is 900 candela, then the light intensity fluctuation amplitude is 100 candela, and the light intensity fluctuation direction is downward.

[0044] For example, if the time interval between the previous time point p and the next time point q is a preset time, and the light intensity data collected by the main control chip at time point p is 900 candela, and the light intensity data collected at time point q is 1000 candela, then the light intensity fluctuation amplitude is 100 candela, and the light intensity fluctuation direction is upward.

[0045] When the controller is held and used, the ambient light intensity decreases, and the controller will undergo a certain pose change after a preset time. Therefore, when the optical sensor detects a decrease in light intensity and the controller's pose information (including rotation angle and / or position) changes within a certain time, the controller's usage state change information is determined to be a change from a placed state to a picked-up state. Conversely, when the controller is released, the ambient light intensity increases, and the controller will not undergo a pose change within a preset time. Therefore, when the optical sensor detects an increase in light intensity and the controller's pose information (including rotation angle and / or position) does not change within a preset time (or the pose change is less than a preset pose change value), the controller's usage state change information is determined to be a change from a picked-up state to a placed state.

[0046] In one possible implementation, the step of determining the handle state change information based on the angle rotation value, the displacement value, the light intensity fluctuation direction, and the light intensity fluctuation amplitude value includes: Step D10: If the angle rotation value is greater than or equal to a preset angle threshold, the displacement value is greater than or equal to a preset displacement threshold, the light intensity fluctuation direction is downward, and the light intensity fluctuation amplitude value is greater than a first preset amplitude value, then the handle state change information is determined to be the first change information, wherein the first change information is the change information of the handle state changing from the placed state to the picked-up state; Step D20: If the angle rotation value is less than a preset angle threshold, the displacement value is less than a preset displacement threshold, the light intensity fluctuation direction is upward, and the light intensity fluctuation amplitude value is greater than a second preset amplitude value, the handle state change information is determined to be the second change information, wherein the second change information is the change information of the handle state from the picked-up state to the placed state.

[0047] In this embodiment, the first preset amplitude value is the threshold for judging that the light intensity data has fluctuated significantly downward. Those skilled in the art can set it themselves. This embodiment does not specifically limit it. The standard is to detect that the light intensity data of the handle grip has fluctuated significantly downward. If the light intensity fluctuation amplitude value is greater than the first preset amplitude value, it is determined that the light intensity of the handle grip has decreased significantly. Conversely, it is determined that the light intensity data has decreased slightly.

[0048] When the hand is in contact with the handle grip, the light received by the handle grip decreases and the light intensity decreases significantly. Therefore, if the light intensity fluctuation direction is downward and the light intensity fluctuation amplitude is greater than the first preset amplitude value, it can be basically determined that the handle state change information is the first change information.

[0049] In this embodiment, the second preset amplitude value is the threshold for judging that the light intensity data has fluctuated significantly upward. Those skilled in the art can set it themselves. This embodiment does not limit it in detail. The standard is to detect that the light intensity data of the handle grip has fluctuated significantly upward. If the light intensity fluctuation amplitude value is greater than the second preset amplitude value, it is determined that the light intensity of the handle grip has increased significantly. Conversely, it is determined that the light intensity data has increased slightly.

[0050] When the hand releases the grip of the handle, the grip receives more light, and the light intensity increases significantly. Therefore, if the direction of the light intensity fluctuation is upward and the amplitude of the light intensity fluctuation is greater than the second preset amplitude value, it can be basically determined that the handle state change information is the second change information.

[0051] In this embodiment, the values ​​of the first preset amplitude value and the second preset amplitude value may be the same or different, and this embodiment does not impose specific limitations on this.

[0052] In this embodiment, the preset angle threshold is the threshold for determining that the current angle of the handle has undergone significant rotation. Those skilled in the art can set it themselves, but this embodiment does not specifically limit it, and the standard is to detect that the current angle of the handle has undergone significant rotation.

[0053] In this embodiment, the preset displacement threshold is the threshold for determining that the position of the handle has moved significantly. Those skilled in the art can set it themselves, but this embodiment does not specifically limit it, and the standard is to detect that the position of the handle has moved significantly.

[0054] When a user picks up and uses the controller, the controller will undergo certain positional changes over a certain period of time. Therefore, the controller's state change information can be determined based on the positional change information (including angle rotation and displacement). If the angle rotation value is greater than or equal to a preset angle threshold, and the displacement value is greater than or equal to a preset displacement threshold, then the controller's state change information can be basically determined to be the first change information. If the angle rotation value is less than the preset angle threshold, and the displacement value is less than the preset displacement threshold, then the controller's state change information can be basically determined to be the second change information.

[0055] In this embodiment, if the angle rotation value is greater than or equal to a preset angle threshold, the displacement value is greater than or equal to a preset displacement threshold, the light intensity fluctuation direction is downward, and the light intensity fluctuation amplitude value is greater than a first preset amplitude value, then the handle state change information can be accurately determined as the first change information, wherein the first change information is the change information of the handle state from the placed state to the picked-up state. Conversely, if the angle rotation value is less than a preset angle threshold, the displacement value is less than a preset displacement threshold, the light intensity fluctuation direction is upward, and the light intensity fluctuation amplitude value is greater than a second preset amplitude value, then the handle state change information can be accurately determined as the second change information, wherein the second change information is the change information of the handle state from the picked-up state to the placed state.

[0056] This embodiment determines the controller state change information by combining the changes in light intensity and posture at the outer shell of the controller grip, thereby providing a more multi-dimensional basis for judging the switching of controller interaction modes, reducing the judgment error caused by only detecting changes in light intensity or posture, and thus improving the accuracy of recognizing controller state change information.

[0057] In one implementable manner, the step of switching the current interaction mode of the head-mounted display device based on the handle state change information includes: Step E10: If the controller state change information is the first change information, then switch the current interaction mode of the head-mounted display device from the controller interaction mode to the non-controller interaction mode. Step E20: If the controller status change information is the second change information, then switch the current interaction mode of the head-mounted display device from the non-controller interaction mode to the controller interaction mode.

[0058] For example, prior to the step of switching the current interaction mode of the head-mounted display device from the controller interaction mode to the non-controller interaction mode, the method further includes: Step E30: Disconnect the communication link established between the handle and the head-mounted display device; Prior to the step of switching the current interaction mode of the head-mounted display device from the non-handheld interaction mode to the handheld interaction mode, the method further includes: Step E40: Establish a communication link between the handle and the head-mounted display device.

[0059] In this embodiment, when the head-mounted display device determines that the controller state change information is the first change information, it switches the current interaction mode of the head-mounted display device from the controller interaction mode to the non-controller interaction mode and disconnects from the controller. It is easy to understand that if the current interaction mode of the head-mounted display device is already the non-controller interaction mode, then there is no need to perform the switch from controller interaction mode to non-controller interaction mode.

[0060] For example, when the head-mounted display device determines that the controller state change information is the second change information, it switches the current interaction mode of the head-mounted display device from the non-controller interaction mode to the controller interaction mode and establishes a connection with the controller. It is easy to understand that if the current interaction mode of the head-mounted display device is already the controller interaction mode, then there is no need to perform the switch from the non-controller interaction mode to the controller interaction mode.

[0061] In one scenario, if the optical sensor is obstructed by other objects (not the hand), it cannot accurately determine the controller's state change information based on changes in light intensity at the grip area, leading to misjudgments. In another scenario, if the object on which the controller is placed (e.g., a table) is moving, it cannot accurately determine the controller's state change information based on changes in the grip area's posture, also resulting in misjudgments. Therefore, this embodiment determines the controller's state change information by combining changes in both light intensity and posture at the grip area's outer shell. This provides a more multi-dimensional basis for judging the switching of the controller's interaction mode, reducing the judgment error caused by detecting only changes in light intensity (or light intensity fluctuations) or posture changes, thereby improving the accuracy of identifying controller state change information.

[0062] It is worth mentioning that this embodiment uses optical sensors, sound sensors, and inertial sensors to collect light intensity fluctuation information, sound change information, and pose change information respectively. Based on the light intensity, sound, and pose, a comprehensive judgment is made, which avoids misjudgment of light intensity fluctuation information or pose change information and improves the accuracy of judging the controller state change information.

[0063] Example 3 This invention also provides an interactive mode switching device, please refer to... Figure 4 The interaction mode switching device is applied to a head-mounted display device, the interaction modes of which include a switchable non-handheld interaction mode and a handheld interaction mode, and the interaction mode switching device includes: The acquisition module 10 is used to acquire sound change information of the handle, wherein the sound change information is acquired by a sound sensor located at the grip part of the handle; The determining module 20 is used to determine whether the usage status of the handle has changed based on the sound change information; The detection module 30 is used to acquire the grip position posture signal and the grip optical signal of the handle if the condition is met. The grip position posture signal is acquired by an inertial sensor located at the grip position of the handle, and the grip optical signal is acquired by an optical sensor located at the grip position of the handle. The switching module 40 is used to switch the current interaction mode of the head-mounted display device according to the posture signal of the gripping part and the optical signal of the gripping part.

[0064] Optionally, module 20 is also used for: Based on the sound change information, determine the sound fluctuation amplitude and sound fluctuation phase characteristics of the handle within a first preset duration; Based on the amplitude and phase characteristics of the sound fluctuations, determine whether the usage status of the handle has changed.

[0065] Optionally, module 20 is also used for: Obtain a pre-calibrated acoustic phase feature template of the vibration state of the gripping part; If the amplitude of the sound fluctuation is greater than or equal to the first preset amplitude value, and the matching degree between the sound fluctuation phase feature and the sound phase feature template is greater than or equal to the preset ratio threshold, then it is determined that the usage state of the handle has changed. If the amplitude of the sound fluctuation is less than the first preset amplitude value, and / or the matching degree between the sound fluctuation phase feature and the sound phase feature template is less than the preset ratio threshold, then it is determined that the usage state of the handle has not changed.

[0066] Optionally, the switching module 40 is also used for: Based on the posture signal of the gripping part, determine the position and posture change information of the handle within a second preset time period, wherein the position and posture change information includes angular rotation value and displacement value; Based on the optical signal of the grip portion, the light intensity fluctuation information of the handle within a third preset time period is determined, wherein the light intensity fluctuation information includes the light intensity fluctuation direction and the light intensity fluctuation amplitude value; The handle status change information is determined based on the angle rotation value, the displacement value, the light intensity fluctuation direction, and the light intensity fluctuation amplitude value. Based on the handle status change information, the current interaction mode of the head-mounted display device is switched.

[0067] Optionally, the switching module 40 is also used for: If the angle rotation value is greater than or equal to a preset angle threshold, the displacement value is greater than or equal to a preset displacement threshold, the light intensity fluctuation direction is downward, and the light intensity fluctuation amplitude value is greater than a first preset amplitude value, then the handle state change information is determined to be the first change information, wherein the first change information is the change information of the handle state changing from the placed state to the picked-up state; If the angle rotation value is less than a preset angle threshold, the displacement value is less than a preset displacement threshold, the light intensity fluctuation direction is upward, and the light intensity fluctuation amplitude value is greater than a second preset amplitude value, the handle state change information is determined to be the second change information, wherein the second change information is the change information of the handle state from the picked-up state to the placed state.

[0068] Optionally, the switching module 40 is also used for: If the controller status change information is the first change information, then the current interaction mode of the head-mounted display device is switched from the controller interaction mode to the non-controller interaction mode; If the controller status change information is the second change information, then the current interaction mode of the head-mounted display device is switched from the non-controller interaction mode to the controller interaction mode.

[0069] Optionally, the switching module 40 is also used for: Before the step of switching the current interaction mode of the head-mounted display device from the controller interaction mode to the non-controller interaction mode, the communication link established between the controller and the head-mounted display device is disconnected; Before the step of switching the current interaction mode of the head-mounted display device from the non-handle interaction mode to the handle interaction mode, a communication link is established between the handle and the head-mounted display device.

[0070] Optionally, the non-handle interaction mode is a bare-hand tracking interaction mode, an eye-tracking interaction mode, or a head-tracking interaction mode.

[0071] The interaction mode switching device provided in this invention, employing the interaction mode switching method described in Embodiment 1 or Embodiment 2, can solve the technical problem of cumbersome operation when users switch hand interaction modes. Compared with the prior art, the beneficial effects of the interaction mode switching device provided in this invention are the same as those of the interaction mode switching method provided in the above embodiments, and other technical features in the interaction mode switching device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0072] Example 4 This invention provides a head-mounted display device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the interactive mode switching method in Embodiment 1 above.

[0073] The following is for reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing a head-mounted display device according to embodiments of the present disclosure. The head-mounted display device in embodiments of the present disclosure includes, but is not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof, etc. Figure 5 The head-mounted display device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0074] like Figure 5 As shown, the head-mounted display device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). The RAM 1004 also stores various programs and data required for the operation of the head-mounted display device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface is also connected to the bus 1005.

[0075] Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the head-mounted display to communicate wirelessly or wiredly with other devices to exchange data. Although head-mounted display devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0076] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0077] The head-mounted display device provided by this invention, employing the interaction mode switching method in the above embodiments, can solve the technical problem of cumbersome operation when users switch hand interaction modes. Compared with the prior art, the beneficial effects of the head-mounted display device provided by this invention are the same as those of the interaction mode switching method provided in the above embodiments, and other technical features in this head-mounted display device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0078] It should be understood that various parts of this disclosure can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0079] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0080] Example 5 This invention provides a computer-readable storage medium having computer-readable program instructions stored thereon, which are used to execute the interactive mode switching method in the above embodiments.

[0081] The computer-readable storage medium provided in this embodiment of the invention may be, for example, a USB flash drive, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0082] The aforementioned computer-readable storage medium may be included in the head-mounted display device; or it may exist independently and not assembled into the head-mounted display device.

[0083] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the head-mounted display device, cause the head-mounted display device to: acquire sound change information of the handle, wherein the sound change information is acquired by a sound sensor located at the grip portion of the handle; determine, based on the sound change information, whether the usage state of the handle has changed; if so, acquire a grip portion posture signal and a grip portion optical signal, wherein the grip portion posture signal is acquired by an inertial sensor located at the grip portion of the handle, and the grip portion optical signal is acquired by an optical sensor located at the grip portion of the handle; and switch the current interaction mode of the head-mounted display device based on the grip portion posture signal and the grip portion optical signal.

[0084] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0087] The computer-readable storage medium provided by this invention stores computer-readable program instructions for executing the above-described interaction mode switching method, which solves the technical problem of cumbersome operation when users switch hand interaction modes. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiments of this invention are the same as the beneficial effects of the interaction mode switching method provided in Embodiment 1 or Embodiment 2, and will not be repeated here.

[0088] Example 6 This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the interaction mode switching method described above.

[0089] The computer program product provided in this application solves the technical problem of cumbersome operation when users switch hand interaction modes. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiments of the present invention are the same as the beneficial effects of the interaction mode switching methods provided in Embodiment 1 or Embodiment 2 above, and will not be repeated here.

[0090] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for switching interaction modes, characterized in that, The interaction mode switching method is applied to a head-mounted display device, wherein the interaction modes of the head-mounted display device include a switchable non-handheld interaction mode and a handheld interaction mode, and the method includes: Acquire sound change information of the handle, wherein the sound change information is collected by a sound sensor located at the grip part of the handle; Based on the sound change information, determine whether the usage status of the handle has changed; If so, the grip position posture signal and grip optical signal of the handle are acquired, wherein the grip position posture signal is acquired by an inertial sensor located at the grip position of the handle, and the grip optical signal is acquired by an optical sensor located at the grip position of the handle. Based on the posture signal of the gripping part, determine the position and posture change information of the handle within a second preset time period, wherein the position and posture change information includes angular rotation value and displacement value; Based on the optical signal of the grip portion, the light intensity fluctuation information of the handle within a third preset time period is determined, wherein the light intensity fluctuation information includes the light intensity fluctuation direction and the light intensity fluctuation amplitude value; If the angle rotation value is greater than or equal to a preset angle threshold, the displacement value is greater than or equal to a preset displacement threshold, the light intensity fluctuation direction is downward, and the light intensity fluctuation amplitude value is greater than a first preset amplitude value, then the handle state change information is determined to be the first change information, wherein the first change information is the change information of the handle state changing from the placed state to the picked-up state; If the angle rotation value is less than a preset angle threshold, the displacement value is less than a preset displacement threshold, the light intensity fluctuation direction is upward, and the light intensity fluctuation amplitude value is greater than a second preset amplitude value, the handle state change information is determined to be the second change information, wherein the second change information is the change information of the handle state from the picked-up state to the placed state; If the controller status change information is the first change information, then the current interaction mode of the head-mounted display device is switched from the non-controller interaction mode to the controller interaction mode; If the controller status change information is the second change information, then the current interaction mode of the head-mounted display device is switched from the controller interaction mode to the non-controller interaction mode.

2. The interaction mode switching method as described in claim 1, characterized in that, The step of determining whether the usage status of the handle has changed based on the sound change information includes: Based on the sound change information, determine the sound fluctuation amplitude and sound fluctuation phase characteristics of the handle within a first preset duration; Based on the amplitude and phase characteristics of the sound fluctuations, determine whether the usage status of the handle has changed.

3. The interaction mode switching method as described in claim 2, characterized in that, The step of determining whether the usage state of the handle has changed based on the sound wave amplitude and the sound wave phase characteristics includes: Obtain a pre-calibrated acoustic phase feature template of the vibration state of the gripping part; If the amplitude of the sound fluctuation is greater than or equal to the first preset amplitude value, and the matching degree between the sound fluctuation phase feature and the sound phase feature template is greater than or equal to the preset ratio threshold, then it is determined that the usage state of the handle has changed. If the amplitude of the sound fluctuation is less than the first preset amplitude value, and / or the matching degree between the sound fluctuation phase feature and the sound phase feature template is less than the preset ratio threshold, then it is determined that the usage state of the handle has not changed.

4. The interaction mode switching method as described in claim 1, characterized in that, Before the step of switching the current interaction mode of the head-mounted display device from the controller interaction mode to the non-controller interaction mode, the method further includes: Disconnect the communication link established between the handle and the head-mounted display device; Prior to the step of switching the current interaction mode of the head-mounted display device from the non-handheld interaction mode to the handheld interaction mode, the method further includes: Establish a communication link between the handle and the head-mounted display device.

5. The interaction mode switching method as described in any one of claims 1 to 4, characterized in that, The non-handheld interaction mode is either a bare-hand tracking interaction mode, an eye-tracking interaction mode, or a head-tracking interaction mode.

6. A head-mounted display device, characterized in that, The head-mounted display device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the interaction mode switching method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program implementing the interaction mode switching method, which is executed by a processor to implement the steps of the interaction mode switching method as described in any one of claims 1 to 5.

Images