Virtual scene interaction method and device, head-mounted display device and medium

Abstract

The invention discloses a virtual scene interaction method, interaction equipment, head-mounted display equipment and a medium. The method comprises the following steps: receiving a touch instruction generated by the interaction equipment in response to user operation; determining a virtual scene of the head-mounted display device; and according to the touch instruction and the virtual scene, controlling a virtual element in the display interface to realize interaction between the interaction device and the head-mounted display device. Through the integration of the touch instruction and the virtual scene, the interaction between the interaction device and the head-mounted display device can be quickly realized according to the virtual scene, and the interaction efficiency is high. The invention also provides a finger clip provided with a trackball and a touch bar, so that the finger clip can interact with the head-mounted display equipment more conveniently.

Description

Technical Field

[0001] The present invention relates to the field of virtual scene technology, and in particular to a virtual scene interaction method, interaction device, head-mounted display device and medium. Background Art

[0002] VR (Virtual Reality), AR (Augmented Reality) and MR (Mixed Reality) can all generate virtual scenes. They are technologies that add virtualization technology to the user's sensory intuition to provide users with a sense of immersion. In order to improve the interactivity between users and virtual scenes, a variety of operable virtual elements (such as virtual buttons, virtual controls, virtual cursors, etc.) are usually provided to users in the display interface. Users can trigger corresponding operations and interact with virtual scenes in the display interface through these operable virtual elements provided in the operation interface.

[0003] Although current interactive devices can interact with virtual scenes, they simply use buttons on the interactive device to move the cursor in the virtual scene to select or operate virtual controls. This is slow and difficult for users to accurately select virtual controls, making it difficult to efficiently achieve interactive needs in various virtual scenes. Therefore, current interactive methods are inconvenient to operate and inefficient. Summary of the invention

[0004] The main purpose of the present invention is to provide a virtual scene interaction method, an interaction device, a head-mounted display device and a storage medium, aiming to solve the problems of inconvenient operation and low efficiency when interacting with a virtual scene.

[0005] In order to achieve the above-mentioned object, the first aspect of the present invention provides a virtual scene interaction method, which is applied to a head-mounted display device, wherein the head-mounted display device outputs a display interface, and the display interface is provided with virtual elements, and the method comprises:

[0006] Receiving a touch control instruction generated by the interactive device in response to a user operation;

[0007] Determining a virtual scene of the head mounted display device;

[0008] According to the touch control instruction and the virtual scene, the virtual elements in the display interface are controlled.

[0009] Optionally, the virtual element includes a focus and a virtual control, and the controlling the virtual element in the display interface according to the touch instruction and the virtual scene includes:

[0010] Invoke a shortcut function according to the touch command and the virtual scene to control a virtual control in the display interface, or move a focus in the display interface according to the touch command and the virtual scene.

[0011] Optionally, after the focus in the display interface is moved, the focus is also highlighted; or, an animation effect is displayed when the focus in the display interface is moved.

[0012] Optionally, the virtual element includes a virtual control and a focus, and the method includes: partitioning the virtual control according to the function corresponding to the virtual control in the display interface based on the virtual scene; and controlling the virtual element in the display interface according to the touch instruction and the virtual scene includes:

[0013] Acquire a functional partition corresponding to the virtual control where the focus is located in the display interface according to the virtual scene, and acquire a next functional partition according to the semantic information, the touch instruction and the functional partition;

[0014] Get the default focus position of the next functional partition;

[0015] The focus in the display interface is moved to the default focus position.

[0016] Optionally, the virtual element is the focus, and after the step of controlling the virtual element in the display interface according to the touch instruction and the virtual scene, the method further includes:

[0017] Obtaining viewing angle switching information of the head mounted display device;

[0018] The focus position in the display interface is updated according to the viewing angle switching information.

[0019] Optionally, the virtual element is a focus, and controlling the virtual element in the display interface according to the touch instruction and the virtual scene includes:

[0020] According to the virtual scene, a core application is obtained;

[0021] When the interactive area of ​​the core application intersects with the field of view of the head-mounted display device, a default focus is obtained according to the core application, the default focus is set as the focus in the display interface, and the focus in the display interface is controlled according to the touch instruction.

[0022] Optionally, the interactive device is provided with a motion sensor, the touch command is gesture information, and calling a shortcut function according to the touch command and the virtual scene includes:

[0023] Acquiring location information of the motion sensor;

[0024] Based on the position information, obtaining gesture information;

[0025] A shortcut function is obtained according to the gesture information and the virtual scene, and the shortcut function is executed.

[0026] A second aspect of the present invention provides an interactive device for interacting with a head-mounted display device, the interactive device being provided with a trackball, and a touch bar being provided on a side wall of the interactive device, the trackball being used to generate a first touch instruction in response to a user operation, the first touch instruction being used to move a focus in a display interface output by the head-mounted display device or to call a shortcut function; the touch bar being used to generate a second touch instruction in response to a user operation, the second touch instruction being used to drag or scale a virtual control in a display interface output by the head-mounted display device.

[0027] A third aspect of the present invention provides a head-mounted display device, which includes a memory, a processor, and a virtual scene interaction program stored in the memory and executable on the processor, wherein the virtual scene interaction program implements any one of the steps of the virtual scene interaction method when executed by the processor.

[0028] A fourth aspect of the present invention provides a computer-readable storage medium, on which a virtual scene interaction program is stored, and when the virtual scene interaction program is executed by a processor, any step of the virtual scene interaction method is implemented.

[0029] As can be seen from the above, the present invention receives the touch command generated by the interactive device in response to the user operation, determines the virtual scene of the head-mounted display device, and then combines the touch command and the virtual scene to control the virtual elements in the display interface to realize the interactive function. By combining the touch command and the virtual scene, the interaction between the interactive device and the head-mounted display device can be quickly realized according to the virtual scene, and the interaction efficiency is high. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0031] Figure 1 It is a schematic diagram of a specific flow of a virtual scene interaction method provided by an embodiment of the present invention;

[0032] Figure 2 is a schematic diagram of virtual navigation buttons and virtual keys of a virtual scene display interface according to an embodiment of the present invention;

[0033] Figure 3 is a schematic diagram of a virtual scene display interface of an embodiment of the present invention;

[0034] Figure 4 Schematic diagram of absolute sequential defocusing according to an embodiment of the present invention;

[0035] Figure 5 It is a schematic diagram of a specific flow of moving the focus of a virtual scene provided by an embodiment of the present invention;

[0036] Figure 6 is a schematic diagram of relative sequence focus shifting provided by an embodiment of the present invention;

[0037] Figure 7 The present invention provides a block diagram of the internal structure of a head mounted display device. DETAILED DESCRIPTION

[0038] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0039] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0040] It should also be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0041] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0042] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0043] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Virtual scene technology is gradually being applied to people's lives, using head-mounted display devices to generate virtual scenes, providing visual, auditory, tactile or other sensory simulation experiences, and users can interact with virtual scenes. Specifically, the display interface output by the head-mounted display device displays various virtual elements, such as cursors, virtual buttons, virtual controls, focus, etc., and users can interact with virtual scenes by clicking, selecting, dragging, zooming, and other operations on virtual elements to execute corresponding functions.

[0046] The head-mounted display device can be an AR device such as AR glasses and AR helmets, or a VR box, an all-in-one VR machine, or an MR head-mounted display device. Users can interact with the head-mounted display device through gestures or finger clips worn on their fingers. The finger clip is an interactive device used in conjunction with the head-mounted display device. However, the interactive device is not limited to the finger clip, but can also be a joystick, a glove, a bracelet, etc. Since only gestures are used for interaction, the user needs to keep shaking his arm in the suspended state, and long-term use may cause hand muscle fatigue, soreness and discomfort. Therefore, gestures and finger clips are often combined to interact with virtual scenes.

[0047] However, most of the current finger clips are button-type finger clips, which use buttons to simply move the cursor in the virtual scene to select virtual elements in the virtual scene. This is slow and difficult for users to accurately select and operate virtual elements, making the operation inconvenient and inefficient.

[0048] The present invention provides a virtual scene interaction method, which can quickly realize the interaction between the interactive device and the head-mounted display device according to the virtual scene by integrating the touch instructions of the interactive device and the virtual scene, and has high interaction efficiency.

[0049] In this embodiment, the interactive device is a finger clip, and the head-mounted display device is an AR glasses. The finger clip and the AR glasses are used to realize the interaction of the virtual scene. Figure 1 As shown, the interactive method includes the following steps:

[0050] Step S100: receiving a touch control instruction generated by an interactive device in response to a user operation;

[0051] Step S200: determining a virtual scene of a head mounted display device;

[0052] Step S300: controlling virtual elements in the display interface according to touch commands and virtual scenes.

[0053] Specifically, the finger clip is an interactive accessory of AR glasses and is used in conjunction with AR glasses. In order to improve the interactive efficiency of the finger clip, this embodiment also improves the finger clip by installing a trackball and a touch bar on the finger clip to replace the original inconvenient buttons. When the user operates the trackball or touches the touch bar, the trackball and the touch bar will generate response information, such as the angle of the trackball offset to the right, the length of the finger on the touch bar, etc. The finger clip converts the response information into a touch command and sends it to the AR glasses. The touch command corresponds to the response information, and the form can be set as needed.

[0054] In order to achieve the deep integration of the operation status of the finger clip and the virtual scene, a concept of virtual scene is also proposed. The factors for classifying virtual scenes mainly include: the current display interface, the viewing angle of AR glasses, the functions running in the background, etc. For example: when the video is displayed on the current display interface, the virtual scene is the video playback scene; when the video is displayed on the current display interface, when the background receives a message, the virtual scene can be a message scene; when the navigation is displayed on the current display interface, the virtual scene is a navigation scene.

[0055] AR glasses control the virtual elements in the display interface according to touch commands and virtual scenes, and realize the interaction between the finger clip and the head-mounted display device, such as: selection of virtual elements, confirmation of buttons, and menu operations. Compared with simple interaction based on finger clip commands, it can be integrated with the virtual scene to better meet the interaction needs.

[0056] In this embodiment, a trackball is provided on the finger clip to replace the up, down, left and right buttons of the existing finger clip, and provides more flexible and precise control. The movement of the cursor can be controlled by rolling the trackball with the finger, which avoids frequent hand movements, reduces hand fatigue, and can reduce the size of the finger clip.

[0057] In one example, a trackball is used to move the focus. Specifically, the focus is used to indicate the currently selected virtual control in the display interface. It can be a dotted frame, which is used to display a dotted frame on the currently selected virtual control; it can also be a background color, which is used to change the background color of the currently selected virtual control; it can also be a foreground color, which is used to change the foreground color of the currently selected virtual control, for example Figure 2 The virtual navigation button A and virtual button B in the display interface. The display interface is composed of virtual controls, including interactive controls and non-interactive controls. Interactive controls are virtual controls that can be clicked, input information or selected. Common ones include: buttons, input boxes, radio buttons, multiple selection boxes, etc. Common non-interactive controls include: text labels. When rolling the trackball, the focus is moved between interactive controls. The direction of movement is not limited and the focus can quickly cross multiple interactive controls at one time, improving operation efficiency and accuracy. The sensitivity of the trackball rolling determines the movement speed of the focus or cursor (the indicator icon that can be moved in the display interface, similar to the cursor displayed by the mouse). If the sensitivity is set high, the focus or cursor moves fast and more interactive controls are crossed at one time; if the sensitivity is set low, the focus or cursor moves slowly and fewer interactive controls are crossed at one time. Compared with using buttons to switch focus in a button-type finger clip, the operation efficiency and operation accuracy are high, which is especially suitable for some AR application scenarios that require more precise and flexible control of the cursor (such as 3D models, games, etc.). The sensitivity setting can refer to the DPI (Dots Per Inch) parameter of the mouse, such as 800DPI, 1000DPI, 1200DPI, etc., respectively, according to the movement of the trackball from slow to fast. After moving the focus, the virtual control where the new focus is located can also be highlighted to facilitate the user to identify the currently selected virtual control; animation or various visual effects can also be used to attract the user's attention when the virtual control is selected or changes from selected to unselected.

[0058] The button-type finger clip in the related art mainly interacts by moving the virtual cursor to the interactive control by pressing the button and clicking it, which not only has low interaction efficiency but also has a single interaction mode. In the embodiment of the present application, a variety of rich interactive functions can be realized after using the trackball, such as calling shortcut functions, quickly switching focus, quickly moving the cursor, selecting virtual controls, etc. In one example, operations such as clicking, double-clicking, and long pressing can be realized by pressing the trackball. Pressing the trackball once means selecting the virtual control in the display interface; pressing the trackball twice means calling out the shortcut panel, and displaying the virtual control related to the shortcut operation in the display interface; long pressing the trackball means opening the voice assistant, and displaying the virtual control related to the voice control in the display interface. For example, when the received touch command is the same as the preset double-click command in the virtual scene, the preset configuration file is read (the comparison relationship between the touch command and the shortcut function in each virtual scene is saved), the shortcut function corresponding to the double-click command is obtained and the shortcut function is called; when the received touch command is the same as the preset long press command in the virtual scene, the preset configuration file is read, the shortcut function corresponding to the long press command is obtained and the shortcut function is called.

[0059] Various sensors (such as accelerometers, gyroscopes, magnetometers and touch sensors) can also be installed in the finger clip to detect the position, posture and movement of the fingers, and transmit this information to the AR glasses, enabling interaction with the AR scene through various methods such as gesture control and touch control.

[0060] In one implementation scenario, the position of the motion sensor on the finger clip is also detected in real time to obtain position information, and the gesture information is determined based on the position information. Then, the shortcut function corresponding to the gesture information is obtained from the preset configuration file and the shortcut function is called. For example: an inertial sensor is configured in the AR glasses. According to the gyroscope and accelerometer in the inertial sensor, the direction, position and speed of the object to which the inertial sensor is attached (for example, the finger bones) can be measured. The inertial sensor is located near the finger (or thumb) bones and can measure the inertial motion of the corresponding bones, including the action and number of times the thumb and index finger touch each other, and call the corresponding shortcut function. For example: the thumb and index finger touch each other once to start the camera function, and the thumb and index finger touch each other twice to start the microphone function.

[0061] A touch bar is a device that can detect the distance a finger slides over and implement interaction based on the distance. Compared with traditional key input, a touch bar can provide richer interaction methods, such as finger sliding, dragging, zooming, etc. These operations can more naturally match human hand movements, improving the smoothness and naturalness of the interaction. Therefore, setting a touch bar in the finger clip can further expand the interactive functions, such as dragging or zooming virtual controls in the display interface.

[0062] In one example, the touch bar is in the shape of a long strip, which can be conveniently set on the side wall of the finger clip without increasing the size of the finger clip. By sliding the touch bar or sliding the touch bar after the virtual element is selected, the virtual elements in the display interface can be changed by sliding, dragging or zooming. For example: scrolling the content currently displayed in the input box in the display interface; dragging the virtual items in the display interface; scrolling the page or list, etc. Therefore, the user can interact with the virtual scene more flexibly.

[0063] A display interface of this embodiment is as follows Figure 3 As shown in the figure, Tab#1-3 is tab page 1-3, List#1-4 is list item 1-4, and Layer#1-3 is level 1-3 page. The display interface includes a dock bar at the bottom, a function area on the left, and a viewing area in the center. The dock bar includes: personal center icon, application icon, quick setting small icon; the function area includes: notification center Tab, notification list, notification details; the viewing area includes: application icon, expanded second and third level pages, quick panel, dialog box button.

[0064] When interacting, the logic of focus movement is: move the focus in a predetermined logical order, such as from left to right or from top to bottom. When the focus is a virtual element at the edge of the interface, move the focus to the next related virtual element. For example: the focus moves to Figure 3 When you are on Tab#1 in the taskbar, scroll the trackball horizontally to switch the focus to Tab#2. Continue scrolling horizontally to switch the focus to Tab#3. When the focus moves to Tab#1, scroll the trackball vertically to switch the focus to the sub-options under the Tab#1 category. This ensures the continuity and functional consistency of the focus during interaction without abruptly moving to the dock.

[0065] The implementation process of moving the focus is: first traverse all interactive virtual controls on the current interface, then sort these virtual controls according to the predetermined focus order according to the touch command, obtain a sorted list, search for the position of the virtual control corresponding to the current focus in the sorted list, and select the next virtual control forward or backward based on the position and set it as the current focus. It is understandable that different virtual scenes correspond to different virtual controls, so when moving the focus, the same touch command may have different results for moving the focus corresponding to different virtual scenes.

[0066] The above-mentioned method of moving the focus is also called absolute sequential focus, which is simple to implement, but the user may have to pass through multiple virtual elements to move from the previous focus to the next focus, and the speed of moving the focus is slow and inefficient. Figure 4As shown, the focus moves from point A to point B, and then from point C to point D in an absolute order (from left to right, from top to bottom). The focus may traverse all interactive virtual controls on the interface. This is not efficient when there are many virtual controls. In order to allow users to locate the target virtual control more quickly, the present invention optimizes the focus movement logic, matches the logical order with the functional partition according to the functional partition, so as to improve the interaction efficiency. The specific steps of moving the focus of the virtual scene are as follows: Figure 5 As shown, including:

[0067] Step S400: acquiring a functional partition of a virtual control where the focus is located in the display interface according to the virtual scene, and acquiring a next functional partition according to the semantic information, the touch command and the functional partition.

[0068] Step S500: Obtaining the default focus position of the next functional partition.

[0069] Step S600: moving the focus in the display interface to a default focus position.

[0070] First, the virtual controls are partitioned in advance according to the functions corresponding to the virtual controls in the virtual scene, and then the functional partition of the virtual control corresponding to the focus in the display interface is obtained according to the virtual scene, and the next functional partition is obtained according to the semantic information, touch instructions and functional partitions, such as Figure 6 As shown in the figure, it is divided into two functional areas, point A and point D are located in the upper functional area, and point B and point C are located in the lower functional area. After the focus is switched to point A, if the touch command is in the downward direction, the next functional area is Figure 6 The lower functional partition in the interface. Assuming that the default focus position of the next functional partition is point B, point B is set as the focus in the display interface, and then the focus movement continues. Therefore, the focus moves from point A to point B, and then from point C to point D according to the relative direction logic (moving the focus from the upper left to the bottom, quickly moving to the lower functional partition, moving the focus from the lower right to the top, and quickly moving to the upper functional partition), which is convenient for users to quickly move the focus according to the functional partition of the current interface, saving interaction time and improving interaction efficiency. The above semantic information can be the associated information of the functional partition. For example, in the interface of a virtual reality game, there are character selection functional partitions, clothing functional partitions, weapon functional partitions, etc. If the current focus is on the character selection functional partition, the semantic information indicates that after selecting the character, the next selection is clothing or weapons, that is, the next functional partition is the clothing functional partition or the weapon functional partition.

[0071] As can be seen from the above, automatically selecting the next appropriate focus according to the context information of the focus can avoid the user from manually moving the focus one by one, reducing the complexity and time cost of the operation, especially in large applications, which can improve the user's work efficiency.

[0072] The position of the default focus of the functional partition is related to the interface hierarchy and interface content. The rules of the default focus are as follows: hierarchy priority, that is, the focus is on the top-level interface by default; core operation priority, the default operation is determined according to the services provided by the specific application, such as opening the music function or Xiaomo suggestion, and the default focus position is determined according to the interaction target with the highest function priority.

[0073] In some AR application scenarios, users often change the viewing angle by moving their heads, but the focus position does not follow the change of the viewing angle, which may make it difficult for users to operate accurately. Therefore, in one example, the viewing angle of the head-mounted display device is detected by a sensor installed on the head-mounted display device. After the focus in the display interface is moved, when the viewing angle of the head-mounted display device is switched, a corresponding signal can be received from the sensor, and then the viewing angle switching information of the head-mounted display device is obtained from the sensor, and then the position of the focus in the display interface is updated according to the viewing angle switching information. For example: the deflection direction and deflection angle of the AR glasses are detected by the gyroscope of the AR glasses, and it is determined that the viewing angle of the AR glasses is converted from the front viewing angle to the left viewing angle, and then the position is automatically adjusted according to the display requirements of the virtual elements in the interface under the left viewing angle to follow the change of the viewing angle and keep synchronization with the user's line of sight. By changing the focus following the viewing angle, the user's operation accuracy and experience can be improved, the user can avoid frequently adjusting the focus position, and the interaction efficiency can be improved. Among them, the perspective switching is not limited to the above-mentioned perspective switching situation, and other perspective switching situations such as switching from the first-person perspective to the third-person perspective can also be used.

[0074] In this embodiment, due to the small field of view of the AR glasses, when the user moves his head to view content at different angles, the previous focus position may still remain on the other side. At this time, it is also necessary to re-judge the user's perspective and update the focus position to the current perspective to avoid "loss of focus".

[0075] In one implementation scenario, when the interactive control intersects with the field of view of the AR glasses, the interactive control moves slightly forward along the Z axis toward the user, and the transparency and saturation of the display gradually increase from weak to strong (transparency 80%→100%, saturation 80%→100%), making it easier for the user to select and position.

[0076] In one implementation scenario, when the field of view of the AR glasses is detected to move, the focus is also intelligently determined based on the virtual scene. Specifically, the core application is determined based on the virtual scene, and when the interaction area of ​​the core application intersects with the field of view of the AR glasses, the default focus is obtained based on the core application, and the current focus is moved to the default focus. For example: when a system notification or message notification is received, the core application is message processing, and the interactive area of ​​the core application is the message notification bar in the display interface. When the field of view of the AR glasses intersects with the message notification bar, the focus is moved to the default focus of the message notification bar, so that the user can quickly click to view details; when the user is listening to music, the core application is music control, and the interactive area of ​​the core application is the music playback component in the display interface. When the field of view of the AR glasses intersects with the music playback component, the focus is moved to the default focus of the music playback component: the pause & play button, and then the focus on the music playback component is moved according to the touch command, so that the user can quickly pause, switch songs, and other operations on the music playback component; when the user is using navigation, the core application is navigation control, and the interactive area of ​​the core application is the navigation component in the display interface. When the field of view of the AR glasses intersects with the navigation component, the focus is moved to the default focus of the navigation component, and then the focus on the navigation component is moved according to the touch command, so that the user can quickly stop navigation or re-plan the route on the navigation component.

[0077] In summary, this embodiment realizes fast interaction by analyzing the virtual scene, and integrates the touch instructions generated by the finger clip and the virtual scene. It can quickly realize the interaction between the finger clip and the VR glasses according to the virtual scene, thereby improving the interaction efficiency.

[0078] The present invention also provides an interactive device, including a finger clip for interacting with a head-mounted display device, and the above-mentioned virtual scene interaction method is realized through the communication between the finger clip and the head-mounted display device. A trackball is provided at the top of the finger clip, and a touch bar is provided on the side wall of the finger clip. Both the trackball and the touch bar can generate touch commands in response to user operations. The touch commands generated by the trackball can move the focus in the display interface output by the head-mounted display device, call shortcut functions, select virtual controls in the display interface output by the head-mounted display device, etc.; the touch commands generated by the touch bar can drag or scale virtual controls in the display interface output by the head-mounted display device, etc. This makes the interaction of virtual scenes simple and efficient.

[0079] Based on the above embodiments, the present invention further provides a head mounted display device, whose principle block diagram can be shown as follows: Figure 7As shown. The head-mounted display device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the head-mounted display device is used to provide computing and control capabilities. The memory of the head-mounted display device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a virtual scene interaction program. The internal memory provides an environment for the operation of the operating system and the virtual scene interaction program in the non-volatile storage medium. The network interface of the head-mounted display device is used to communicate with the interactive device through a network connection. When the virtual scene interaction program is executed by the processor, the steps of any one of the above-mentioned virtual scene interaction methods are implemented.

[0080] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the head-mounted display device to which the solution of the present invention is applied. The specific head-mounted display device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0081] In one embodiment, a head mounted display device is provided. The head mounted display device includes a memory, a processor, and a virtual scene interaction program stored in the memory and executable on the processor. When the virtual scene interaction program is executed by the processor, the following operation instructions are performed:

[0082] Receiving a touch control instruction generated by the interactive device in response to a user operation;

[0083] Determining a virtual scene of the head mounted display device;

[0084] According to the touch control instruction and the virtual scene, the virtual elements in the display interface are controlled.

[0085] Optionally, the virtual element includes a focus and a virtual control, and the controlling the virtual element in the display interface according to the touch instruction and the virtual scene includes:

[0086] A shortcut function is called according to the touch command and the virtual scene, or a focus in the display interface is moved according to the touch command and the virtual scene.

[0087] Optionally, after the focus in the display interface is moved, the focus is also highlighted; or, an animation effect is displayed when the focus in the display interface is moved.

[0088] Optionally, the virtual element includes a virtual control and a focus, and the virtual control is partitioned in advance according to the function corresponding to the virtual control in the display interface based on the virtual scene, and the controlling of the virtual element in the display interface according to the touch instruction and the virtual scene includes:

[0089] Acquire a functional partition corresponding to the virtual control where the focus is located in the display interface according to the virtual scene, and acquire a next functional partition according to the semantic information, the touch instruction and the functional partition;

[0090] Get the default focus position of the next functional partition;

[0091] The focus in the display interface is moved to the default focus position.

[0092] Optionally, the virtual element is the focus, and after the step of controlling the virtual element in the display interface according to the touch instruction and the virtual scene, the method further includes:

[0093] Obtaining viewing angle switching information of the head mounted display device;

[0094] The focus position in the display interface is updated according to the viewing angle switching information.

[0095] Optionally, the virtual element is a focus, and controlling the virtual element in the display interface according to the touch instruction and the virtual scene includes:

[0096] According to the virtual scene, a core application is obtained;

[0097] When the interactive area of ​​the core application intersects with the field of view of the head-mounted display device, a default focus is obtained according to the core application, the default focus is set as the focus in the display interface, and the focus in the display interface is controlled according to the touch instruction.

[0098] Optionally, the interactive device is provided with a motion sensor, the touch command is gesture information, and calling a shortcut function according to the touch command and the virtual scene includes:

[0099] Acquiring location information of the motion sensor;

[0100] Based on the position information, obtaining gesture information;

[0101] A shortcut function is obtained according to the gesture information and the virtual scene, and the shortcut function is executed.

[0102] An embodiment of the present invention further provides a computer-readable storage medium, on which a virtual scene interaction program is stored. When the virtual scene interaction program is executed by a processor, the steps of any one of the virtual scene interaction methods provided in the embodiments of the present invention are implemented.

[0103] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0104] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0105] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0107] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / terminal equipment and method can be implemented in other ways. For example, the apparatus / terminal equipment embodiments described above are only illustrative, for example, the division of the above modules or units is only a logical function division, and in actual implementation, other division methods can be used, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0108] If the above-mentioned integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the above-mentioned computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0109] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A virtual scene interaction method, characterized in that: Applied to a head mounted display device, the head mounted display device outputs a display interface, the display interface is provided with virtual elements, the method comprises: Receiving a touch control instruction generated by the interactive device in response to a user operation; Determining a virtual scene of the head mounted display device; According to the touch control instruction and the virtual scene, the virtual elements in the display interface are controlled.

2. The virtual scene interaction method according to claim 1, characterized in that: The virtual elements include a focus and a virtual control, and the controlling the virtual elements in the display interface according to the touch control instruction and the virtual scene includes: Invoke a shortcut function according to the touch command and the virtual scene to control a virtual control in the display interface, or move a focus in the display interface according to the touch command and the virtual scene.

3. The virtual scene interaction method according to claim 2, characterized in that: After the focus in the display interface is moved, the focus is also highlighted; or, an animation effect is displayed when the focus in the display interface is moved.

4. The virtual scene interaction method according to claim 2, characterized in that: The interactive device is provided with a motion sensor, the touch command is gesture information, and calling a shortcut function according to the touch command and the virtual scene includes: Acquiring location information of the motion sensor; Based on the position information, obtaining gesture information; A shortcut function is obtained according to the gesture information and the virtual scene, and the shortcut function is executed.

5. The virtual scene interaction method according to claim 1, characterized in that: The virtual element includes a virtual control and a focus, and the method includes: Based on the virtual scene, partitioning the virtual controls in the display interface according to the functions corresponding to the virtual controls; The controlling the virtual elements in the display interface according to the touch command and the virtual scene includes: Acquire a functional partition corresponding to the virtual control where the focus is located in the display interface according to the virtual scene, and acquire a next functional partition according to the semantic information, the touch instruction and the functional partition; Get the default focus position of the next functional partition; The focus in the display interface is moved to the default focus position.

6. The virtual scene interaction method according to claim 1, characterized in that: The virtual element is the focus, and after the step of controlling the virtual element in the display interface according to the touch instruction and the virtual scene, the method further includes: Obtaining viewing angle switching information of the head mounted display device; The focus position in the display interface is updated according to the viewing angle switching information.

7. The virtual scene interaction method according to claim 1, characterized in that: The virtual element is the focus, and the controlling the virtual element in the display interface according to the touch command and the virtual scene includes: According to the virtual scene, a core application is obtained; When the interactive area of ​​the core application intersects with the field of view of the head-mounted display device, a default focus is obtained according to the core application, the default focus is set as the focus in the display interface, and the focus in the display interface is controlled according to the touch instruction.

8. An interactive device, characterized in that: Used to interact with a head-mounted display device, the interactive device is provided with a trackball, and the side wall of the interactive device is provided with a touch bar, the trackball is used to generate a first touch instruction in response to a user operation, and the first touch instruction is used to move the focus in the display interface output by the head-mounted display device or call a shortcut function; the touch bar is used to generate a second touch instruction in response to a user operation, and the second touch instruction is used to drag or scale a virtual control in the display interface output by the head-mounted display device.

9. A head mounted display device, characterized in that: The head mounted display device comprises a memory, a processor and a virtual scene interaction program stored in the memory and executable on the processor. When the virtual scene interaction program is executed by the processor, the steps of the virtual scene interaction method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a virtual scene interaction program, and when the virtual scene interaction program is executed by the processor, the steps of the virtual scene interaction method according to any one of claims 1 to 7 are implemented.

Images