Virtual picture display method and device, electronic equipment and storage medium

By obtaining user attitude information and line of sight in real time, and switching the display method of virtual screens according to the motion state, the problem of virtual screens in the headset device interfering with the real environment is solved, and accessibility observation and security after rapid user attitude changes are achieved.

CN118916110AActive Publication Date: 2024-11-08NETEASE (HANGZHOU) NETWORK CO LTD
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
CN202411017617.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-08
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

When using a headset device, it is difficult for users to quickly block the interference of virtual screens in emergencies to observe the real environment, and the prior art cannot meet the user's rapid posture changes and observation needs.

Method used

By obtaining the user's posture information and vision in real time, determining the user's movement state, and switching the display mode of the virtual screen according to the movement state, reducing the coverage ratio of the virtual screen facing the real world, such as clustering and shrinking the virtual screen along the line of sight or adjusting transparency. Ensure that users can observe the real environment in a timely manner.

Benefits of technology

It realizes the timely reduction of interference from virtual images when users need to pay attention to the real environment, improves the user's barrier-free observation ability after rapid posture changes, and ensures the security and perceived experience of user behavior.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916110A_ABST
    Figure CN118916110A_ABST
Patent Text Reader

Abstract

The invention provides a virtual picture display method and device, electronic equipment and a storage medium, a graphical user interface is provided through head-mounted display equipment, the graphical user interface displays a virtual picture and a real-world picture, and the method comprises the following steps: obtaining posture information and sight of a user in real time through the head-mounted display equipment; determining the motion state of the user according to the posture information; switching the display mode of the virtual picture from a first display mode to a second display mode in response to the switching of the motion state from the first state to the second state and the leaving of the visual line from the virtual picture; wherein the coverage ratio of the virtual picture to the real world picture in the first display mode is different from the coverage ratio of the virtual picture to the real world picture in the second display mode. The display mode of the virtual picture is adjusted according to the motion state of the user, and interference of the virtual picture is shielded in time when the user needs to pay attention to the real environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for displaying a virtual screen. Background Art

[0002] When using a head-mounted display device (Vision Pro, VP), users can switch between the virtual image and the real environment at any time to enhance the user's perceptual experience. For example, if the user needs to focus on the real environment, the virtual image displayed on the current graphical user interface needs to be switched to the real environment, but during the switching process, the display of the virtual image will interfere with the display of the real environment.

[0003] In the prior art, the interference of the virtual image is reduced by the user actively adjusting the display range of the virtual image, or by placing the virtual image directly at a certain position in the real environment. However, the above methods are difficult to meet the user's observation needs in emergency situations, and the observation needs of the real environment are limited. Therefore, how to timely block the interference of the virtual image when the user needs to pay attention to the real environment so as to directly observe the real environment is an urgent problem to be solved in this field. Summary of the invention

[0004] The purpose of the present application is to provide a method, device, electronic device and storage medium for displaying a virtual image in order to address the deficiencies in the above-mentioned prior art, so as to solve the problem in the prior art of how to timely shield the interference of the virtual image so as to directly observe the real environment when the user needs to pay attention to the real environment.

[0005] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for displaying a virtual screen, wherein a graphical user interface is provided through a head display device, and the graphical user interface displays a virtual screen and a real world screen, and the method includes:

[0007] Acquire the user's posture information and sight line in real time through the head display device;

[0008] Determining the motion state of the user according to the posture information;

[0009] In response to the movement state switching from the first state to the second state and the line of sight leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode, and the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

[0010] As a possible implementation manner, switching the display mode of the virtual screen from the first display mode to the second display mode includes:

[0011] The virtual screen is clustered and shrunk along a direction opposite to the direction of the line of sight, and the clustered and shrunk virtual screen is displayed.

[0012] As a possible implementation manner, the method further includes:

[0013] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight focusing on the virtual picture, switching the display mode of the virtual picture from the second display mode to a third display mode;

[0014] Wherein, in the third display mode, the coverage ratio of the virtual image to the real world image is greater than a first preset value, and the third display mode includes: a display area and a display position.

[0015] As a possible implementation manner, the third display mode is a previous display mode of the second display mode.

[0016] As a possible implementation, the display position is determined by the following process:

[0017] Acquire a subsequent sight line of the sight line focused on the virtual picture;

[0018] The display position is determined according to the position corresponding to the subsequent line of sight.

[0019] As a possible implementation manner, determining the motion state of the user according to the posture information includes:

[0020] Determine whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value;

[0021] If so, it is determined that the motion state of the user at the current moment is the first state; otherwise, it is determined that the motion state of the user at the current moment is the second state.

[0022] As a possible implementation, the posture information includes at least one or more of the following: angular acceleration, acceleration;

[0023] The step of determining whether the change value of the posture information in the motion period corresponding to the current moment is less than a preset value includes:

[0024] Obtaining the angular acceleration and acceleration at each moment in the movement period, the end moment of the movement period being the current moment;

[0025] If the difference between the angular acceleration at each moment in the movement period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the movement period corresponding to the current moment is less than the preset value.

[0026] In a second aspect, an embodiment of the present application provides a device for displaying a virtual screen, which provides a graphical user interface through a head display device, and the graphical user interface displays a virtual screen and a real world screen, and the device includes:

[0027] An acquisition module, used to acquire the user's posture information and sight line in real time through the head display device;

[0028] A determination module, used to determine the motion state of the user according to the posture information;

[0029] A switching module is used to switch the display mode of the virtual screen from a first display mode to a second display mode in response to the motion state switching from a first state to a second state and the line of sight leaving the virtual screen, wherein the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

[0030] As a possible implementation manner, the switching module is specifically used for:

[0031] The virtual screen is clustered and shrunk along a direction opposite to the direction of the line of sight, and the clustered and shrunk virtual screen is displayed.

[0032] As a possible implementation manner, the switching module is further used for:

[0033] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight focusing on the virtual picture, switching the display mode of the virtual picture from the second display mode to the third display mode;

[0034] Wherein, in the third display mode, the coverage ratio of the virtual image to the real world image is greater than a first preset value, and the third display mode includes: a display area and a display position.

[0035] As a possible implementation manner, the third display mode is a previous display mode of the second display mode.

[0036] As a possible implementation, the display position is determined by the following process:

[0037] Acquire a subsequent sight line of the sight line focused on the virtual picture;

[0038] The display position is determined according to the position corresponding to the subsequent line of sight.

[0039] As a possible implementation manner, the determining module is specifically used for:

[0040] Determine whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value;

[0041] If so, it is determined that the motion state of the user at the current moment is the first state; otherwise, it is determined that the motion state of the user at the current moment is the second state.

[0042] As a possible implementation, the posture information includes at least one or more of the following: angular acceleration, acceleration; the determination module is specifically used for:

[0043] Obtaining the angular acceleration and acceleration at each moment in the movement period, the end moment of the movement period being the current moment;

[0044] If the difference between the angular acceleration at each moment in the movement period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the movement period corresponding to the current moment is less than the preset value.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the virtual screen display method as described in any one of the first aspects above.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for displaying a virtual screen as described in any one of the first aspects above are executed.

[0047] According to the display method, device, electronic device and storage medium of the virtual screen of the embodiment of the present application, the user's posture information and line of sight are obtained in real time through the head display device; the user's motion state is determined according to the posture information; in response to the motion state switching from the first state to the second state and the line of sight leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode, and the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode. According to the embodiment of the present application, the first state and the second state respectively indicate the user state when the user's posture is constant or changes to different degrees, and the user's motion state is determined to be stable or unstable based on the user's posture information. When the user state switches from a stable state to an unstable state, and the user's line of sight is not focused on the virtual screen, it is determined that the user is undergoing a rapid posture change, thereby triggering the virtual screen to enter a motion auxiliary state, changing the display logic of the virtual screen, that is, switching the display mode of the virtual screen from the first display mode to the second display mode, thereby reducing the coverage ratio of the virtual screen to the real world screen, so as to achieve timely shielding of the interference of the virtual screen to directly observe the real environment when the user needs to pay attention to the real environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A schematic diagram showing a flow chart of a method for displaying a virtual screen provided in an embodiment of the present application is shown;

[0050] Figure 2 A schematic diagram of a switching interface of a virtual screen display mode provided in an embodiment of the present application is shown;

[0051] Figure 3 A schematic diagram of a switching interface of another virtual screen display mode provided in an embodiment of the present application is shown;

[0052] Figure 4 A schematic diagram showing a flow chart of a method for determining a motion state provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram showing a virtual screen visualization solution provided by an embodiment of the present application is shown;

[0054] Figure 6A schematic diagram showing another virtual screen visualization solution provided by an embodiment of the present application is shown;

[0055] Figure 7 A schematic diagram of the structure of a virtual screen display device provided in an embodiment of the present application is shown;

[0056] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0057] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0058] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0059] In order to enable those skilled in the art to use the content of this application, the following implementation is given in conjunction with the specific application scenario "Motion-assisted observation design based on rapid posture changes". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is mainly described around the display method of the virtual screen, it should be understood that this is only an exemplary embodiment.

[0060] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0061] In the prior art, the first method of actively observing the real environment by adjusting the display range and display state of the virtual screen by the user is only applicable to the case where the user actively switches the subject of observation, and it is difficult to meet the user's need to observe the real environment in emergency situations, such as standing up or sitting down, accidentally falling, turning back to respond to others' calls, and temporarily needing to get up to get something. The second method is that the user can directly place the virtual screen at a certain position in the real environment. When the user has a rapid posture change, other fields of vision will not be blocked, but the position where the virtual screen has been placed has the need to observe the real environment, which involves the need to actively move the virtual screen away, and the observation of the real environment is still limited.

[0062] In response to the above technical problems, an embodiment of the present application provides a method for displaying a virtual screen. In response to the problem that it is difficult for users to quickly block the interference of virtual screens when using a Vision Pro device, a motion-assisted observation design scheme based on rapid posture changes of users is proposed from three aspects: core judgment when the user produces rapid posture changes, display logic after the virtual screen enters a motion-assisted state, and recognition of the virtual screen exiting the motion-assisted state. This helps users reduce the interference of virtual screens in a timely manner when they need to pay attention to the real environment.

[0063] In one embodiment of the present application, the method for displaying a virtual screen can be run on a local terminal device or a server. When the method for displaying a virtual screen is run on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0064] In an optional implementation, various cloud applications, such as cloud games, can be run under the cloud interaction system. Taking cloud games as an example, cloud games refer to a gaming method based on cloud computing. In the operation mode of cloud games, the running body of the game program and the game screen presentation body are separated. The storage and operation of the game loading method are completed on the cloud game server. The role of the client device is used for data reception, transmission and presentation of the game screen. For example, the client device can be a head-mounted display device, but the information processing is performed by the cloud game server in the cloud. When playing the game, the player operates the client device to send operation instructions to the cloud game server. The cloud game server runs the game according to the operation instructions, encodes and compresses the game screen and other data, and returns it to the client device through the network. Finally, the client device decodes and outputs the game screen.

[0065] In an optional embodiment, taking a game as an example, a local terminal device stores a game program and is used to present a game screen. The local terminal device is used to interact with the player through a graphical user interface, that is, the game program is downloaded and installed and run by an electronic device in a conventional manner. The local terminal device may provide the graphical user interface to the player in a variety of ways, for example, it may be rendered and displayed on the display screen of the terminal, or provided to the player through a holographic projection. For example, the local terminal device may include a display screen and a processor, the display screen is used to present a graphical user interface, the graphical user interface includes a game screen, and the processor is used to run the game, generate a graphical user interface, and control the display of the graphical user interface on the display screen. In an embodiment of the present application, the local terminal device may be a head-mounted display device.

[0066] In a possible implementation, an embodiment of the present application provides a method for displaying a virtual screen, providing a graphical user interface through a head display device, and the graphical user interface displays a virtual screen and a real-world screen.

[0067] Figure 1 The flowchart of a method for displaying a virtual screen provided by an embodiment of the present application is shown. Figure 1 As shown, the method specifically comprises the following steps:

[0068] S101. Obtain the user's posture information and sight line in real time through a head display device.

[0069] Optionally, the mixed reality system of the head-mounted display device Vision Pro can detect head movement, assist in judging changes in head posture, and track eye movements to clarify the user's gaze direction. On this basis, the user's line of sight position can be determined in real time using the line of sight tracking technology of the head-mounted display device Vision Pro. For example, by real-time monitoring and capturing the user's eye position and movement direction, and then determining the user's line of sight focus through complex image processing and algorithms, the user's line of sight is acquired.

[0070] Optionally, based on the gyroscope, angular velocity sensor, acceleration sensor and other sensing devices and detection technology carried by the head display device Vision Pro, detection data of the user's head movement and eye movement are obtained to monitor the user's movement status, and then the user's real-time movement status is calculated through the detection data to obtain the user's posture information.

[0071] S102: Determine the user's motion state according to the posture information.

[0072] Optionally, the user's motion state includes a first state and a second state. The first state refers to the user being in a stable and focused state, the user's motion state is constant, and the constant user state includes the user's posture remaining unchanged or the user's posture changes slightly and relatively steadily. The second state refers to the user being in a state of rapid posture changes, and the user's posture is obviously changing rapidly.

[0073] Optionally, if the user's motion state is determined to be constant based on the user's posture information, the user is judged to be in the first state at this time. If the user's posture information is monitored to be greater than a preset value, the user is judged to be rapidly changing posture, which means that the user is in the second state at this time.

[0074] S103 , in response to the movement state switching from the first state to the second state and the sight leaving the virtual picture, switching the display mode of the virtual picture from the first display mode to the second display mode.

[0075] The coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode. For example, in the first display mode, the coverage ratio of the virtual screen to the real world screen is greater than a first preset value, and in the second display mode, the coverage ratio of the virtual screen to the real world screen is less than a second preset value.

[0076] Optionally, when the user is in the first state, if it is detected that the user is changing his posture rapidly, it is determined that the user's motion state has changed, that is, the user's state is no longer stable, and changes from the first state to the second state. On this basis, in response to the user's motion state switching from the first state to the second state, the display mode of the virtual screen is changed, and the display mode of the virtual screen is switched from the first display mode to the second display mode.

[0077] For example, when the user's motion state is in the first state, the virtual screen is in the first display mode, that is, the virtual screen can be displayed normally, and the virtual screen can cover the real world screen. When the user's motion state switches from the first state to the second state, that is, the user has the need to observe the real environment at this time, it is necessary to reduce the interference of the virtual screen on the real environment. At this time, the display mode of the virtual screen is switched from the first display mode to the second display mode. Since the coverage ratio of the virtual screen to the real world screen in the second display mode is less than the coverage ratio of the virtual screen to the real world screen in the first display mode, the occlusion of the virtual screen on the real world screen is reduced.

[0078] Based on this, according to the display method of the virtual screen provided by the embodiment of the present application, the first state and the second state respectively indicate the user state when the user posture is constant or changes to different degrees, and determine whether the user's motion state is stable or unstable based on the user's posture information. When the user state switches from a stable state to an unstable state, and the user's line of sight is not focused on the virtual screen, it is determined that the user is undergoing a rapid posture change, which in turn triggers the virtual screen to enter a motion auxiliary state, and changes the display logic of the virtual screen, that is, switches the display mode of the virtual screen from the first display mode to the second display mode, thereby reducing the coverage ratio of the virtual screen to the real world screen, thereby achieving timely shielding of the interference of the virtual screen to directly observe the real environment when the user needs to pay attention to the real environment.

[0079] As a possible implementation, the above step S103 switches the display mode of the virtual screen from the first display mode to the second display mode, including:

[0080] The virtual screen is clustered and shrunk in a direction opposite to the direction of the line of sight, and the virtual screen after the clustering and shrinking is displayed.

[0081] Optionally, clustering refers to reducing the display area by shrinking the virtual screen, storing the virtual screen in a unified entrance, folding the virtual screen, etc. and gathering them in the same position.

[0082] For example, refer to Figure 2 As shown, the clustering shrinking direction of the virtual screen is related to the user's line of sight, that is, the direction of the line of sight guides the subsequent adjustment direction of the virtual screen. For example, the virtual screen can be clustered and shrunk in the direction opposite to the line of sight. It should be noted that the clustering shrinking direction of the virtual screen may not be completely shrunk in the direction opposite to the line of sight. It can be that after reversing the user's moving direction, the shrinking direction after the direction of movement is reversed as the clustering direction of the virtual screen. Specifically, the shrinking direction can be divided into 4x, where x = 1, 2, and clustering is performed according to similar directions.

[0083] For example, refer to Figure 3 As shown, the elements in the graphical user interface are grouped according to their distance from the user's line of sight in the opposite direction. The spatial partitioning technology or the geometric algorithm directly based on the user's line of sight can be used to efficiently cluster the elements in the virtual screen. For each clustered element, its size or visual performance can be proportionally reduced according to the distance of each element from the starting point of the line of sight in the opposite direction. It should be noted that the farther the element is from the starting point of the line of sight in the opposite direction, the greater the degree of shrinkage can be, so as to simulate the visual effect of "shrinking in the direction opposite to the line of sight".

[0084] Optionally, according to different operating states of the user, the transparency of the virtual screen can also be changed to avoid the virtual screen from blocking the real world screen. The transparency of the virtual screen refers to the degree of opacity of the virtual screen. For example, if the transparency of the virtual screen is 0%, it means that the virtual screen is completely opaque, and the content of the real world screen covered by the virtual screen is completely invisible. If the transparency of the virtual screen is 100%, it means that the virtual screen is completely transparent, and the content of the real world screen covered by the virtual screen is completely visible, but the virtual screen itself is invisible.

[0085] On this basis, the method also includes:

[0086] In response to the motion state switching from the first state to the second state and the sight leaving the virtual picture, the transparency of the virtual picture is adjusted to a preset value.

[0087] Exemplarily, when it is detected that the user is undergoing a rapid posture change and the user's line of sight is away from the virtual screen, that is, the user has a need to pay attention to the real environment, the transparency of the virtual screen can be adjusted from the third preset value to the fourth preset value, wherein the third preset value is less than the fourth preset value. The third preset value and the fourth preset value here can be set to a custom setting based on the condition that the user's observation of the screen content is not affected. For example, when the transparency of the virtual screen is the third preset value, the visibility of the real world screen cannot affect the visibility of the virtual screen, so as to ensure that the user can clearly see the content of the virtual screen, and when the transparency of the virtual screen is the fourth preset value, the visibility of the virtual screen cannot affect the visibility of the real world screen, so as to ensure that the user can clearly see the content of the real world screen.

[0088] Based on the above content, the transparency of the virtual screen in the first display mode is different from the transparency of the virtual screen in the second display mode. The above step S103 switches the display mode of the virtual screen from the first display mode to the second display mode, and also includes:

[0089] The virtual picture is clustered and shrunk in a direction opposite to the direction of the line of sight, and the transparency of the virtual picture is adjusted. In addition, the transparency of the virtual picture in the first display mode is less than the transparency of the virtual picture in the second display mode.

[0090] For example, when it is detected that the user's motion state switches from the first state to the second state, and the user's line of sight leaves the virtual screen, the virtual screen is clustered and shrunk in the direction opposite to the user's line of sight, reducing the display area of ​​the virtual screen while increasing the transparency of the virtual screen. In this way, when the user's posture changes significantly, the problem of the virtual screen blocking the real world screen is solved by changing the display position, display area and transparency of the virtual screen.

[0091] Based on this, the display mode of the virtual screen is switched in response to changes in user posture, which facilitates the observation of the real scene after the user's posture changes and ensures the safety of user behavior.

[0092] As a possible implementation, when the user has a need to pay attention to the real environment, in order to avoid the problem of the virtual image blocking the real world image, the method further includes:

[0093] In response to the motion state switching from the first state to the second state, based on the display information of the real world picture, an idle display area in the graphical user interface is determined, and the virtual picture is displayed in the idle display area.

[0094] Exemplarily, the idle display area refers to an area in the graphical user interface that is not occupied by other important information or controls, that is, an area without display content. When the user's motion state switches from the first state to the second state, that is, when the user's posture changes significantly, the virtual screen can be moved to the idle display area for display. Specifically, according to the display position coordinates of the real-world screen in the graphical user interface, the area occupied by the real-world screen in the graphical user interface is determined, and then it is determined whether there is display content in the display area other than the occupied area in the graphical user interface, and the display area without display content is used as the idle display area.

[0095] Furthermore, the offset of the virtual screen is determined according to the display position coordinates of the virtual screen and the display position coordinates of the idle display area, and the virtual screen is controlled to move to the idle display area for display based on the offset. In this process, if the display area of ​​the current virtual screen is larger than the display area of ​​the idle display area, that is, the virtual screen cannot be fully displayed in the idle display area, the display area of ​​the virtual screen is adjusted until the virtual screen can be fully displayed in the idle display area.

[0096] Based on this, by determining the idle display area in the graphical user interface and moving the virtual image to the idle display area for display, this operation can not only avoid the problem of the virtual image blocking the real world image, but also can fully display the virtual image and the real world image on the graphical user interface at the same time, thereby ensuring the integrity of the displayed content and enhancing the user's perceptual experience.

[0097] Figure 4 FIG. 1 is a flow chart of a method for determining a motion state provided in an embodiment of the present application. As a possible implementation method, refer to Figure 4 As shown, the above step S102 determines the user's motion state according to the posture information, and specifically includes the following steps:

[0098] S401, determining whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value;

[0099] Optionally, the angular acceleration and acceleration at each moment in the movement period are obtained, and the end moment of the movement period is the current moment; if the difference between the angular acceleration at each moment in the movement period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the movement period corresponding to the current moment is less than the preset value.

[0100] Exemplarily, the posture information includes at least one or more of the following: angular acceleration, acceleration. Among them, the preset value corresponding to the angular acceleration is mainly determined based on the monitored pitch angle and yaw angle of the user's head, and the preset value corresponding to the acceleration is mainly based on the time it takes for a person to walk from rest to start. In addition, the posture information can also include position change information, which includes the position change amount between the position before and after the user moves.

[0101] Exemplarily, if the user is undergoing rapid posture changes, the mixed reality system based on the head display device Vision Pro can directly detect the direction and angle of movement of the user's head, that is, obtain the user's position after movement. According to the user's initial line of sight position and the position after movement, the user's movement direction and angle can be determined, such as how many degrees to rotate in a certain direction, or how much distance to move in a certain direction. Among them, the user's movement direction and angle can also be used to guide the orientation information when clustering virtual images. In addition, the user's movement direction and angle can also be used to ensure that the clustered and shrunk virtual image is away from the user's current line of sight, that is, the distance between the cluster storage position of the virtual image and the user's initial line of sight is greater than the movement distance of the user's line of sight.

[0102] S402: If yes, determine that the user's motion state at the current moment is the first state; otherwise, determine that the user's motion state at the current moment is the second state.

[0103] Exemplarily, if the difference between the angular acceleration at each moment in the exercise period and the angular acceleration at the previous moment is less than the first preset value, and the difference between the acceleration at each moment and the acceleration at the previous moment is less than the second preset value, it means that the user's exercise state at this time is relatively constant, that is, the user's exercise state at the current moment is determined to be a stable state. On the contrary, if the difference between the angular acceleration at each moment in the exercise period and the angular acceleration at the previous moment is greater than the first preset value, or the difference between the acceleration at each moment and the acceleration at the previous moment is greater than the second preset value, it means that the user is undergoing a rapid posture change, that is, the user's exercise state at the current moment is determined to be an unstable state.

[0104] Accordingly, when the posture information is position change information, the magnitude of the user's posture change can be determined by comparing the position change between the user's position before and after movement with a preset threshold, thereby determining whether the user's movement state is stable or unstable.

[0105] Based on this, according to the degree of change of the user's posture information, it is determined whether the user's motion state is in a stable state or an unstable state, which is convenient for determining the display method of the virtual screen displayed in the graphical user interface and the real-world screen according to the user's motion state, so as to reduce the interference of the virtual screen on the real-world screen when the user observes the real-world screen.

[0106] As a possible implementation, the method for displaying the virtual screen further includes:

[0107] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight is focused on the virtual picture, the display mode of the virtual picture is switched from the second display mode to the third display mode; wherein, in the third display mode, the coverage ratio of the virtual picture to the real world picture is greater than the first preset value, and the third display mode includes: display area and display position.

[0108] Optionally, in response to the user's motion state switching from a stable state to an unstable state, in order to avoid interference of the virtual screen with the real world screen, the display mode of the virtual screen can be switched from the first display mode to the second display mode, that is, the virtual screen is displayed in clusters, so as to reduce the occlusion ratio of the virtual screen to the real world screen. Correspondingly, when the user's motion state is restored from an unstable state to a stable state, and the user's line of sight is refocused on the virtual screen, the display mode of the virtual screen can also be switched from the second display mode to the third display mode, that is, the virtual screen is restored to display.

[0109] It should be noted that the third display mode is the previous display mode of the second display mode, and the third display mode can be the same as the first display mode or different from the first display mode. However, the same thing is that in the first display mode and the third display mode, the virtual screen can be displayed normally, while in the second display mode, the virtual screen enters the motion-assisted observation state and will quickly cluster and retract in the direction opposite to the user's line of sight.

[0110] For example, if the user's motion state is restored from an unstable state to a stable state, and the user's sight is refocused on the virtual screen, the display state of the virtual screen can be restored to normal, that is, the clustered and shrunk virtual screen is re-displayed. Specifically, the virtual screen can be re-displayed at the display position before the clustering and shrinking of the virtual screen.

[0111] Among them, judging whether the user's motion state has returned to a stable state is also based on whether the user's posture has not changed rapidly within a period of time. This period of time can be set to 2 seconds. The specific value is not limited here and can be set according to the accuracy of pre-fetching.

[0112] It should be noted that it takes a confirmation process for the user's sight to refocus on the virtual screen. For example, it can be observed that the user's sight refocuses on the virtual screen for z seconds, where z can be a value between 0.8 and 3.

[0113] Exemplarily, the display mode of the virtual screen is switched from the second display mode to the third display mode, and after the user's line of sight is refocused on the virtual screen, the user can refer to Figure 5 and Figure 6 The visualization scheme shown provides visual feedback on the virtual screen.

[0114] It should be noted that the head-mounted display device Vision Pro supports head-locked content and world-locked content, and the embodiments of the present application are mainly aimed at head-locked content, that is, the virtual image and the user's head always remain relatively still.

[0115] Optionally, the display position is determined by the following process: obtaining a line of sight subsequent to the line of sight focused on the virtual picture; and determining the display position according to a position corresponding to the subsequent line of sight.

[0116] For example, the next sight line after the sight line focused on the virtual screen can still be obtained through the sight line tracking technology of the head display device Vision Pro. On this basis, the position corresponding to the next sight line is used as the display position of the virtual screen. Before that, the content display of the display position can also be pre-adjusted or optimized.

[0117] Based on this, according to the embodiment of the present application, based on the user's demand and judgment for the motion assistance state in the head display device Vision Pro, and the adaptive display of the virtual screen in the motion assistance state, the user can switch between the virtual screen and the real world screen at any time, which is particularly convenient for the user to observe the real scene without obstacles after the rapid posture change, ensuring the safety of the user's movement behavior. At the same time, the embodiment of the present application meets the real use needs of users and further improves the convenience of using the head display device in daily life.

[0118] Based on the same inventive concept, the embodiment of the present application also provides a virtual screen display device corresponding to the virtual screen display method. Since the principle of solving the problem by the virtual screen display device in the embodiment of the present application is similar to the above-mentioned virtual screen display method in the embodiment of the present application, the implementation of the virtual screen display device can refer to the implementation of the virtual screen display method, and the repeated parts will not be repeated.

[0119] Figure 7 The structure diagram of a virtual screen display device provided by an embodiment of the present application is shown. Figure 7 As shown, the virtual screen display device 700 includes: an acquisition module 701, a determination module 702, and a switching module 703, wherein:

[0120] An acquisition module 701 is used to acquire the user's posture information and sight line in real time through a head display device;

[0121] A determination module 702, used to determine the user's motion state according to the posture information;

[0122] The switching module 703 is used to switch the display mode of the virtual screen from the first display mode to the second display mode in response to the movement state switching from the first state to the second state and the line of sight leaving the virtual screen. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

[0123] Based on this, according to the display device of the virtual screen of the embodiment of the present application, the first state and the second state respectively indicate the user state when the user posture is constant or changes to different degrees, and determine whether the user's motion state is stable or unstable based on the user's posture information. When the user state switches from a stable state to an unstable state, and the user's line of sight is not focused on the virtual screen, it is determined that the user is undergoing a rapid posture change, which in turn triggers the virtual screen to enter a motion auxiliary state, and changes the display logic of the virtual screen, that is, switches the display mode of the virtual screen from the first display mode to the second display mode, thereby reducing the coverage ratio of the virtual screen to the real world screen, thereby achieving timely shielding of the interference of the virtual screen to directly observe the real environment when the user needs to pay attention to the real environment.

[0124] In a possible implementation manner, the switching module 703 is specifically configured to:

[0125] The virtual screen is clustered and shrunk in a direction opposite to the direction of the line of sight, and the virtual screen after the clustering and shrinking is displayed.

[0126] In a possible implementation manner, the switching module 703 is further configured to:

[0127] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight is focused on the virtual picture, the display mode of the virtual picture is switched from the second display mode to the third display mode;

[0128] Among them, in the third display mode, the coverage ratio of the virtual picture to the real world picture is greater than the first preset value, and the third display mode includes: display area and display position.

[0129] In a possible implementation manner, the third display mode is a display mode previous to the second display mode.

[0130] In a possible implementation manner, the determination module 702 is further configured to:

[0131] Acquire a subsequent sight line of the sight line focused on the virtual picture;

[0132] The display position is determined according to the position corresponding to the next line of sight.

[0133] In a possible implementation manner, the determination module 702 is specifically configured to:

[0134] Determine whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value;

[0135] If so, it is determined that the user's motion state at the current moment is the first state; otherwise, it is determined that the user's motion state at the current moment is the second state.

[0136] In a possible implementation manner, the posture information includes at least one or more of the following: angular acceleration, acceleration; the above-mentioned determination module 702 is specifically used for:

[0137] Get the angular acceleration and acceleration at each moment in the exercise period, and the end time of the exercise period is the current moment;

[0138] If the difference between the angular acceleration at each moment in the motion period and the angular acceleration at the previous moment is less than the first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than the second preset value, then it is determined that the change value of the posture information in the motion period corresponding to the current moment is less than the preset value.

[0139] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0140] The present application embodiment also provides an electronic device 800, such as Figure 8 As shown, it is a schematic diagram of the structure of an electronic device 800 provided in an embodiment of the present application, including: a processor 801, a memory 802, and optionally, a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device 800 is running, the processor 801 communicates with the memory 802 through the bus 803. When the machine-readable instructions are executed by the processor 801, the following processing is performed:

[0141] Obtain the user's posture information and line of sight in real time through the head display device;

[0142] Determine the user's motion state based on the posture information;

[0143] In response to the movement state switching from the first state to the second state and the line of sight leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode, and the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

[0144] In a possible implementation manner, when the processor 801 executes the instruction to switch the display mode of the virtual screen from the first display mode to the second display mode, it is specifically configured to:

[0145] The virtual screen is clustered and shrunk in a direction opposite to the direction of the line of sight, and the virtual screen after the clustering and shrinking is displayed.

[0146] In a possible implementation manner, the processor 801 is further configured to:

[0147] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight is focused on the virtual picture, the display mode of the virtual picture is switched from the second display mode to the third display mode;

[0148] Among them, in the third display mode, the coverage ratio of the virtual picture to the real world picture is greater than the first preset value, and the third display mode includes: display area and display position.

[0149] In a possible implementation manner, the third display mode is a display mode previous to the second display mode.

[0150] In a possible implementation manner, when executing the instruction for determining the display position, the processor 801 is specifically configured to:

[0151] Acquire a subsequent sight line of the sight line focused on the virtual picture;

[0152] The display position is determined according to the position corresponding to the next line of sight.

[0153] In a possible implementation manner, when the processor 801 executes the instruction to determine the motion state of the user according to the posture information, it is specifically configured to:

[0154] Determine whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value;

[0155] If so, it is determined that the user's motion state at the current moment is the first state; otherwise, it is determined that the user's motion state at the current moment is the second state.

[0156] In a possible implementation manner, the posture information includes at least one or more of the following: angular acceleration, acceleration; when the processor 801 executes an instruction to determine whether a change value of the posture information in the motion period corresponding to the current moment is less than a preset value, it is specifically used to:

[0157] Get the angular acceleration and acceleration at each moment in the exercise period, and the end time of the exercise period is the current moment;

[0158] If the difference between the angular acceleration at each moment in the motion period and the angular acceleration at the previous moment is less than the first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than the second preset value, then it is determined that the change value of the posture information in the motion period corresponding to the current moment is less than the preset value.

[0159] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are performed:

[0160] Obtain the user's posture information and line of sight in real time through the head display device;

[0161] Determine the user's motion state based on the posture information;

[0162] In response to the movement state switching from the first state to the second state and the line of sight leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode, and the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

[0163] In a possible implementation manner, when the processor executes the instruction to switch the display mode of the virtual screen from the first display mode to the second display mode, it is specifically configured to:

[0164] The virtual screen is clustered and shrunk in a direction opposite to the direction of the line of sight, and the virtual screen after the clustering and shrinking is displayed.

[0165] In a possible implementation manner, the processor is further configured to:

[0166] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight is focused on the virtual picture, the display mode of the virtual picture is switched from the second display mode to the third display mode;

[0167] Among them, in the third display mode, the coverage ratio of the virtual picture to the real world picture is greater than the first preset value, and the third display mode includes: display area and display position.

[0168] In a possible implementation manner, the third display mode is a display mode previous to the second display mode.

[0169] In a possible implementation manner, when the processor executes the instruction for determining the display position, it is specifically configured to:

[0170] Acquire a subsequent sight line of the sight line focused on the virtual picture;

[0171] The display position is determined according to the position corresponding to the next line of sight.

[0172] In a possible implementation manner, when the processor executes the instruction to determine the motion state of the user according to the posture information, it is specifically used to:

[0173] Determine whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value;

[0174] If so, it is determined that the user's motion state at the current moment is the first state; otherwise, it is determined that the user's motion state at the current moment is the second state.

[0175] In a possible implementation manner, the posture information includes at least one or more of the following: angular acceleration, acceleration; when the processor 801 executes an instruction to determine whether a change value of the posture information in the motion period corresponding to the current moment is less than a preset value, it is specifically used to:

[0176] Get the angular acceleration and acceleration at each moment in the exercise period, and the end time of the exercise period is the current moment;

[0177] If the difference between the angular acceleration at each moment in the motion period and the angular acceleration at the previous moment is less than the first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than the second preset value, then it is determined that the change value of the posture information in the motion period corresponding to the current moment is less than the preset value.

[0178] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0179] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0180] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for displaying a virtual screen, characterized in that: Providing a graphical user interface by means of a head display device, wherein the graphical user interface displays a virtual image and a real world image, the method comprising: Acquire the user's posture information and sight line in real time through the head display device; Determining the motion state of the user according to the posture information; In response to the movement state switching from the first state to the second state and the line of sight leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode, and the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

2. The method for displaying a virtual screen according to claim 1, characterized in that: The step of switching the display mode of the virtual screen from the first display mode to the second display mode includes: The virtual screen is clustered and shrunk along a direction opposite to the direction of the line of sight, and the clustered and shrunk virtual screen is displayed.

3. The method for displaying a virtual screen according to claim 1, characterized in that: Also includes: In response to the motion state switching from the second state to the first state, and the first state lasting for a preset time, and the line of sight focusing on the virtual picture, switching the display mode of the virtual picture from the second display mode to a third display mode; Wherein, in the third display mode, the coverage ratio of the virtual image to the real world image is greater than a first preset value, and the third display mode includes: a display area and a display position.

4. The method for displaying a virtual screen according to claim 3, characterized in that: The third display mode is a display mode previous to the second display mode.

5. The method for displaying a virtual screen according to claim 3, characterized in that: The display position is determined by the following process: Acquire a subsequent sight line of the sight line focused on the virtual picture; The display position is determined according to the position corresponding to the subsequent line of sight.

6. The method for displaying a virtual screen according to any one of claims 1 to 5, characterized in that: Determining the motion state of the user according to the posture information includes: Determine whether the change value of the posture information in the exercise period corresponding to the current moment is less than a preset value; If so, it is determined that the motion state of the user at the current moment is the first state; otherwise, it is determined that the motion state of the user at the current moment is the second state.

7. The method for displaying a virtual screen according to claim 6, characterized in that: The posture information includes at least one or more of the following: angular acceleration, acceleration; The step of determining whether the change value of the posture information in the motion period corresponding to the current moment is less than a preset value includes: Obtaining the angular acceleration and acceleration at each moment in the movement period, the end moment of the movement period being the current moment; If the difference between the angular acceleration at each moment in the movement period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the movement period corresponding to the current moment is less than the preset value.

8. A virtual image display device, characterized in that: A graphical user interface is provided through a head display device, wherein the graphical user interface displays a virtual image and a real world image, and the apparatus comprises: An acquisition module, used to acquire the user's posture information and sight line in real time through the head display device; A determination module, used to determine the motion state of the user according to the posture information; A switching module is used to switch the display mode of the virtual screen from a first display mode to a second display mode in response to the motion state switching from a first state to a second state and the line of sight leaving the virtual screen, wherein the coverage ratio of the virtual screen to the real world screen in the first display mode is different from the coverage ratio of the virtual screen to the real world screen in the second display mode.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the method for displaying a virtual screen as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for displaying a virtual picture as claimed in any one of claims 1 to 7 are executed.

Citation Information

Patent Citations

  • Information processing method and electronic device

    CN104750249A

  • Method and device for switching environment picture

    CN106527682A

  • Switching method and device for virtual mode and realistic mode, and virtual reality equipment

    CN106951316A

  • Method and system for switching between VR mode and AR mode based on MR head-mounted display equipment

    CN107507280A

  • Interaction method and device in virtual reality spatial environment

    CN107957774A