Display method and device of virtual picture, electronic equipment and storage medium

Abstract

The application provides a virtual picture display method and device, electronic equipment and storage medium, a graphical user interface is provided through a head-mounted device, the graphical user interface displays a virtual picture and a real world picture, the method comprises the following steps: acquiring the posture information and the line of sight of the user in real time through the head-mounted device; determining the motion state of the user according to the posture information; in response to the motion state switching from a first state to a second state and the line of sight leaving the virtual picture, switching the display mode of the virtual picture from a first display mode to a second display mode; wherein the coverage ratio of the virtual picture to the real world picture in the first display mode is different from the coverage ratio of the virtual picture to the real world picture in the second display mode. The application adjusts the display mode of the virtual picture according to the motion state of the user, and shields the interference of the virtual picture in time when the user needs to focus on the real environment.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a method, apparatus, electronic device, and storage medium for displaying virtual images. Background Technology

[0002] When using a head-mounted display (Vision Pro, VP), users can switch between virtual and real environments at any time to enhance their sensory experience. For example, if a user needs to focus on the real environment, they need to switch from the virtual image displayed on the current graphical user interface to the real environment. However, during this switching process, the display of the virtual image can interfere with the display of the real environment.

[0003] Existing technologies reduce interference from virtual images by having users actively adjust their display range or by placing the virtual image directly in a location within the real environment. However, these methods are insufficient to meet users' observation needs in unexpected situations and limit their ability to observe the real environment. Therefore, how to promptly block out virtual image interference to allow users to directly observe the real environment when they need to focus on it is a pressing problem in this field. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of the prior art by providing a method, apparatus, electronic device, and storage medium for displaying virtual images, thereby solving the problem in the prior art of how to promptly shield the interference of virtual images to directly observe the real environment when the user needs to focus on the real environment.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0006] In a first aspect, embodiments of this application provide a method for displaying a virtual image, which provides a graphical user interface through a head-mounted display device. The graphical user interface displays both a virtual image and a real-world image. The method includes:

[0007] The head-mounted display device acquires the user's posture information and gaze in real time.

[0008] The user's motion state is determined based on the posture information;

[0009] In response to the motion state switching from the first state to the second state and the gaze leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

[0010] As one possible implementation, switching the display mode of the virtual image from the first display mode to the second display mode includes:

[0011] The virtual image is clustered and shrunk in the direction opposite to the direction of the line of sight, and the clustered and shrunk virtual image is displayed.

[0012] As one possible implementation, the method further includes:

[0013] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and the gaze focusing on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode;

[0014] In the third display mode, the virtual image covers a larger proportion of the real-world image than a first preset value. The third display mode includes: display area and display position.

[0015] As one possible implementation, the third display method is the preceding display method of the second display method.

[0016] As one possible implementation, the display position is determined through the following process:

[0017] Obtain the next line of sight that is focused on the virtual image;

[0018] The display position is determined based on the position corresponding to the next line of sight.

[0019] As one possible implementation, determining the user's motion state based on the posture information includes:

[0020] Determine whether the change in attitude information within the current motion period is less than a preset value;

[0021] If so, the user's current motion state is determined to be the first state; otherwise, the user's current motion state is determined to be the second state.

[0022] As one possible implementation, the attitude information includes at least one or more of the following: angular acceleration, acceleration;

[0023] Determining whether the change value of posture information within the motion period corresponding to the current moment is less than a preset value includes:

[0024] The angular acceleration and acceleration at each moment within the motion period are obtained, and the end time of the motion period is the current moment.

[0025] If the difference between the angular acceleration at each moment in the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the motion period corresponding to the current moment is less than the preset value.

[0026] Secondly, embodiments of this application provide a virtual display device that provides a graphical user interface via a head-mounted display device. The graphical user interface displays both virtual and real-world images. The device includes:

[0027] The acquisition module is used to acquire the user's posture information and gaze in real time through the head-mounted display device;

[0028] The determination module is used to determine the user's motion state based on the posture information;

[0029] The switching module is used to switch the display mode of the virtual screen from the first display mode to the second display mode in response to the motion state switching from the first state to the second state and the gaze leaving the virtual screen. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

[0030] As one possible implementation, the switching module is specifically used for:

[0031] The virtual image is clustered and shrunk in the direction opposite to the direction of the line of sight, and the clustered and shrunk virtual image is displayed.

[0032] As one possible implementation, the switching module is also used for:

[0033] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and the gaze focusing on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode;

[0034] In the third display mode, the virtual image covers a larger proportion of the real-world image than a first preset value. The third display mode includes: display area and display position.

[0035] As one possible implementation, the third display method is the preceding display method of the second display method.

[0036] As one possible implementation, the display position is determined through the following process:

[0037] Obtain the next line of sight that is focused on the virtual image;

[0038] The display position is determined based on the position corresponding to the next line of sight.

[0039] As one possible implementation, the determining module is specifically used for:

[0040] Determine whether the change in attitude information within the current motion period is less than a preset value;

[0041] If so, the user's current motion state is determined to be the first state; otherwise, the user's current motion state is determined to be the second state.

[0042] As one possible implementation, the attitude information includes at least one or more of the following: angular acceleration, acceleration; the determining module is specifically used for:

[0043] The angular acceleration and acceleration at each moment within the motion period are obtained, and the end time of the motion period is the current moment.

[0044] If the difference between the angular acceleration at each moment in the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the motion period corresponding to the current moment is less than the preset value.

[0045] Thirdly, embodiments of this application provide an electronic device, including: a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the virtual screen display method as described in any of the first aspects above.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the virtual screen display method as described in any of the first aspects above.

[0047] The virtual screen display method, apparatus, electronic device, and storage medium according to embodiments of this application acquire user posture information and gaze in real time through a head-mounted display device; determine the user's motion state based on the posture information; and switch the display mode of the virtual screen from a first display mode to a second display mode in response to the motion state switching from a first state to a second state and the gaze leaving the virtual screen. The coverage ratio of the virtual screen to the real-world screen in the first display mode is different from that in the second display mode. According to embodiments of this application, the first state and the second state respectively indicate the user's state when the user's posture is constant or undergoing different degrees of change. Based on the user's posture information, it is determined whether the user's motion state is stable or unstable. When the user's state switches from a stable state to an unstable state, and the user's gaze is not focused on the virtual screen, it is determined that the user is undergoing a rapid posture change, thereby triggering the virtual screen to enter a motion-assisted state and changing the display logic of the virtual screen, that is, switching the display mode of the virtual screen from the first display mode to the second display mode, thereby reducing the coverage ratio of the virtual screen to the real-world screen, so as to realize timely shielding of the interference of the virtual screen to directly observe the real environment when the user needs to pay attention to the real environment. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating a method for displaying a virtual screen according to an embodiment of this application is shown;

[0050] Figure 2 This illustration shows a schematic diagram of a virtual screen display switching interface provided in an embodiment of this application;

[0051] Figure 3 This illustration shows a schematic diagram of a switching interface for another virtual screen display method provided in an embodiment of this application;

[0052] Figure 4 A flowchart illustrating a method for determining motion state provided in an embodiment of this application is shown.

[0053] Figure 5 This illustration shows a schematic diagram of a visualization scheme for a virtual screen provided in an embodiment of this application;

[0054] Figure 6This illustration shows a schematic diagram of another virtual screen visualization scheme provided in an embodiment of this application;

[0055] Figure 7 A schematic diagram of the structure of a virtual screen display device provided in an embodiment of this application is shown;

[0056] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0058] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0059] To enable those skilled in the art to utilize the content of this application, and in conjunction with the specific application scenario "motion-assisted observation design based on rapid posture changes," the following implementation is provided. For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application primarily describes methods for displaying virtual images, it should be understood that this is merely an exemplary embodiment.

[0060] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0061] In existing technologies, the first method, where users actively adjust the display range and state of the virtual screen to proactively observe the real environment, is only applicable when users actively switch their observation focus. It struggles to meet users' needs for observing the real environment in unexpected situations, such as standing up / sitting down, accidentally falling, turning around to respond to someone's call, or needing to get up to retrieve something. The second method, where users can directly place the virtual screen in a specific location in the real environment, ensures that other areas of the user's field of vision are not obstructed during rapid posture changes. However, if the location where the virtual screen is placed requires observation of the real environment, it necessitates actively moving the virtual screen, thus limiting observation of the real environment.

[0062] To address the aforementioned technical issues, this application provides a method for displaying virtual images. Addressing the difficulty for users to quickly filter out interference from virtual images when using the Vision Pro device, this method proposes a motion-assisted observation design based on rapid user posture changes. This design addresses three aspects: the core judgment when the user undergoes rapid posture changes, the display logic after the virtual image enters motion-assisted mode, and the recognition of the virtual image exiting motion-assisted mode. This helps users reduce interference from virtual images when they need to focus on the real environment.

[0063] The virtual screen display method in one embodiment of this application can run on a local terminal device or a server. When the virtual screen display method runs on a server, the method can be implemented and executed based on a cloud interaction system, wherein the cloud interaction system includes a server and a client device.

[0064] In an optional implementation, various cloud applications, such as cloud gaming, can run under the cloud interaction system. Taking cloud gaming as an example, cloud gaming refers to a gaming method based on cloud computing. In the cloud gaming operating mode, the game program and the game screen presentation are separated. The storage and execution of the game loading method are completed on the cloud gaming server. The client device is used for data reception, transmission, and game screen presentation. For example, the client device can be a head-mounted display, but the information processing is handled by the cloud gaming server. When playing the game, the player operates the client device to send operation commands to the cloud gaming server. The cloud gaming server runs the game according to the operation commands, encodes and compresses game screen data, returns it to the client device via the network, and finally, the client device decodes and outputs the game screen.

[0065] In an optional implementation, taking a game as an example, the local terminal device stores the game program and is used to display the game screen. The local terminal device is used to interact with the player through a graphical user interface, that is, conventionally downloading, installing, and running the game program via an electronic device. The local terminal device can provide the graphical user interface to the player in various ways, such as rendering it on the terminal's display screen or providing it to the player through holographic projection. For example, the local terminal device can include a display screen and a processor. The display screen is used to display the graphical user interface, which includes the game screen, and the processor is used to run the game, generate the graphical user interface, and control the display of the graphical user interface on the display screen. In this embodiment, the local terminal device can be a head-mounted display device.

[0066] In one possible implementation, this application provides a method for displaying virtual images, which uses a head-mounted display device to provide a graphical user interface that displays both virtual and real-world images.

[0067] Figure 1 A flowchart illustrating a method for displaying a virtual screen according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, the method specifically includes the following steps:

[0068] S101. Acquire the user's posture information and gaze in real time through the head-mounted display device.

[0069] Optionally, the mixed reality system of the Vision Pro head-mounted display can detect head movements, assist in judging changes in head posture, and track eye movements to determine the user's gaze direction. Based on this, the Vision Pro's eye-tracking technology can determine the user's gaze position in real time. For example, by monitoring and capturing the user's eye position and movement direction in real time, and then using complex image processing and algorithms to determine the user's gaze focus, the user's gaze is obtained.

[0070] Optionally, based on the gyroscope, angular velocity sensor, accelerometer and other sensing devices and detection technologies equipped on the Vision Pro head-mounted display device, the detection data of the user's head movement and eye movement can be obtained to monitor the user's movement state, and then the user's real-time movement status can be calculated through the detection data to obtain the user's posture information.

[0071] S102. Determine the user's motion state based on the posture information.

[0072] Optionally, the user's motion state includes a first state and a second state. The first state refers to a stable and focused state where the user's motion is constant, meaning the user's posture remains unchanged or changes are small and relatively smooth. The second state, on the other hand, refers to a state where the user's posture is undergoing rapid changes, with the user's posture clearly changing rapidly.

[0073] Optionally, if the user's motion state is determined to be constant based on the user's posture information, then the user is judged to be in the first state described above. If the user's posture information is detected to be greater than a preset value, then the user is judged to be undergoing a rapid posture change, which means the user is currently in the second state described above.

[0074] S103. In response to the motion state switching from the first state to the second state and the gaze leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode.

[0075] The coverage ratio of the virtual image to the real-world image in the first display mode differs from that in the second display mode. For example, in the first display mode, the coverage ratio of the virtual image to the real-world image is greater than a first preset value, while in the second display mode, the coverage ratio is less than a second preset value.

[0076] Optionally, when the user is in the first state, if a rapid change in the user's posture is detected, it is determined that the user's motion state has changed, that is, the user's state is no longer stable, and the user changes from the first state to the second state. Based on this, in response to the user's motion state switching from the first state to the second state, the display mode of the virtual screen is changed from the first display mode to the second display mode.

[0077] For example, when the user's movement state is in the first state, the virtual image is in the first display mode, meaning the virtual image can be displayed normally and can cover the real-world image. However, when the user's movement state switches from the first state to the second state, meaning the user now needs to observe the real environment, it is necessary to reduce the interference of the virtual image on the real environment. At this time, the display mode of the virtual image is switched from the first display mode to the second display mode. Since the coverage ratio of the virtual image to the real-world image in the second display mode is smaller than that in the first display mode, the occlusion of the real-world image by the virtual image is reduced.

[0078] Based on this, according to the virtual screen display method provided in the embodiments of this application, the first state and the second state respectively indicate the user's state when the user's posture is constant or changes to different degrees. Based on the user's posture information, it is determined whether the user's motion state is stable or unstable. When the user's state changes from stable to unstable and the user's gaze is not focused on the virtual screen, it is determined that the user is undergoing a rapid posture change, thereby triggering the virtual screen to enter a motion assistance state and changing the display logic of the virtual screen, that is, switching the display mode of the virtual screen from the first display mode to the second display mode, thereby reducing the coverage ratio of the virtual screen on the real world screen, so as to realize that when the user needs to pay attention to the real environment, the interference of the virtual screen can be blocked in time to directly observe the real environment.

[0079] As one possible implementation, step S103 above switches the display mode of the virtual screen from the first display mode to the second display mode, including:

[0080] Cluster and shrink the virtual image in the direction opposite to the line of sight, and then display the clustered and shrunken virtual image.

[0081] Alternatively, clustering refers to reducing the display area and grouping virtual images together in the same location, such as shrinking the virtual images, storing them in a unified entry point, or folding the virtual images.

[0082] For example, refer to Figure 2 As shown, the clustering and shrinking direction of the virtual image is related to the user's line of sight; that is, the direction of the line of sight guides the subsequent adjustment direction of the virtual image. For example, the virtual image can be clustered and shrunk along the direction opposite to the direction of the line of sight. It should be noted that the clustering and shrinking direction of the virtual image does not necessarily have to be exactly the opposite of the direction of the line of sight. It can be that the user's movement direction is reversed, and the shrinking direction after the movement direction is reversed is used as the clustering direction of the virtual image. Specifically, the shrinking direction can be divided into 4x groups, where x = 1, 2, and clustered according to similar directions.

[0083] For example, refer to Figure 3 As shown, elements in the graphical user interface are grouped according to their distance from the user's line of sight in the opposite direction. Spatial partitioning techniques or geometric algorithms directly based on the user's line of sight can be used to efficiently cluster elements in the virtual screen. For each cluster, the size or visual representation of each element can be proportionally reduced based on its distance from the starting point of the line of sight in the opposite direction. It is important to note that the farther an element is from the starting point of the line of sight in the opposite direction, the greater the degree of shrinkage can be, to simulate the visual effect of "shrinking in the opposite direction of the line of sight."

[0084] Optionally, depending on the user's different operating states, the transparency of the virtual screen can be adjusted to avoid the virtual screen obscuring the real-world scene. The transparency of the virtual screen refers to its degree of opacity. For example, if the virtual screen's transparency is 0%, it means the virtual screen is completely opaque, and the content of the real-world scene covered by the virtual screen is completely invisible. Conversely, if the virtual screen's transparency is 100%, it means the virtual screen is completely transparent, and the content of the real-world scene covered by the virtual screen is fully visible, but the virtual screen itself is invisible.

[0085] Based on this, the method also includes:

[0086] In response to the movement state switching from the first state to the second state and the gaze leaving the virtual screen, the transparency of the virtual screen is adjusted to the preset value.

[0087] For example, when a rapid change in user posture is detected, and the user's gaze leaves the virtual screen—meaning the user needs to focus on the real-world environment—the transparency of the virtual screen can be adjusted from a third preset value to a fourth preset value, where the third preset value is less than the fourth preset value. The settings of the third and fourth preset values ​​can be customized based on the condition of not affecting the user's observation of the screen content. For instance, when the transparency of the virtual screen is at the third preset value, the visibility of the real-world screen should not affect the visibility of the virtual screen, ensuring the user can clearly see the content of the virtual screen. Similarly, when the transparency of the virtual screen is at the fourth preset value, the visibility of the virtual screen should not affect the visibility of the real-world screen, ensuring the user can clearly see the content of the real-world screen.

[0088] Based on the above, the transparency of the virtual image in the first display mode is different from that in the second display mode. Step S103, which switches the display mode of the virtual image from the first display mode to the second display mode, further includes:

[0089] The virtual image is clustered and shrunk in the direction opposite to the viewing direction, and its transparency is adjusted. Furthermore, the transparency of the virtual image in the first display mode is less than that in the second display mode.

[0090] For example, when a user's movement state is detected to switch from a first state to a second state, and the user's gaze leaves the virtual screen, the virtual screen is clustered and shrunk in the direction opposite to the user's gaze, reducing the display area of ​​the virtual screen while increasing its transparency. This solves the problem of virtual screen occlusion of the real-world scene when the user's posture changes significantly by altering the display position, display area, and transparency of the virtual screen.

[0091] Based on this, the display mode of the virtual screen is switched in response to changes in the user's posture, which facilitates the observation of the real scene after the user's posture changes, and ensures the safety of the user's behavior.

[0092] As a possible implementation, when users have a need to focus on the real-world environment, to avoid the problem of virtual images obscuring the real-world image, the method also includes:

[0093] In response to the motion state switching from the first state to the second state, based on the display information of the real-world screen, the idle display area in the graphical user interface is determined, and a virtual screen is displayed in the idle display area.

[0094] For example, an idle display area refers to an area in the graphical user interface that is not occupied by other important information or controls, i.e., an area without displayed content. When the user's movement state changes from a first state to a second state, i.e., when the user's posture changes significantly, the virtual image can be moved to this idle display area for display. Specifically, based on the display position coordinates of the real-world image in the graphical user interface, the area occupied by the real-world image in the graphical user interface is determined, and then it is determined whether there is displayed content in the display area of ​​the graphical user interface other than the occupied area. The display area without displayed content is designated as the idle display area.

[0095] Furthermore, based on the display position coordinates of the virtual image and the display position coordinates of the idle display area, the offset of the virtual image is determined, and the virtual image is moved to the idle display area for display based on the offset. During this process, if the current display area of ​​the virtual image is larger than the display area of ​​the idle display area, that is, the virtual image cannot be fully displayed in the idle display area, the display area of ​​the virtual image is adjusted until the virtual image can be fully displayed in the idle display area.

[0096] Based on this, by identifying an empty display area in the graphical user interface and moving the virtual image to that empty display area, this operation can not only avoid the problem of the virtual image obscuring the real-world image, but also simultaneously and completely display both the virtual image and the real-world image on the graphical user interface, thereby ensuring the integrity of the displayed content and improving the user's perception experience.

[0097] Figure 4 A flowchart illustrating a method for determining motion state according to an embodiment of this application is shown. As one possible implementation, refer to... Figure 4 As shown, step S102 above determines the user's motion state based on the posture information, specifically including the following steps:

[0098] S401. Determine whether the change value of the posture information within the motion period corresponding to the current moment is less than the preset value;

[0099] Optionally, the angular acceleration and acceleration at each moment during the motion period are obtained, and the end time of the motion period is the current time; if the difference between the angular acceleration at each moment during the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the attitude information in the motion period corresponding to the current time is less than the preset value.

[0100] For example, the attitude information includes at least one or more of the following: angular acceleration and acceleration. The preset value for angular acceleration is primarily determined based on the monitored pitch angle, yaw angle, and rotational angular acceleration of the user's head, while the preset value for acceleration is primarily referenced to the time it takes for a person to start walking from a standstill. Additionally, the attitude information may also include position change information, which includes the amount of position change between the user's position before and after movement.

[0101] For example, if a user is undergoing a rapid change in posture, the mixed reality system based on the Vision Pro head-mounted display can directly detect the direction and angle of the user's head movement, thus obtaining the user's post-movement orientation. Based on the user's initial gaze position and post-movement orientation, the user's movement direction and angle can be determined, such as how many degrees to rotate in a certain direction or how much distance to move in a certain direction. The user's movement direction and angle can also be used to guide the orientation information when clustering virtual images. Furthermore, the user's movement direction and angle can also be used to ensure that the clustered virtual images are far from the user's current gaze, meaning the distance between the clustered location of the virtual images and the user's initial gaze is greater than the distance the user's gaze has moved.

[0102] S402. If yes, then determine the user's current motion state as the first state; otherwise, determine the user's current motion state as the second state.

[0103] For example, if the difference between the angular acceleration at each moment during the motion period and the angular acceleration at the previous moment is less than a first preset value, and the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it indicates that the user's motion state is relatively constant, that is, the user's motion state at the current moment is determined to be a stable state. Conversely, if the difference between the angular acceleration at each moment during the motion period and the angular acceleration at the previous moment is greater than the first preset value, or the difference between the acceleration at each moment and the acceleration at the previous moment is greater than the second preset value, it indicates that the user is undergoing rapid posture changes, that is, the user's motion state at the current moment is determined to be an unstable state.

[0104] Accordingly, when the posture information is position change information, the magnitude of the user's posture change can be determined by comparing the amount of position change between the user's position before and after the movement with a preset threshold, thereby determining whether the user's motion state is in a stable or unstable state.

[0105] Based on this, the degree of change in the user's posture information can be used to determine whether the user's motion state is stable or unstable. This allows for the determination of how the virtual and real-world images are displayed in the graphical user interface, thus reducing the interference of the virtual images on the real-world images when the user is observing them.

[0106] As one possible implementation, the above-mentioned method for displaying virtual images also includes:

[0107] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and with the gaze focused on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode; wherein, in the third display mode, the coverage ratio of the virtual screen to the real world screen is greater than the first preset value, and the third display mode includes: display area and display position.

[0108] Optionally, in response to the user's motion state switching from a stable state to an unstable state, to avoid interference between the virtual image and the real-world image, the display mode of the virtual image can be switched from the first display mode to the second display mode, that is, the virtual image is displayed in clusters, thereby reducing the proportion of virtual image occlusion on the real-world image. Correspondingly, when the user's motion state returns from the unstable state to a stable state, and the user's gaze refocuses on the virtual image, the display mode of the virtual image can also be switched from the second display mode to the third display mode, that is, the virtual image is restored to display.

[0109] It should be noted that the third display mode is the preceding display mode of the second display mode. The third display mode can be the same as or different from the first display mode. However, what they have in common is that the virtual image can be displayed normally in both the first and third display modes. In the second display mode, the virtual image enters a motion-assisted observation state and will quickly cluster and collapse along the direction opposite to the user's line of sight.

[0110] For example, if the user's motion state recovers from an unstable state to a stable state, and the user's gaze refocuses on the virtual screen, the display state of the virtual screen can return to normal, that is, the clustered and shrunk virtual screen can be re-displayed. Specifically, the virtual screen can be re-displayed at the display position before the virtual screen was clustered and shrunk.

[0111] The determination of whether a user's motion state has returned to a stable state is also based on whether the user's posture has changed rapidly within a certain period of time. This period of time can be set to 2 seconds, and the specific value is not limited here. It can be set according to the accuracy of the pre-fetching.

[0112] It should be noted that a confirmation process is required for the user's gaze to refocus on the virtual screen. For example, the user's gaze can be observed to refocus on the virtual screen for z seconds, where z can take a value between 0.8 and 3.

[0113] For example, by switching the display mode of the virtual image from the second display mode to the third display mode, and after the user's gaze refocuses on the virtual image, it can be referred to... Figure 5 and Figure 6 The visualization scheme shown provides visual feedback on the virtual screen.

[0114] It should be noted that the Vision Pro head-mounted display supports head-locked content and world-locked content, while the embodiments of this application mainly focus on head-locked content, that is, the virtual screen and the user's head always remain relatively still.

[0115] Optionally, the display position is determined by the following process: obtaining the next line of sight of the focused virtual image; and determining the display position based on the position corresponding to the next line of sight.

[0116] For example, the next gaze following the one focused on the virtual image can still be obtained using the gaze-tracking technology of the Vision Pro head-mounted display. Based on this, the position corresponding to the next gaze is used as the display position of the virtual image. Before this, the content display at this position can also be pre-adjusted or optimized.

[0117] Based on this, according to the embodiments of this application, based on the user's needs and judgments regarding the motion assistance state in the Vision Pro head-mounted display device, and the adaptive display of the virtual image in the motion assistance state, the user can switch between the virtual image and the real-world image at any time. This is particularly convenient for users to observe the real scene without obstacles after rapid posture changes, ensuring the safety of the user's movement behavior. At the same time, the embodiments of this application meet the real-world usage needs of users, further improving the convenience of using the head-mounted display device in daily life.

[0118] Based on the same inventive concept, this application also provides a virtual screen display device corresponding to the virtual screen display method. Since the principle of solving the problem by the virtual screen display device in this application is similar to the virtual screen display method described above in this application, the implementation of the virtual screen display device can refer to the implementation of the virtual screen display method, and the repeated parts will not be described again.

[0119] Figure 7 A schematic diagram of a virtual screen display device according to an embodiment of this application is shown. (Refer to...) Figure 7 As shown, the virtual screen display device 700 includes: an acquisition module 701, a determination module 702, and a switching module 703, wherein:

[0120] The acquisition module 701 is used to acquire the user's posture information and gaze in real time through the head-mounted display device;

[0121] The determination module 702 is used to determine the user's motion state based on the posture information;

[0122] The switching module 703 is used to switch the display mode of the virtual screen from the first display mode to the second display mode in response to the motion state switching from the first state to the second state and the gaze leaving the virtual screen. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

[0123] Based on this, the virtual screen display device according to the embodiments of this application indicates the user's state when the user's posture is constant or changes to different degrees, respectively. Based on the user's posture information, it is determined whether the user's motion state is stable or unstable. When the user's state changes from stable to unstable and the user's gaze is not focused on the virtual screen, it is determined that the user is undergoing a rapid posture change, thereby triggering the virtual screen to enter a motion assistance state and changing the display logic of the virtual screen, that is, switching the display mode of the virtual screen from the first display mode to the second display mode, thereby reducing the coverage ratio of the virtual screen on the real world screen, so as to realize that when the user needs to pay attention to the real environment, the interference of the virtual screen can be blocked in time to directly observe the real environment.

[0124] In one possible implementation, the switching module 703 is specifically used for:

[0125] Cluster and shrink the virtual image in the direction opposite to the line of sight, and then display the clustered and shrunken virtual image.

[0126] In one possible implementation, the switching module 703 is further configured to:

[0127] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and with the gaze focused on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode;

[0128] In the third display mode, the virtual image covers a larger proportion of the real-world image than the first preset value. The third display mode includes the display area and the display position.

[0129] In one possible implementation, the third display mode is the preceding display mode of the second display mode.

[0130] In one possible implementation, the determining module 702 is further configured to:

[0131] To obtain the next line of sight that is focused on the virtual screen;

[0132] The display position is determined based on the position corresponding to the next line of sight.

[0133] In one possible implementation, the determining module 702 is specifically used for:

[0134] Determine whether the change in attitude information within the current motion period is less than a preset value;

[0135] If so, the user's current motion state is determined to be the first state; otherwise, the user's current motion state is determined to be the second state.

[0136] In one possible implementation, the attitude information includes at least one or more of the following: angular acceleration, acceleration; the determination module 702 is specifically used for:

[0137] Obtain the angular acceleration and acceleration at each moment during the motion period, with the end of the motion period being the current moment;

[0138] If the difference between the angular acceleration at each moment during the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the attitude information in the motion period corresponding to the current moment is less than the preset value.

[0139] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0140] This application also provides an electronic device 800, such as... Figure 8 The diagram shows the structure of an electronic device 800 provided in this application embodiment, including: a processor 801 and a memory 802. Optionally, it may also include a bus 803. The memory 802 stores machine-readable instructions executable by the processor 801. When the electronic device 800 is running, the processor 801 and the memory 802 communicate via the bus 803. When the machine-readable instructions are executed by the processor 801, the following processing is performed:

[0141] The user's posture and gaze information are acquired in real time through the head-mounted display device;

[0142] Determine the user's motion state based on posture information;

[0143] In response to the movement state switching from the first state to the second state and the gaze leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

[0144] In one possible implementation, when the processor 801 executes the instruction to switch the display mode of the virtual screen from the first display mode to the second display mode, it specifically performs the following:

[0145] Cluster and shrink the virtual image in the direction opposite to the line of sight, and then display the clustered and shrunken virtual image.

[0146] In one possible implementation, the processor 801 is further configured to:

[0147] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and with the gaze focused on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode;

[0148] In the third display mode, the virtual image covers a larger proportion of the real-world image than the first preset value. The third display mode includes the display area and the display position.

[0149] In one possible implementation, the third display mode is the preceding display mode of the second display mode.

[0150] In one possible implementation, when executing the instruction to determine the display position, the processor 801 specifically performs the following:

[0151] To obtain the next line of sight that is focused on the virtual screen;

[0152] The display position is determined based on the position corresponding to the next line of sight.

[0153] In one possible implementation, when the processor 801 executes the instruction to determine the user's motion state based on the attitude information, it specifically performs the following:

[0154] Determine whether the change in attitude information within the current motion period is less than a preset value;

[0155] If so, the user's current motion state is determined to be the first state; otherwise, the user's current motion state is determined to be the second state.

[0156] In one possible implementation, the attitude information includes at least one or more of the following: angular acceleration, acceleration; when the processor 801 executes an instruction to determine whether the change value of the attitude information during the motion period corresponding to the current moment is less than a preset value, it is specifically used for:

[0157] Obtain the angular acceleration and acceleration at each moment during the motion period, with the end of the motion period being the current moment;

[0158] If the difference between the angular acceleration at each moment during the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the attitude information in the motion period corresponding to the current moment is less than the preset value.

[0159] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps:

[0160] The user's posture and gaze information are acquired in real time through the head-mounted display device;

[0161] Determine the user's motion state based on posture information;

[0162] In response to the movement state switching from the first state to the second state and the gaze leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

[0163] In one possible implementation, when the processor executes the instruction to switch the display mode of the virtual image from a first display mode to a second display mode, it specifically performs the following:

[0164] Cluster and shrink the virtual image in the direction opposite to the line of sight, and then display the clustered and shrunken virtual image.

[0165] In one possible implementation, the processor is further configured to:

[0166] In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and with the gaze focused on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode;

[0167] In the third display mode, the virtual image covers a larger proportion of the real-world image than the first preset value. The third display mode includes the display area and the display position.

[0168] In one possible implementation, the third display mode is the preceding display mode of the second display mode.

[0169] In one possible implementation, when the processor executes the instruction to determine the display position, it specifically performs the following:

[0170] To obtain the next line of sight that is focused on the virtual screen;

[0171] The display position is determined based on the position corresponding to the next line of sight.

[0172] In one possible implementation, when the processor executes instructions to determine the user's motion state based on attitude information, it specifically performs the following:

[0173] Determine whether the change in attitude information within the current motion period is less than a preset value;

[0174] If so, the user's current motion state is determined to be the first state; otherwise, the user's current motion state is determined to be the second state.

[0175] In one possible implementation, the attitude information includes at least one or more of the following: angular acceleration, acceleration; when the processor 801 executes an instruction to determine whether the change value of the attitude information during the motion period corresponding to the current moment is less than a preset value, it is specifically used for:

[0176] Obtain the angular acceleration and acceleration at each moment during the motion period, with the end of the motion period being the current moment;

[0177] If the difference between the angular acceleration at each moment during the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the attitude information in the motion period corresponding to the current moment is less than the preset value.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0179] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for displaying a virtual image, characterized in that, The method includes providing a graphical user interface via a head-mounted display device, the graphical user interface displaying virtual images and real-world images, the method comprising: The head-mounted display device acquires the user's posture information and gaze in real time. Based on the posture information, the user's motion state is determined. The motion state includes a first state and a second state. The first state refers to the user being in a stable and focused state, and the second state refers to the user being in a state of rapid posture change. In response to the motion state switching from the first state to the second state and the gaze leaving the virtual screen, the display mode of the virtual screen is switched from the first display mode to the second display mode. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

2. The method for displaying virtual images according to claim 1, characterized in that, The step of switching the display mode of the virtual image from the first display mode to the second display mode includes: The virtual image is clustered and shrunk in the direction opposite to the direction of the line of sight, and the clustered and shrunk virtual image is displayed.

3. The method for displaying virtual images according to claim 1, characterized in that, Also includes: In response to the motion state switching from the second state to the first state, and the first state lasting for a preset duration, and the gaze focusing on the virtual screen, the display mode of the virtual screen is switched from the second display mode to the third display mode; In the third display mode, the virtual image covers a larger proportion of the real-world image than a first preset value. The third display mode includes: display area and display position.

4. The method for displaying a virtual image according to claim 3, characterized in that, The third display mode is the preceding display mode of the second display mode.

5. The method for displaying a virtual image according to claim 3, characterized in that, The display position is determined through the following process: Obtain the next line of sight that is focused on the virtual image; The display position is determined based on the position corresponding to the next line of sight.

6. The method for displaying a virtual image according to any one of claims 1-5, characterized in that, Determining the user's motion state based on the posture information includes: Determine whether the change in attitude information within the current motion period is less than a preset value; If so, the user's current motion state is determined to be the first state; otherwise, the user's current motion state is determined to be the second state.

7. The method for displaying a virtual image according to claim 6, characterized in that, The attitude information includes at least one or more of the following: angular acceleration, acceleration; Determining whether the change value of posture information within the motion period corresponding to the current moment is less than a preset value includes: The angular acceleration and acceleration at each moment within the motion period are obtained, and the end time of the motion period is the current moment. If the difference between the angular acceleration at each moment in the motion period and the angular acceleration at the previous moment is less than a first preset value, and / or the difference between the acceleration at each moment and the acceleration at the previous moment is less than a second preset value, then it is determined that the change value of the posture information in the motion period corresponding to the current moment is less than the preset value.

8. A display device for virtual images, characterized in that, A graphical user interface is provided via a head-mounted display device, the graphical user interface displaying virtual and real-world images, the device comprising: The acquisition module is used to acquire the user's posture information and gaze in real time through the head-mounted display device; The determination module is used to determine the user's motion state based on the posture information. The motion state includes a first state and a second state. The first state refers to the user being in a stable and focused state, and the second state refers to the user being in a state of rapid posture change. The switching module is used to switch the display mode of the virtual screen from the first display mode to the second display mode in response to the motion state switching from the first state to the second state and the gaze leaving the virtual screen. The coverage ratio of the virtual screen to the real world screen in the first display mode is different from that in the second display mode.

9. An electronic device, characterized in that, include: The device includes a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is in operation, are executed by the processor to perform the steps of the virtual screen display method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method for displaying a virtual screen as described in any one of claims 1 to 7.

Images