Display method, device and equipment of head-mounted device and medium

Abstract

This application discloses a display method, apparatus, device, and medium for a head-mounted device, relating to the field of head-mounted device technology. The method includes: acquiring the gaze state of a wearer of the head-mounted device for each display area, the gaze state including one of being gazed upon, about to gaze upon, or ignoring the area; the head-mounted device including at least two display areas; acquiring display data sent by at least two terminal devices respectively; and performing any of the following storage scheduling operations based on the gaze state: first, storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state, and scheduling display data from the storage spaces according to their scheduling priority; second, storing the display data sent by the at least two terminal devices into matching storage spaces, and scheduling display data from the storage spaces according to the gaze state; and performing a display operation on the scheduled display data according to the gaze state.

Description

Technical Field

[0001] This application relates to the field of head-mounted device technology, and more specifically, to a display method for a head-mounted device, a display device for a head-mounted device, a head-mounted device, and a computer-readable storage medium. Background Technology

[0002] With the development of technology and the economy, smart head-mounted devices (such as AR / MR / VR / XR glasses) have been used more and more widely. Currently, smart head-mounted devices are generally divided into two forms: integrated and separate.

[0003] Smart head-mounted devices can perform streaming functions. That is, at least one terminal device (also known as a streaming device) sends display data (also known as streaming data) running in the foreground and / or background to the smart head-mounted device, which then displays the data sent by the terminal device.

[0004] However, when at least two terminal devices send display data to the smart head-mounted device—for example, when both terminal device A and terminal device B send display data to the smart head-mounted device—if terminal device A sends its display data to the smart head-mounted device before terminal device B, but the wearer is currently viewing the display area used to display the data sent by terminal device B, there is a situation where the display area being viewed may not yet have had time to display the data sent by terminal device B. In other words, there is a display delay issue with the display area the wearer is currently viewing. This degrades the user experience. Summary of the Invention

[0005] One objective of this application is to provide a new technical solution for head-mounted device displays.

[0006] According to a first aspect of this application, a display method for a head-mounted device is provided, comprising:

[0007] The gaze state of the wearer of the head-mounted device for each display area is obtained, and the gaze state includes one of being gazed at, about to gaze at, and ignoring; the head-mounted device includes at least two display areas.

[0008] Acquire display data sent by at least two terminal devices;

[0009] Based on the gaze state, a storage scheduling operation is performed, the storage scheduling operation including any of the following:

[0010] The first step involves storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state, and scheduling the display data from the storage spaces according to the scheduling priority of the storage spaces.

[0011] The second step involves storing the display data sent by the at least two terminal devices into matching storage spaces, and scheduling the display data from the storage spaces according to the gaze state.

[0012] Based on the gaze state, a display operation is performed on the scheduled display data.

[0013] Optionally, storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state includes:

[0014] When the gaze state of the display area is gaze, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the first scheduling priority;

[0015] When the gaze state of the display area is about to be gazed, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the second scheduling priority;

[0016] When the gaze state of the display area is ignored, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the third scheduling priority.

[0017] Wherein, the first scheduling priority is greater than the second scheduling priority, and the second scheduling priority is greater than the third scheduling priority.

[0018] Optionally, scheduling display data from the storage space based on the gaze state includes:

[0019] Based on the gaze state, the scheduling priority among the at least two storage spaces is determined;

[0020] Display data is scheduled from the storage space according to the scheduling priority of the storage space.

[0021] Optionally, scheduling display data from the storage space according to the scheduling priority of the storage space includes:

[0022] Data is scheduled from the storage space according to the data scheduling ratio corresponding to the scheduling priority of the storage space.

[0023] Alternatively, display data can be scheduled from the storage space in sequence according to the scheduling priority of the storage space.

[0024] Optionally, the method further includes:

[0025] The data scheduling ratio is determined based on the scheduling priority of the storage space and the type of display data sent by the terminal device corresponding to the display area.

[0026] Optionally, performing a display operation on the scheduled display data based on the gaze state includes:

[0027] When the gaze state of the display area is in gaze or about to gaze, the operation of displaying the display data sent by the terminal device corresponding to the scheduled display area is performed in the display area;

[0028] If the gaze state of the display area is not gazed, the display data sent by the terminal device corresponding to the scheduled display area is discarded.

[0029] Optionally, obtaining the gaze state of the wearer of the head-mounted device for each display area includes:

[0030] Obtain head posture data of the wearer of the head-mounted device;

[0031] Based on the head posture data, the gaze state of the wearer of the head-mounted device for each display area is determined.

[0032] According to a second aspect of this application, a display device for a head-mounted device is provided, comprising:

[0033] The first acquisition module is used to acquire the gaze state of the wearer of the head-mounted device for each display area. The gaze state includes one of being gazed at, about to gaze at, and ignoring. The head-mounted device includes at least two display areas.

[0034] The second acquisition module is used to acquire display data sent by at least two terminal devices respectively;

[0035] A storage scheduling module is configured to perform a storage scheduling operation based on the gaze state, wherein the storage scheduling operation includes any one of the following;

[0036] The first step involves storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state, and scheduling the display data from the storage spaces according to the scheduling priority of the storage spaces.

[0037] The second step involves storing the display data sent by the at least two terminal devices into matching storage spaces, and scheduling the display data from the storage spaces according to the gaze state.

[0038] The display module is used to perform display operations on the scheduled display data according to the gaze state.

[0039] According to a third aspect of this application, a head-mounted device is provided, the head-mounted device comprising the means as described in the second aspect; or,

[0040] The head-mounted device includes a memory and a processor, the memory for storing computer instructions, and the processor for retrieving the computer instructions from the memory to perform a display method of the head-mounted device as described in any one of the first aspects.

[0041] According to a fourth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a display method for a head-mounted device according to any one of the first aspects.

[0042] This application provides a display method for a head-mounted device, comprising: acquiring the gaze state of a wearer of the head-mounted device for each display area, the gaze state including one of being gazed upon, about to be gazed upon, and ignoring the display area; the head-mounted device including at least two display areas; acquiring display data sent by at least two terminal devices respectively; and performing any of the following storage scheduling operations according to the gaze state: first, storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state, and scheduling display data from the storage spaces according to the scheduling priority of the storage spaces; second, storing the display data sent by the at least two terminal devices into matching storage spaces, and scheduling display data from the storage spaces according to the gaze state; and performing a display operation on the scheduled display data according to the gaze state. Based on this method, timely display of data can be achieved for the display areas being gazed upon by the wearer. This improves the user experience.

[0043] Other features and advantages of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the present application and, together with their description, serve to explain the principles of the present application.

[0045] Figure 1 This is a framework of the hardware configuration of a head-mounted device for implementing a display method for a head-mounted device according to an embodiment of this application. Figure 1 ;

[0046] Figure 2 This is a flowchart illustrating a method for implementing a display on a head-mounted device according to an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of a scenario for implementing a display method for a head-mounted device according to an embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the structure of a display device for implementing a head-mounted device according to an embodiment of this application;

[0049] Figure 5 This is a framework of the hardware configuration of a head-mounted device for implementing a display method for a head-mounted device according to an embodiment of this application. Figure 2 . Detailed Implementation

[0050] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0051] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0052] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0053] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0054] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0055] Figure 1 This is a framework of the hardware configuration of a head-mounted device for implementing a display method for a head-mounted device according to an embodiment of this application. Figure 1 The head-mounted device 1000 can be exemplified as AR / MR / XR / VR glasses, AR / MR / XR / VR helmets, etc. Furthermore, the head-mounted device 1000 can be an integrated head-mounted device or a separate head-mounted device. For a separate head-mounted device, it includes a computing unit (commonly referred to as a PUCK) and a head-mounted display device communicatively connected to the computing unit. The computing unit performs data processing, and the head-mounted display device displays the data.

[0056] The head-mounted device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, and a sensing device 1900, etc. The processor 1100 may be a central processing unit (CPU), a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touchscreen, a keyboard, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800. The sensing device 1900 may be, for example, an IMU sensor, used to collect head posture data of the wearer of the head-mounted device.

[0057] Despite Figure 1 The head-mounted device 1000 shows multiple devices, but this application may only involve some of them. For example, the head-mounted device 1000 may only involve the memory 1200 and the processor 1100.

[0058] In the embodiments applied in this application, the memory 1200 of the head-mounted device 1000 is used to store instructions for controlling the processor 1100 to execute the display method of the head-mounted device provided in the embodiments of this application.

[0059] In the above description, those skilled in the art can design instructions based on the scheme disclosed in this application. How the instructions control the processor to operate is well known in the art, and therefore will not be described in detail here.

[0060] Figure 2 This is a flowchart illustrating a method for implementing a display on a head-mounted device according to an embodiment of this application. The method is applied to a head-mounted device, and the application scenario is as follows: at least two terminal devices are communicatively connected to the head-mounted device; for any terminal device, based on the communication connection with the head-mounted device, the terminal device sends display data to the head-mounted device, which is display data running in the foreground or background of the terminal device; the head-mounted device includes a display screen, which includes at least two display areas, one of which displays the processed display data sent by the designated terminal device.

[0061] In one embodiment of this application, the terminal device, based on its communication connection with the head-mounted device, can send display data to the head-mounted device via streaming. Therefore, the terminal device can also be referred to as a streaming device.

[0062] In one example, such as Figure 3As shown, taking terminal devices A, B, and C as examples, and the head-mounted device including display areas 1, 2, and 3 as examples, the display data sent by terminal device A is displayed in display area 1 of the head-mounted device after being rendered and processed by the head-mounted device; the display data sent by terminal device B is displayed in display area 2 of the head-mounted device after being rendered and processed by the head-mounted device; and the display data sent by terminal device C is displayed in display area 3 of the head-mounted device after being rendered and processed by the head-mounted device.

[0063] like Figure 2 As shown, the display method of the head-mounted device provided in this application embodiment includes the following steps S210 to S240.

[0064] Step S210: Obtain the gaze state of the wearer of the head-mounted device for each display area.

[0065] The gaze state includes one of the following: gaze in progress, about to gaze, or ignoring. It is understood that the display area for an ignoring gaze state can be at least one.

[0066] In this embodiment, "about to gaze" specifically refers to preparing to gaze. "Gaze in progress" specifically refers to currently gazing. "Ignoring" specifically refers to any gaze state other than gazing in progress or about to gaze.

[0067] Understandably, the gaze state of the wearer of the head-mounted device towards each display area is used to describe which display area the wearer is currently looking at, which display area they are about to look at, and which display areas they are ignoring.

[0068] In one embodiment of this application, step S210 can be specifically implemented by the following steps S211 to S212.

[0069] Step S210: Obtain head posture data of the wearer of the head-mounted device.

[0070] In one embodiment of this application, an IMU sensor can be installed on the head-mounted device, and the sensing data collected by the IMU sensor can be used as the posture data of the wearer of the head-mounted device.

[0071] Step S211: Based on head pose data, determine the gaze state of the wearer of the head-mounted device for each display area.

[0072] In this embodiment, the wearer's gaze direction and head movement trend can be determined based on head posture data. Based on this, in one embodiment of this application, the gaze state of the display area pointed to by the gaze direction is determined as "gazing". The gaze state of the display areas adjacent to the gazed display area along the head movement trend direction is determined as "about to gaze". The gaze state of the display areas other than the gazed and about to gaze display areas is determined as "ignored".

[0073] Furthermore, in another embodiment of this application, the wearer's gaze state towards each display area can be more accurately determined by combining the gaze direction and head movement trend with the head movement speed.

[0074] Step S220: Obtain display data sent by at least two terminal devices respectively.

[0075] In this embodiment, after obtaining the display data sent by at least two terminal devices respectively based on the above step S220, the obtained display data sent by at least two terminal devices is stored and scheduled according to the following step S230.

[0076] Step S230: Perform a storage scheduling operation based on the gaze state.

[0077] In this embodiment, the storage scheduling operation specifically refers to the operation of storing the display data sent by the terminal device into the storage space, and scheduling the stored display data from the storage space. The display method of the head-mounted device provided in this embodiment includes the following two storage scheduling operations. The first storage scheduling operation includes the following steps S2311 and S2312.

[0078] Step S2311: Based on the gaze state, store the display data sent by at least two terminal devices into matching storage spaces respectively.

[0079] In one embodiment of this application, the aforementioned storage space may specifically be a storage queue. Of course, it may also be other types of storage space.

[0080] In this embodiment, different storage spaces correspond to different and fixed scheduling priorities. Based on this, the above step S2311 can be specifically implemented through the following steps S2311-1 to S2311-3.

[0081] Step S2311-1: When the gaze state of the display area is gazed, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the first scheduling priority.

[0082] Step S2311-2: When the gaze state of the display area is about to be gazed, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the second scheduling priority.

[0083] Step S2311-3: When the gaze state of the display area is ignored, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the third scheduling priority.

[0084] Among them, the first scheduling priority is greater than the second scheduling priority, and the second scheduling priority is greater than the third scheduling priority.

[0085] In this embodiment, when the viewing state of the display area is "attention," the display data sent by the terminal device corresponding to the display area is stored in the storage space with the highest scheduling priority, i.e., the storage space with the first scheduling priority. When the viewing state of the display area is "about to be viewed," the display data sent by the terminal device corresponding to the display area is stored in the storage space with the second highest scheduling priority, i.e., the storage space with the second scheduling priority. And when the viewing state of the display area is "ignored," the display data sent by the terminal device corresponding to the display area is stored in the storage space with the lowest scheduling priority, i.e., the storage space with the third scheduling priority.

[0086] Step S2312: Schedule display data from storage space according to the scheduling priority of storage space.

[0087] In one embodiment of this application, step S2312 can be implemented by scheduling display data only from the storage space with the first scheduling priority. Based on this, it can be ensured that the display area under gaze can display data in a timely manner, thus avoiding the display delay problem of the display area being viewed by the wearer.

[0088] In one embodiment of this application, the specific implementation of step S2312 above can also be as follows: step S2312-1.

[0089] Step S2312-1: According to the scheduling priority of the storage space, the display data is scheduled from the storage space in turn.

[0090] In this embodiment, step S2312-1 is specifically implemented as follows: Display data in the storage space with the first scheduling priority is prioritized, followed by display data in the storage space with the second scheduling priority, and finally display data in the storage space with the third scheduling priority. Based on this, it can be ensured that the display data in the storage space with the first scheduling priority is given priority for rendering and other processing to be displayed in the corresponding display area. Furthermore, the display data in the storage space with the second scheduling priority is immediately processed for rendering and other processing to be displayed in the corresponding display area. Finally, the display data in the storage space with the third scheduling priority is processed for rendering and other processing to be displayed in the corresponding display area. This means that, based on step S2312-1, it can be ensured that the display area under gaze can display data in a timely manner, thus avoiding the display delay problem that exists in the display area that the wearer is currently viewing. Furthermore, when the wearer subsequently gazes at the display area they are about to gaze at, the display area they are about to gaze at can also display data in a timely manner. This also avoids the display delay problem that exists in the display area that the wearer is about to view.

[0091] In one embodiment of this application, the specific implementation of step S2312 above can also be as follows: step S2312-2.

[0092] Step S2312-2: According to the data scheduling ratio corresponding to the scheduling priority of the storage space, schedule display data from the storage space respectively.

[0093] In this embodiment, the data scheduling ratio specifically refers to the proportion of data scheduled from storage spaces with different scheduling priorities in a single operation.

[0094] In one embodiment of this application, the data scheduling ratio can be set based on experience. The setting principle is: the data scheduling ratio of the storage space corresponding to the data sent by the terminal device corresponding to the displayed area being viewed is the highest, followed by the data scheduling ratio of the storage space corresponding to the data sent by the terminal device corresponding to the displayed area being viewed, and the data scheduling ratio of the storage space corresponding to the data sent by the terminal device corresponding to the ignored displayed area is the lowest.

[0095] In this embodiment, the amount of display data that the head-mounted device can schedule at one time is fixed, for example, 100 frames of image data can be scheduled at one time. However, the scheduling requirement corresponding to the gaze area is usually lower than the amount of display data that the head-mounted device can schedule at one time. For example, the scheduling requirement for the gaze area is 80 frames of image data scheduled each time. Based on this, while scheduling display data that meets the scheduling requirement of the gaze area from the storage space with the first scheduling priority, display data from the storage space with the second scheduling priority can also be scheduled simultaneously. Alternatively, display data from the storage space with the second scheduling priority and the storage space with the third scheduling priority can also be scheduled simultaneously. For example, in one scheduling, 80 frames of image data are scheduled from the storage space with the first scheduling priority, and 20 frames of image data are scheduled from the storage space with the second scheduling priority. In this way, the display data in the gaze area can be updated to a certain extent when the wearer subsequently gazes at the gaze area.

[0096] It should be noted that, based on the above embodiments, in order to further avoid the situation where display data in the corresponding display area is delayed due to untimely scheduling of display data in the storage space, display data that should have been scheduled but was not can be deleted from the storage space. For example, if the head-mounted device can schedule 100 frames of image data at a time, based on the above steps S2312-2, 80 frames of image data are scheduled from the storage space with the first scheduling priority, and 20 frames of image data are scheduled from the storage space with the second scheduling priority. At the same time, the 20 frames of image data that are temporally consecutive with the 80 frames of image data already scheduled are deleted from the storage space with the first scheduling priority. And, the 80 frames of image data that are temporally consecutive with the 20 frames of image data already scheduled are deleted from the storage space with the second scheduling priority.

[0097] In another embodiment of this application, the display method of the head-mounted device provided in this application embodiment further includes a step of personalized determination of the data scheduling ratio in step S2312-2 above, namely the following step S2312-21.

[0098] Steps S2312-21: Determine the data scheduling ratio based on the scheduling priority of the storage space and the type of display data sent by the terminal device corresponding to the display area.

[0099] In this embodiment, the type of display data sent by the terminal is used to describe the application type corresponding to the display data. When the application type is different, the scheduling requirements for the corresponding display area of ​​the terminal device sending the display data are different. Based on this, in one embodiment of this application, the data scheduling ratio of the storage space with the first scheduling priority is determined as the ratio of the amount of data that can at least satisfy the scheduling requirements of the display area under the gaze to the amount of display data that can be scheduled at one time. The data scheduling ratio of the storage space with the second scheduling priority is determined as the difference between 1 and the data scheduling ratio of the storage space with the first scheduling priority. And, the data scheduling ratio of the storage space with the third scheduling priority is determined as 0.

[0100] Based on the above steps S2312-21, the display method for head-mounted devices provided in this application embodiment also provides a way to flexibly determine the data scheduling ratio of storage space according to the type of display data sent by the terminal device.

[0101] In this embodiment, the second storage scheduling operation in step S230 above includes the following steps S2321 and S2322.

[0102] Step S2321: Store the display data sent by at least two terminal devices into matching storage spaces respectively.

[0103] In this embodiment, the display data sent by a terminal device corresponds to a fixed storage space.

[0104] In one example, the display data 'a' sent by terminal device A always corresponds to storage space 1, the display data 'b' sent by terminal device B always corresponds to storage space 2, and the display data 'c' sent by terminal device C always corresponds to storage space 3.

[0105] Step S2322: Based on the gaze state, schedule display data from the storage space.

[0106] In this embodiment, step S2322 can be specifically implemented through the following steps S2322-1 and S2322-2.

[0107] Step S2322-1: Determine the scheduling priority between at least two storage spaces based on the gaze state.

[0108] In this embodiment, step S322-1 is specifically implemented as follows: the storage space corresponding to the display data sent by the terminal device corresponding to the display area in the gaze state is gazed is determined as the storage space with the highest scheduling priority, i.e., the storage space with the first scheduling priority; the storage space corresponding to the display data sent by the terminal device corresponding to the display area in the gaze state is about to be gazed is determined as the storage space with the second highest scheduling priority, i.e., the storage space with the second scheduling priority; and the storage space corresponding to the display data sent by the terminal device corresponding to the display area in the gaze state is ignored is determined as the storage space with the lowest scheduling priority, i.e., the storage space with the third scheduling priority.

[0109] Step S2322-2: Schedule display data from storage space according to the scheduling priority of storage space.

[0110] In this embodiment, step S2322-2 can be implemented by either step S2312-1 or step S2312-2. Further details will not be provided in this embodiment.

[0111] Step S240: Perform a display operation on the scheduled display data according to the gaze state.

[0112] In one embodiment of this application, step S240 can be specifically implemented through the following steps S241 and S242.

[0113] Step S241: When the gaze state of the display area is gaze or about to gaze, perform the operation of displaying the display data sent by the terminal device corresponding to the scheduled display area in the display area.

[0114] Step S242: If the gaze state of the display area is not gazed, discard the display data sent by the terminal device corresponding to the scheduled display area.

[0115] In this embodiment, the operation of displaying the display data in the display area specifically includes performing data processing operations such as rendering on the display data, and then displaying the processed data in the matching display area.

[0116] In this embodiment, when the gaze state of the display area is "attention" or "about to be attention," the operation of displaying the data sent by the terminal device corresponding to the scheduled display area is performed on the display area. This ensures that the data displayed in the currently viewed display area is updated in a timely manner. This avoids the display delay problem that may occur in the display area the wearer is currently viewing. Simultaneously, it also ensures that the data displayed in the display area about to be viewed is updated to a certain extent. This improves the user experience. Furthermore, it avoids the display delay problem that may occur in the display area the wearer is about to view.

[0117] Simultaneously, when the viewing state of the display area is "ignored," the display data sent by the terminal device corresponding to the scheduled display area is discarded. Based on this, since the wearer ignores the display area and will not look at it again, not performing display operations on the display data sent by the terminal device corresponding to the scheduled display area will not affect the wearer's viewing experience. Thus, discarding the display data sent by the terminal device corresponding to the ignored display area reduces the computational overhead of the head-mounted device. Furthermore, discarding the display data sent by the terminal device corresponding to the ignored display area also avoids storing outdated display data in the corresponding storage space. This prevents the problem of data delay when the wearer subsequently looks at the ignored display area.

[0118] Based on the above, this application provides a display method for a head-mounted device, comprising: acquiring the gaze state of a wearer of the head-mounted device for each display area, the gaze state including one of being gazed upon, about to be gazed upon, and ignoring the area; the head-mounted device including at least two display areas; acquiring display data sent by at least two terminal devices respectively; and performing any of the following storage scheduling operations according to the gaze state: first, storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state, and scheduling display data from the storage spaces according to the scheduling priority of the storage spaces; second, storing the display data sent by the at least two terminal devices into matching storage spaces, and scheduling display data from the storage spaces according to the gaze state; and performing a display operation on the scheduled display data according to the gaze state. Based on this method, timely display of data can be achieved for the display areas being gazed upon by the wearer. This improves the user experience.

[0119] Based on the above, in one example of this application, the terminal devices are terminal device A, terminal device B, and terminal device C, and the head-mounted device includes display areas display area one, display area two, and display area three. Display data sent by terminal device A is displayed in display area one of the head-mounted device after rendering and other processing; display data sent by terminal device B is displayed in display area two of the head-mounted device after rendering and other processing; and display data sent by terminal device C is displayed in display area three of the head-mounted device after rendering and other processing. The gaze state of display area two is "gazing," the gaze state of display area three is "about to gaze," and the gaze state of display area one is "ignored." The scheduling priority of storage space 1 is a fixed first scheduling priority, the scheduling priority of storage space 2 is a fixed second scheduling priority, and the scheduling priority of storage space 3 is a fixed third scheduling priority. Figure 3As shown, based on the display method of the head-mounted device provided in this application embodiment, display data a obtained from terminal device A is stored in storage space 3, display data obtained from terminal device B is stored in storage space 1, and display data obtained from terminal device C is stored in storage space 2. Further, display data b is scheduled from storage space 1, display data c is scheduled from storage space 2, and display data a is scheduled from storage space 3 in sequence. Based on this, the head-mounted device first processes display data b (rendering and other processing) and displays it on the screen, then processes display data c (rendering and other processing) and displays it on the screen, and finally processes display data a (rendering and other processing) and displays it on the screen. In this way, the display area that the wearer is currently looking at is prioritized over the display area that the wearer is about to look at, and the display area that the wearer is about to look at is prioritized over the display area that the wearer is ignoring. This enables timely display of data on the display area that the wearer is looking at. This improves the user experience.

[0120] This application also provides a display device 400 for a head-mounted device, such as... Figure 4 As shown, it includes:

[0121] The first acquisition module 410 is used to acquire the gaze state of the wearer of the head-mounted device for each display area. The gaze state includes one of being gazed at, about to gaze at, and ignoring. The head-mounted device includes at least two display areas.

[0122] The second acquisition module 420 is used to acquire display data sent by at least two terminal devices respectively;

[0123] The storage scheduling module 430 is configured to perform a storage scheduling operation based on the gaze state, wherein the storage scheduling operation includes any one of the following:

[0124] The first step involves storing the display data sent by the at least two terminal devices into matching storage spaces according to the gaze state, and scheduling the display data from the storage spaces according to the scheduling priority of the storage spaces.

[0125] The second step involves storing the display data sent by the at least two terminal devices into matching storage spaces, and scheduling the display data from the storage spaces according to the gaze state.

[0126] Display module 440 is used to perform display operations on the scheduled display data according to the gaze state.

[0127] This device enables real-time data display on the area of ​​view that the wearer is looking at. This improves the user experience.

[0128] In one embodiment of this application, the storage scheduling module 430 is specifically used for:

[0129] When the gaze state of the display area is gaze, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the first scheduling priority;

[0130] When the gaze state of the display area is about to be gazed, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the second scheduling priority;

[0131] When the gaze state of the display area is ignored, the display data sent by the terminal device corresponding to the display area is stored in the storage space with the third scheduling priority.

[0132] Wherein, the first scheduling priority is greater than the second scheduling priority, and the second scheduling priority is greater than the third scheduling priority.

[0133] In one embodiment of this application, the storage scheduling module 430 is specifically used for:

[0134] Based on the gaze state, the scheduling priority among the at least two storage spaces is determined;

[0135] Display data is scheduled from the storage space according to the scheduling priority of the storage space.

[0136] In one embodiment of this application, the storage scheduling module 430 is specifically used for:

[0137] Data is scheduled from the storage space according to the data scheduling ratio corresponding to the scheduling priority of the storage space.

[0138] Alternatively, display data can be scheduled from the storage space in sequence according to the scheduling priority of the storage space.

[0139] In one embodiment of this application, the display device 400 of the head-mounted device provided in this application embodiment further includes:

[0140] The determining module is used to determine the data scheduling ratio based on the scheduling priority of the storage space and the type of display data sent by the terminal device corresponding to the display area.

[0141] In one embodiment of this application, the display module 440 is specifically used for:

[0142] When the gaze state of the display area is in gaze or about to gaze, the operation of displaying the display data sent by the terminal device corresponding to the scheduled display area is performed in the display area;

[0143] If the gaze state of the display area is not gazed, the display data sent by the terminal device corresponding to the scheduled display area is discarded.

[0144] In one embodiment of this application, the first acquisition module 410 is used to acquire head posture data of the wearer of the head-mounted device;

[0145] Based on the head posture data, the gaze state of the wearer of the head-mounted device for each display area is determined.

[0146] This application also provides a head-mounted device, which includes a display device 400 as described in any of the head-mounted devices above.

[0147] Or, such as Figure 5 As shown, the head-mounted device 500 includes a memory 510 and a processor 520. The memory 510 is used to store computer instructions, and the processor 520 is used to retrieve the computer instructions from the memory 510 to execute the display method of the head-mounted device as described in any of the above method embodiments.

[0148] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the display method of the head-mounted device according to any one of the above method embodiments.

[0149] This application may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this application.

[0150] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0151] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0152] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0153] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0154] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0155] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be well known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0157] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this application is defined by the appended claims.

Claims

1. A display method of a head-mounted device, characterized by, The method comprises: obtaining a gaze state of a wearer of the head-mounted device for each display region, the gaze state comprising one of gazing, about to gaze and not gazing, the head-mounted device comprising at least two display regions; obtaining display data respectively transmitted by at least two terminal devices; performing a storage scheduling operation according to the gaze state, the storage scheduling operation comprising: storing the display data respectively transmitted by the at least two terminal devices into a matching storage space according to the gaze state, and scheduling display data from the storage space according to a scheduling priority of the storage space; performing a display operation on the scheduled display data according to the gaze state; wherein the scheduling display data from the storage space according to the scheduling priority of the storage space comprises: scheduling display data from the storage space according to a data scheduling ratio corresponding to the scheduling priority of the storage space, the data scheduling ratio being a ratio of the amount of data scheduled from the storage spaces of different scheduling priorities in turn.

2. The method of claim 1, wherein, The storing the display data respectively transmitted by the at least two terminal devices into a matching storage space according to the gaze state comprises: in a case where the gaze state of the display region is gazing, storing the display data transmitted by the terminal device corresponding to the display region into a storage space of a first scheduling priority; in a case where the gaze state of the display region is about to gaze, storing the display data transmitted by the terminal device corresponding to the display region into a storage space of a second scheduling priority; in a case where the gaze state of the display region is not gazing, storing the display data transmitted by the terminal device corresponding to the display region into a storage space of a third scheduling priority; wherein the first scheduling priority is greater than the second scheduling priority, and the second scheduling priority is greater than the third scheduling priority.

3. The method of claim 1, wherein, The method further comprises: determining the data scheduling ratio according to the scheduling priority of the storage space and a type of the display data transmitted by the terminal device corresponding to the display region.

4. The method of claim 1, wherein, The performing a display operation on the scheduled display data according to the gaze state comprises: in a case where the gaze state of the display region is gazing or about to gaze, performing an operation of displaying the display data transmitted by the terminal device corresponding to the display region on the display region; in a case where the gaze state of the display region is not gazing, discarding the display data transmitted by the terminal device corresponding to the display region.

5. The method of claim 1, wherein, The obtaining a gaze state of a wearer of the head-mounted device for each display region comprises: obtaining head pose data of the wearer of the head-mounted device; determining the gaze state of the wearer of the head-mounted device for each display region according to the head pose data.

6. A display device of a head-mounted device, characterized by, The method comprises: a first obtaining module, configured to obtain a gaze state of a wearer of the head-mounted device for each display region, the gaze state comprising one of gazing, about to gaze and not gazing, the head-mounted device comprising at least two display regions; a second obtaining module, configured to obtain display data respectively transmitted by at least two terminal devices; a storage scheduling module, configured to perform a storage scheduling operation according to the gaze state, the storage scheduling operation comprising: storing the display data respectively transmitted by the at least two terminal devices into a matching storage space according to the gaze state, and scheduling display data from the storage space according to a scheduling priority of the storage space; The storage scheduling module is configured to perform a storage scheduling operation according to the gaze state, and the storage scheduling operation comprises: storing display data transmitted by the at least two terminal devices into matched storage spaces respectively according to the gaze state, and scheduling display data from the storage spaces according to scheduling priorities of the storage spaces. The display module is configured to perform a display operation on the scheduled display data according to the gaze state. The storage scheduling module is specifically configured to: schedule display data from the storage spaces according to data scheduling proportions corresponding to the scheduling priorities of the storage spaces, and the data scheduling proportions are proportions of amounts of data scheduled from the storage spaces in different scheduling priorities in sequence.

7. A head-mounted device, characterized by The head-mounted device comprises the apparatus of claim 6; or The head-mounted device comprises a memory and a processor, the memory is configured to store computer instructions, and the processor is configured to call the computer instructions from the memory to perform the display method of the head-mounted device according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer program is stored on the head-mounted device and is configured to implement the display method of the head-mounted device according to any one of claims 1-5 when executed by the processor.

Images