Display method, head-mounted display device, electronic device and storage medium

Abstract

The invention discloses a display method, a head-mounted display device, an electronic device and a storage medium, and relates to the technical field of augmented reality, and the disclosed display method comprises the steps: determining time delay information according to the time difference between receiving and forwarding of communication data; determining predicted on-screen time according to the time delay information; and outputting a display picture by predicting the on-screen time. The time delay information is determined according to the time difference between receiving and forwarding of the communication data, then the display picture is output according to the predicted on-screen time output by the time delay information, it can be ensured that the actual display time of the display picture is consistent with the expected display time, the display picture is prevented from being displayed in advance or in a delayed mode, and then the display effect is improved.

Description

Display methods, head-mounted display devices, electronic devices, and storage media Technical Field

[0001] This application relates to the field of extended reality technology, and more particularly to a display method, head-mounted display device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of technology, extended reality (XR) technology, including augmented reality (AR), virtual reality (VR), and mixed reality (MR), is gradually changing people's lifestyles with its unique charm.

[0003] However, there is a time delay in the process of generating and displaying the image in XR devices. If the image is not displayed at the expected actual screen time, it may cause the image to be displayed earlier or later, affecting the display effect. Summary of the Invention

[0004] The main objective of this application is to provide a display method, head-mounted display device, electronic device, and storage medium, which aims to solve the technical problem that the process from the generation to the display of the display image in the prior art has a time delay, resulting in the display image being displayed too early or too late, thus affecting the display effect.

[0005] To achieve the above objectives, this application provides a display method, the display method comprising:

[0006] Determine the delay information based on the time difference between when communication data is received and when it is forwarded;

[0007] The predicted on-screen time is determined based on the latency information;

[0008] The output display screen is generated based on the predicted on-screen time.

[0009] In one embodiment, the latency information includes rendering latency information and display latency information, and the step of determining the latency information based on the time difference between the reception and forwarding of communication data includes:

[0010] The system receives communication data sent by a computing device, which carries the rendering latency information. The rendering latency information is determined by the time difference between the data in the computing device being received and forwarded by the encoding module.

[0011] The display delay information is determined based on the time difference between the reception and forwarding of the communication data.

[0012] Extract the rendering latency information from the communication data.

[0013] In one embodiment, the display latency information includes communication latency information and decoding latency information, and the step of determining the display latency information based on the time difference between the reception and forwarding of the communication data includes:

[0014] The communication delay information is determined based on the time difference between the reception and forwarding of the communication data by the communication module.

[0015] The decoding delay information is determined by the time difference between the reception and forwarding of the communication data by the decoding module.

[0016] In one embodiment, the step of determining the predicted on-screen time based on the delay information includes:

[0017] Obtain a preset deep learning model, which is a long short-term memory network model obtained by training based on training data consisting of parameters corresponding to multiple time delay information and multiple time indicators.

[0018] The initial time index is obtained by the preset deep learning model based on the communication latency information, the decoding latency information, and the rendering latency information.

[0019] The initial time index is rounded to obtain the target time index;

[0020] The predicted on-screen time is determined based on the target time metric and the current display refresh rate.

[0021] In one embodiment, the step of obtaining an initial time index based on the communication latency information, the decoding latency information, and the rendering latency information using the preset deep learning model includes:

[0022] Obtain the training data of the preset deep learning model, wherein the training data includes a first type of training data corresponding to the communication latency information, a second type of training data corresponding to the decoding latency information, and a third type of training data corresponding to the rendering latency information;

[0023] The communication delay information is standardized and normalized based on the mean and variance of the first type of training data to obtain the communication delay parameters;

[0024] The decoding delay information is standardized and normalized based on the mean and variance of the second type of training data to obtain decoding delay parameters;

[0025] The rendering latency information is standardized and normalized based on the mean and variance of the third type of training data to obtain the rendering latency parameters.

[0026] The communication latency parameter, the decoding latency parameter, and the rendering latency parameter are input into the preset deep learning model to obtain the initial time index.

[0027] In one embodiment, after the step of rounding the initial time index to obtain the target time index, the method further includes:

[0028] Determine whether the target time indicator has reached a preset threshold.

[0029] If so, the target time indicator is set as the preset indicator threshold.

[0030] In one embodiment, the step of outputting the display image based on the predicted screen time includes:

[0031] Generate attitude prediction data based on the predicted screen display time;

[0032] The attitude prediction data is sent to a computing device so that the computing device can feed back rendering data based on the attitude prediction data.

[0033] The output display screen is based on the rendered data.

[0034] Furthermore, to achieve the above objectives, this application also proposes a head-mounted display device, the head-mounted display device comprising:

[0035] The time analysis module is used to determine delay information based on the time difference between the reception and forwarding of communication data;

[0036] The time prediction module is used to determine the predicted on-screen time based on the delay information;

[0037] The display module is used to output the display screen based on the predicted on-screen time.

[0038] In addition, to achieve the above objectives, this application also proposes an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the display method as described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the display method described above.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] This application determines latency information based on the time difference between the reception and forwarding of communication data; determines the predicted on-screen time based on the latency information; and outputs the display screen based on the predicted on-screen time. Because this application uses the time difference between the reception and forwarding of communication data to determine latency information, and then uses the predicted on-screen time output from the latency information to output the display screen, it can ensure that the actual display time of the display screen is consistent with the expected display time, avoiding the display screen from being displayed too early or too late, thereby improving the display effect. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 is a flowchart illustrating the first embodiment of the method of this application;

[0045] Figure 2 is a flowchart illustrating the second embodiment of the method of this application;

[0046] Figure 3 is an example diagram of the modular structure of the split AR device of this application;

[0047] Figure 4 is a flowchart illustrating the third embodiment of the method of this application;

[0048] Figure 5 is a schematic diagram of the basic architecture of the preset deep learning model in the third embodiment of the method shown in this application;

[0049] Figure 6 is a schematic diagram of the module structure of the head-mounted display device of this application;

[0050] Figure 7 is a schematic diagram of the hardware operating environment involved in the display method in the embodiments of this application.

[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0053] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of this application embodiment is: the head-mounted display device determines the delay information based on the time difference between the reception and forwarding of communication data; determines the predicted screen display time based on the delay information; and outputs the display screen based on the predicted screen display time.

[0055] In this embodiment, for ease of description, the following description uses a head-mounted display device as the execution subject.

[0056] Existing XR devices suffer from a time delay between the generation and display of the image. When the image does not appear at the expected actual display time, it may appear earlier or later, affecting the display effect. For example, if the display time obtained using attitude prediction data is 10 seconds without delay, and the optimal display time to reduce the impact of delay is 20 seconds, then using the existing method, the image may appear 10 seconds earlier, resulting in a poor display effect.

[0057] This application provides a solution in which a head-mounted display device uses the time difference between the reception and forwarding of communication data to determine the delay information, and then uses the predicted on-screen time output by the delay information to output the display image. Therefore, it can ensure that the actual display time of the display image is consistent with the expected display time, avoid the display image being displayed too early or too late, and thus improve the display effect.

[0058] Based on this, this application provides a display method. Referring to FIG1, FIG1 is a flowchart of the first embodiment of the display method of this application.

[0059] In this embodiment, the display method includes steps S10 to S30:

[0060] Step S10: Determine the delay information based on the time difference between the reception and forwarding of communication data.

[0061] It should be noted that the executing entity in this embodiment can be a head-mounted display device with screen display, network communication, and program execution functions that applies XR technology, such as an AR device, VR device, MR device, etc., or an XR electronic device capable of achieving the above functions. The following description uses an AR device as an example, specifically a head-mounted display device within an AR device, to illustrate this embodiment and the subsequent embodiments.

[0062] This split-type AR device also includes a computing device. This computing device can be a device with display, network communication, and program execution capabilities, and applies rendering technology, such as a mobile phone, tablet, or personal computer. It can also be other electronic devices or terminals that perform the same or similar functions. The head-mounted display in the AR device can form a split structure with the computing device. The head-mounted display and the computing device can interact with each other. The computing device uses the data sent by the head-mounted display to render data, generating rendered data. The head-mounted display can then correct the rendered data sent by the computing device and output the displayed image on the screen.

[0063] Understandably, the aforementioned communication data could be data exchanged between the head-mounted display device and the computing device.

[0064] It should be noted that the above latency information can be information characterizing the network latency of a split AR device consisting of a head-mounted display device and a computing device.

[0065] In practical implementation, when a user uses a split-type AR device consisting of a head-mounted display and a computing device, the head-mounted display can establish a communication connection with the computing device. Subsequently, the head-mounted display and computing device can interact and transmit communication data to each other. The head-mounted display can determine the time when each of its internal modules receives the communication data and the time when it forwards the communication data to the next module, determining the latency information on the head-mounted display side based on the time difference between the reception and forwarding times. Correspondingly, the computing device can determine the time when each of its internal modules receives the communication data and the time when it forwards the communication data to the next module or the head-mounted display, determining the latency information on the computing device side based on the time difference between the reception and forwarding times, and then carrying the latency information on the computing device side with the communication data feedback to the head-mounted display. The head-mounted display can use its latency information and the latency information on the computing device side as the overall latency information of the split-type AR device.

[0066] Step S20: Determine the predicted screen display time based on the delay information.

[0067] It should be noted that the predicted on-screen time mentioned above represents the optimal time for the displayed image to finally appear on the screen of the head-mounted display device. Outputting the displayed image on the display interface when the preset on-screen time is reached can reduce the impact of network latency and prevent the displayed image from appearing too early or too late.

[0068] In practical implementation, the optimal on-screen time under different latency information can be predetermined, and a mapping relationship can be established between the optimal on-screen time and the corresponding latency information. After determining the current latency information of the split AR device, the aforementioned head-mounted display device can determine the corresponding optimal on-screen time from this preset relationship as the current predicted on-screen time.

[0069] Step S30: Output the display screen based on the predicted on-screen time.

[0070] In its implementation, the aforementioned head-mounted display device can interact with a computing device to render virtual content, generate corresponding display images, and output the display images when the predicted on-screen time is reached.

[0071] In one feasible implementation, step S30 includes steps S301 to S303:

[0072] Step S301: Generate attitude prediction data based on the predicted on-screen time.

[0073] It should be noted that the aforementioned attitude prediction data can be data characterizing the position and orientation of the head-mounted display device in three-dimensional space. Specifically, the attitude prediction data involves the accurate position of the head-mounted display device relative to its surrounding environment, as well as attitude changes such as tilt, rotation, and yaw of the head-mounted display device.

[0074] In its implementation, the aforementioned head-mounted display device can track and determine its attitude in space using built-in sensors, such as an inertial measurement unit (IMU), generating attitude prediction data. The sensors measure the acceleration and angular velocity of the head-mounted display device, and algorithms process this data to estimate its position and attitude. Using the predicted on-screen time as input, attitude prediction data containing the predicted on-screen time is generated to ensure that the head-mounted display device can accurately overlay the displayed image onto the user's real-world field of view when the predicted on-screen time is reached. A feasible predicted on-screen time is the next vertical synchronization cycle of the head-mounted display device plus N vertical synchronization cycles. This vertical synchronization cycle can be the time interval between two consecutive refreshes of the head-mounted display device's screen, and can be the reciprocal of the screen's refresh rate. For example, on a 60Hz screen, the vertical synchronization cycle can be 1s / 60, or 16.67ms.

[0075] Step S302: The attitude prediction data is sent to the computing device so that the computing device can feed back rendering data based on the attitude prediction data.

[0076] In its implementation, the aforementioned head-mounted display device acquires attitude prediction data, including the predicted on-screen time, at fixed intervals (e.g., 1 second or 5 seconds) and sends this attitude prediction data to the computing device. Upon receiving the attitude prediction data, the computing device uses it to render a virtual image. The rendering process includes calculating the correct position, size, and orientation of the virtual image within the user's field of vision using the attitude prediction data. After completing the calculation, rendering data is generated and sent to the head-mounted display device.

[0077] Step S303: Output the display screen based on the rendered data.

[0078] In practice, the head-mounted display device can compensate and correct the rendered data again based on the posture prediction data to form the final display image. When the predicted on-screen time is reached, the display image is accurately output on the display interface for the user to view.

[0079] The display method provided in this embodiment determines delay information based on the time difference between the reception and forwarding of communication data; determines the predicted on-screen time based on the delay information; and outputs the display screen based on the predicted on-screen time. Because this application uses the time difference between the reception and forwarding of communication data to determine delay information, and then uses the predicted on-screen time output from the delay information to output the display screen, it can ensure that the actual display time of the display screen is consistent with the expected display time, avoiding the display screen from being displayed too early or too late, thereby improving the display effect.

[0080] Based on the first embodiment of this application, a second embodiment of this application is proposed. In this second embodiment, any content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. Based on this, please refer to Figure 2, which is a flowchart illustrating the second embodiment of the method of this application.

[0081] In this embodiment, the latency information includes rendering latency information and display latency information, and step S10 may include steps S101 to S103:

[0082] Step S101: Receive communication data sent by the computing device carrying the rendering end latency information, wherein the rendering end latency information is determined by the time difference between the data in the computing device being received and forwarded by the encoding module.

[0083] It should be noted that the above rendering latency information can be information that characterizes the network latency of computing devices.

[0084] It is understandable that the aforementioned display latency information can be used to characterize the network latency of head-mounted display devices.

[0085] It should be noted that the above-mentioned encoding module can be a module in a computing device that has data encoding function.

[0086] In its implementation, after rendering the attitude prediction data, the computing device sends the rendered data to its internal encoding module. The encoding module encodes the rendered data. During this process, the computing device records the time when the rendered data is received by the encoding module and the time when the encoded data is forwarded. It then calculates the time difference between these two times to obtain the rendering latency information. The rendered data received and the forwarded encoded data can be used as communication data transmitted within the encoding module. The computing device can incorporate the rendering latency information into the encoded data and then send this encoded data as communication data to the head-mounted display device.

[0087] Step S102: Determine the display terminal delay information based on the time difference between the reception and forwarding of the communication data.

[0088] In a specific implementation, when the aforementioned head-mounted display device receives communication data from the computing device in its internal module, it can calculate the time difference between the moment when the internal module receives the communication data and the moment when it forwards the communication data to obtain the display latency information. At the same time, the head-mounted display device can extract the rendering latency information from the communication data and use the display latency information and the rendering latency information as the overall latency information of the split AR device.

[0089] In one feasible implementation, the display latency information includes communication latency information and decoding latency information, and step S102 includes steps S1021 to S1022:

[0090] Step S1021: Determine the communication delay information based on the time difference between the reception and forwarding of the communication data by the communication module.

[0091] It should be noted that the aforementioned communication latency information can be information characterizing the network latency of the communication module in a head-mounted display device. This communication module can be a module that interacts with a computing device.

[0092] It is understandable that the aforementioned decoding latency information can be information characterizing the network latency of the decoding module in the head-mounted display device. This decoding module can be a module in the head-mounted display device that has data decoding capabilities.

[0093] In its implementation, the aforementioned head-mounted display device can receive encoded data sent by the computing device via a communication module. After receiving the encoded data, the communication module can forward it to the next module. During this process, the head-mounted display device can record the time when the encoded data is received by the communication module and the time when it is forwarded, and calculate the time difference between these two times to obtain communication delay information. The encoded data received and forwarded by the communication module can be used as the communication data transmitted within the communication module.

[0094] Step S1022: Determine the decoding delay information based on the time difference between the reception and forwarding of the communication data by the decoding module.

[0095] In its implementation, the decoding module in the aforementioned head-mounted display device receives encoded data forwarded by the communication module, decodes the encoded data to obtain rendered data containing rendering latency information, and forwards this rendered data to the next module. During this process, the head-mounted display device records the time when the encoded data is received by the decoding module and the time when the rendered data is forwarded by the decoding module, and calculates the time difference between these two times to obtain decoding latency information. The encoded data received and the rendered data forwarded by the decoding module can then be used as the communication data transmitted within the decoding module.

[0096] Step S103: Extract the rendering end latency information from the communication data.

[0097] In a specific implementation, the Asynchronous Timewarp (ATW) module in the aforementioned head-mounted display device can receive rendering data transmitted by the decoding module. The head-mounted display device can obtain the rendering data in the ATW module and extract the rendering latency information contained in the rendering data, thereby obtaining communication latency information, decoding latency information, and rendering latency information. The sum of the latency corresponding to the communication latency information, decoding latency information, and rendering latency information is used as the latency information of the split AR device.

[0098] For ease of understanding, Figure 3 is used as a reference, but it does not limit the scope of this application. Figure 3 is an example diagram of the modular structure of the split AR device of this application. Figure 3 shows a feasible example structure and does not limit the scope of this application. In Figure 3, the split AR device includes a head-mounted display device and a computing device. The head-mounted display device includes a time analysis module, a communication receiving module, a decoding module, an ATW module, a posture generation module, a posture prediction module, and a display module. The computing device's application runtime environment includes a posture usage module. The computing device also includes an encoding module and a communication transmitting module. The communication receiving module of the head-mounted display device communicates with the communication transmitting module in the computing device via a network, involving protocols such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). The communication receiving module of the head-mounted display device can be used as the communication module described above.

[0099] As shown in Figure 3, the attitude generation module in the head-mounted display device generates attitude prediction data containing the predicted on-screen time. The attitude usage module retrieves the attitude prediction data from the attitude generation module at fixed intervals (e.g., 1 second or 5 seconds). The attitude usage module can also send the attitude prediction data to the attitude usage module in the computing device. The attitude usage module uses the attitude prediction data for rendering. After generating rendering data, the rendering data is encoded by the encoding module and then sent to the communication receiving module of the head-mounted display device via the communication sending module. The communication receiving module forwards the encoded data to the decoding module, which decodes the encoded data to obtain the rendering data, which is then sent to the ATW module. The time analysis module can extract the rendering latency information from the ATW module. During this process, the communication receiving module can add timestamps of its received and forwarded encoded data to the encoded data while forwarding it. The decoding module can add timestamps of its received and forwarded encoded data to the rendering time while decoding the encoded data. Therefore, the rendering time obtained after decoding by the decoding module includes the timestamps added by the communication receiving module and the timestamps added by the decoding module. The time analysis module can extract the timestamps added by the communication receiving module and the decoding module from the rendering data in the ATW module. It can determine the communication delay information based on the timestamp added by the communication receiving module and the decoding delay information based on the timestamp of the decoding module, thereby determining the delay information of the split AR device.

[0100] This embodiment receives communication data carrying rendering latency information sent by a computing device, extracts the rendering latency information from the communication data, determines the communication latency information based on the time difference between the communication data being received and forwarded by the communication module, and determines the decoding latency information based on the time difference between the communication data being received and forwarded by the decoding module. This accurately determines the latency information on both the computing device side and the head-mounted display device side, indirectly improving the display accuracy of the XR device.

[0101] Based on the second embodiment of this application, a third embodiment of this application is proposed. In the third embodiment of this application, content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, please refer to Figure 4, which is a flowchart illustrating the third embodiment of the method of this application.

[0102] In this embodiment, step S20 includes steps S201 to S204:

[0103] Step S201: Obtain a preset deep learning model, which is a long short-term memory network model obtained by training based on training data consisting of parameters corresponding to multiple time delay information and multiple time indicators.

[0104] It should be noted that the above time indicators can be parameters representing the number of vertical synchronization cycles.

[0105] In the specific implementation, referring to Figure 5, which is a schematic diagram of the basic architecture of the preset deep learning model in the third embodiment of the method of this application, X t For frame t input, h t For frame t output, X t-1 For frame t-1 input, h t-1 For frame t-1 output, X t+1 For frame t+1 input, h t+1 The output is for frame t+1. Taking frame t as the input, the forgetting gate in the Long Short-Term Memory (LSTM) model includes a left-hand σ function, which processes the input X at frame t. t The forgetting gate value is calculated by combining the hidden state from the previous time step. This value is between 0 and 1, where 0 represents complete forgetting and 1 represents complete retention. The input gate of the Long Short-Term Memory (LSTM) model includes a σ function and a tanh layer on the middle side. The σ function controls the input X of frame t. t The information needed to update the memory cells is used to create a new candidate value vector through a tanh layer, which may be added to the memory cells. The output gate of the Long Short-Term Memory (LSTM) model consists of a σ function on the right and a tanh layer. The output gate value is calculated by σ on the input Xt of frame t and the hidden state of the previous time step. This value is between 0 and 1, where 0 represents no output and 1 represents full output. Then, the cell state is processed by the tanh function, outputting a result between -1 and 1, which is multiplied by the output gate value to obtain the final output h of frame t. t The descriptions of frames t-1 and t+1 can be found in the description of frame t above, and will not be repeated here.

[0106] It should be understood that when constructing a pre-defined deep learning model based on a long short-term memory network model, the parameters corresponding to the latency information in the training data can be used as inputs for each frame, and the time metrics can be used as outputs for each frame.

[0107] Step S202: Obtain an initial time index based on the communication latency information, the decoding latency information, and the rendering latency information using the preset deep learning model.

[0108] In a specific implementation, the aforementioned head-mounted display device can use the currently determined communication latency information, decoding latency information, and rendering latency information as input to a preset deep learning model, and receive the initial time index output by the preset deep learning model based on the communication latency information, decoding latency information, and rendering latency information.

[0109] In one feasible implementation, step S202 includes steps S2021 to S2025:

[0110] Step S2021: Obtain the training data of the preset deep learning model. The training data includes a first type of training data corresponding to the communication latency information, a second type of training data corresponding to the decoding latency information, and a third type of training data corresponding to the rendering latency information.

[0111] It should be noted that the first type of training data mentioned above can be parameters corresponding to historical communication latency information generated before the current moment. The second type of training data mentioned above can be parameters corresponding to historical decoding latency information generated before the current moment. The third type of training data mentioned above can be parameters corresponding to historical rendering latency information generated before the current moment.

[0112] In a specific implementation, the aforementioned head-mounted display device can acquire locally stored historical data, and determine from the historical data the first type of training data corresponding to communication latency information, the second type of training data corresponding to decoding latency information, and the third type of training data corresponding to rendering latency information, and construct training data for a preset deep learning model based on the first type of training data, the second type of training data, and the third type of training data.

[0113] Step S2022: Standardize and normalize the communication delay information based on the mean and variance of the first type of training data to obtain communication delay parameters.

[0114] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the communication delay information to obtain the communication delay parameters:

[0115]

[0116] In the formula, x is the communication delay parameter. ij1 σ1 represents the communication delay information, μ1 represents the mean of each parameter in the first type of training data, and σ1 represents the variance of each parameter in the first type of training data.

[0117] Step S2023: Standardize and normalize the decoding delay information based on the mean and variance of the second type of training data to obtain decoding delay parameters.

[0118] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the decoding delay information to obtain the communication delay parameters:

[0119]

[0120] In the formula, x is the decoding delay parameter. ij2 To decode the latency information, μ2 is the mean of each parameter in the second type of training data, and σ2 is the variance of each parameter in the second type of training data.

[0121] Step S2024: Standardize and normalize the rendering latency information based on the mean and variance of the third type of training data to obtain the rendering latency parameters.

[0122] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the rendering latency information to obtain the communication latency parameters:

[0123]

[0124] In the formula, x is the rendering latency parameter. ij3 σ represents the rendering latency information, μ3 represents the mean of each parameter in the third type of training data, and σ3 represents the variance of each parameter in the third type of training data.

[0125] Step S2025: Input the communication latency parameter, the decoding latency parameter, and the rendering latency parameter into the preset deep learning model to obtain the initial time index.

[0126] In its specific implementation, the aforementioned head-mounted display device standardizes and normalizes the communication latency information, decoding latency information, and rendering latency information to obtain the corresponding communication latency parameters, decoding latency parameters, and rendering latency parameters. Then, it inputs these parameters into a preset deep learning model to obtain initial time metrics.

[0127] It should be understood that by standardizing and normalizing the communication latency information, decoding latency information, and rendering latency information, the consistency of the measurement of the three types of data can be ensured, and the training accuracy of the initial time index output by the preset deep learning model can be improved.

[0128] Step S203: Round the initial time index to obtain the target time index.

[0129] In the specific implementation, the target time index is used to characterize the number of vertical synchronization cycles. The target time index should be an integer. Therefore, after the head-mounted display device obtains the initial time index output by the preset deep learning model, it can round the initial time index to obtain the target time index.

[0130] In one feasible implementation, after step S203, steps S2031 to S2032 are further included:

[0131] Step S2031: Determine whether the target time indicator has reached the preset indicator threshold.

[0132] It should be noted that the aforementioned preset threshold values ​​can be used as calibration values ​​to determine whether the target time indicator is too large.

[0133] In a specific implementation, the aforementioned head-mounted display device can obtain the target time indicator through a preset deep learning model, and then compare the target time indicator with a preset indicator threshold to determine whether the target time indicator has reached the preset indicator threshold.

[0134] Step S2032: If yes, then set the target time indicator to the preset indicator threshold.

[0135] In practical implementation, when the aforementioned head-mounted display device detects that the target time indicator has reached a preset threshold, in order to avoid an excessively large target time indicator leading to a greater computational burden for subsequent predictions of the on-screen time, the value of the target time indicator can be reset to the preset threshold to limit the target time indicator. Conversely, when the target time indicator is detected to have not reached the preset threshold, the current value of the target time indicator can be retained for subsequent calculations of the predicted on-screen time.

[0136] Step S204: Determine the predicted on-screen time based on the target time indicator and the current display refresh rate.

[0137] In its implementation, the aforementioned head-mounted display device can calculate the reciprocal of the current display refresh rate to obtain the vertical synchronization period. Then, it uses the target time index as the target quantity and sets the predicted screen display time to the next vertical synchronization period plus the target number of vertical synchronization periods. For example, using vsync as the vertical synchronization period and N as the target time index, the predicted screen display time can be the next vsync of the head-mounted display device + N vsync cycles.

[0138] This embodiment uses a preset deep learning model with communication latency information, decoding latency information, and rendering latency information as model inputs to obtain an initial time index. At the same time, the initial time index is rounded to obtain a target time index. Thus, based on the target time index and the current display refresh rate, the on-screen time can be predicted more accurately, thereby improving the display accuracy of the head-mounted display device.

[0139] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method shown in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0140] This application also provides a head-mounted display device. Please refer to Figure 6, which is a schematic diagram of the module structure of the head-mounted display device of this application. In Figure 6, the head-mounted display device includes:

[0141] The time analysis module 10 is used to determine the delay information based on the time difference between the reception and forwarding of communication data.

[0142] The time prediction module 20 is used to determine the predicted screen display time based on the time delay information.

[0143] Display module 30 is used to output the display screen based on the predicted on-screen time.

[0144] The head-mounted display device provided in this application, employing the display method described in the above embodiments, can solve the technical problem in the prior art where the process from image generation to display is delayed, causing the image to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the display method provided in the above embodiments, and other technical features in the head-mounted display device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0145] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the display method in Embodiment 1 or 4 described above.

[0146] Referring to Figure 7, which is a schematic diagram of the hardware operating environment involved in the display method in this embodiment, the electronic device in this embodiment may be, but is not limited to, a split-type XR device combining a head-mounted display device such as AR glasses, VR glasses, or MR glasses with a terminal such as a mobile phone, laptop computer, in-vehicle terminal, or desktop computer. The electronic device shown in Figure 7 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0147] As shown in Figure 7, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. While electronic devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0148] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0149] The electronic device provided in this application, employing the display method described in the above embodiments, can solve the technical problem in the prior art where the process from generating to displaying a screen involves a time delay, causing the screen to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the display method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0150] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0151] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0152] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the display method in the above embodiments.

[0153] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0154] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0155] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0157] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0158] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described display method. This solves the technical problem in the prior art where the process from image generation to display of the screen involves a time delay, causing the screen to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the display method provided in the above embodiments, and will not be repeated here.

[0159] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A display method, characterized in that, The display method includes: determining delay information based on the time difference between the reception and forwarding of communication data; determining the predicted on-screen time based on the delay information; and outputting the display screen based on the predicted on-screen time.

2. The display method as described in claim 1, characterized in that, The latency information includes rendering latency information and display latency information. The step of determining the latency information based on the time difference between the reception and forwarding of communication data includes: receiving communication data sent by a computing device carrying the rendering latency information, wherein the rendering latency information is determined by the time difference between the reception and forwarding of data in the computing device by the encoding module; determining the display latency information based on the time difference between the reception and forwarding of the communication data; and extracting the rendering latency information from the communication data.

3. The display method as described in claim 2, characterized in that, The display latency information includes communication latency information and decoding latency information. The step of determining the display latency information based on the time difference between the reception and forwarding of the communication data includes: determining the communication latency information based on the time difference between the reception and forwarding of the communication data by the communication module; and determining the decoding latency information based on the time difference between the reception and forwarding of the communication data by the decoding module.

4. The display method as described in claim 3, characterized in that, The step of determining the predicted on-screen time based on the latency information includes: acquiring a preset deep learning model, wherein the preset deep learning model is a long short-term memory network model trained on training data consisting of parameters corresponding to multiple latency information and multiple time indicators; obtaining an initial time indicator based on the communication latency information, the decoding latency information, and the rendering latency information through the preset deep learning model; rounding the initial time indicator to obtain a target time indicator; and determining the predicted on-screen time based on the target time indicator and the current display refresh rate.

5. The display method as described in claim 4, characterized in that, The step of obtaining an initial time index based on the communication latency information, the decoding latency information, and the rendering latency information using the preset deep learning model includes: acquiring the training data of the preset deep learning model, wherein the training data includes a first type of training data corresponding to the communication latency information, a second type of training data corresponding to the decoding latency information, and a third type of training data corresponding to the rendering latency information; standardizing and normalizing the communication latency information according to the mean and variance of the first type of training data to obtain communication latency parameters; standardizing and normalizing the decoding latency information according to the mean and variance of the second type of training data to obtain decoding latency parameters; standardizing and normalizing the rendering latency information according to the mean and variance of the third type of training data to obtain rendering latency parameters; and inputting the communication latency parameters, the decoding latency parameters, and the rendering latency parameters into the preset deep learning model to obtain the initial time index.

6. The display method as described in claim 4, characterized in that, After the step of rounding the initial time index to obtain the target time index, the method further includes: determining whether the target time index reaches a preset index threshold; if so, setting the target time index as the preset index threshold.

7. The display method as described in claim 1, characterized in that, The step of outputting the display screen based on the predicted on-screen time includes: generating attitude prediction data based on the predicted on-screen time; sending the attitude prediction data to a computing device so that the computing device can feed back rendering data based on the attitude prediction data; and outputting the display screen based on the rendering data.

8. A head-mounted display device, characterized in that, The head-mounted display device includes: a time analysis module for determining delay information based on the time difference between the reception and forwarding of communication data; a time prediction module for determining the predicted on-screen time based on the delay information; and a display module for outputting the display screen based on the predicted on-screen time.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the display method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the display method as described in any one of claims 1 to 7.

Images