Display method, head-mounted display device, electronic device and storage medium

Abstract

The invention discloses a display method, head-mounted display equipment, electronic equipment and a storage medium, and relates to the technical field of augmented reality, the disclosed display method comprises the following steps: determining time difference information according to a time difference value when two adjacent frames of data are processed; determining predicted on-screen time according to the time difference information; and outputting a display picture by predicting the on-screen time. The time difference information is determined according to the time difference value when the two adjacent frames of data are processed, then the display picture is output according to the predicted on-screen time output by the time difference information, it can be ensured that the actual display time of the display picture is consistent with the expected display time, the display picture is prevented from being displayed in advance or in a delayed mode, and then the display effect is improved.

Description

Display methods, head-mounted display devices, electronic devices, and storage media Technical Field

[0001] This application relates to the field of extended reality technology, and more particularly to a display method, head-mounted display device, electronic device, and storage medium. Background Technology

[0002] With the rapid development of technology, extended reality (XR) technology, including augmented reality (AR), virtual reality (VR), and mixed reality (MR), is gradually changing people's lifestyles with its unique charm.

[0003] However, there is a time delay in the process of generating and displaying the image in XR devices. If the image is not displayed at the expected actual screen time, it may cause the image to be displayed earlier or later, affecting the display effect. Summary of the Invention

[0004] The main objective of this application is to provide a display method, head-mounted display device, electronic device, and storage medium, which aims to solve the technical problem that the process from the generation to the display of the display image in the prior art has a time delay, resulting in the display image being displayed too early or too late, thus affecting the display effect.

[0005] To achieve the above objectives, this application provides a display method, the display method comprising:

[0006] Time difference information is determined based on the time difference when two adjacent frames of data are processed.

[0007] The predicted screen display time is determined based on the time difference information;

[0008] The output display screen is generated based on the predicted on-screen time.

[0009] In one embodiment, the time difference information includes encoding time difference information, transmission time difference information, and display time difference information. The step of determining the time difference information based on the time difference when two adjacent frames of data are processed includes:

[0010] The system receives the encoding time difference information and the transmission time difference information sent by the computing device. The encoding time difference information is determined by the time difference between two adjacent frames of data in the computing device when they are processed by the encoding module, and the transmission time difference information is determined by the time difference between two adjacent frames of data in the computing device when they are processed by the communication transmission module.

[0011] The display time difference information is determined based on the time difference when two adjacent frames of data are processed.

[0012] In one embodiment, the display end time difference information includes reception time difference information and decoding time difference information. The step of determining the display end time difference information based on the time difference when two adjacent frames of data are processed includes:

[0013] The receiving time difference information is determined based on the time difference when two adjacent frames of data are processed by the communication receiving module.

[0014] The decoding time difference information is determined based on the time difference when two adjacent frames of data are processed by the decoding module.

[0015] In one embodiment, the step of determining the predicted on-screen time based on the time difference information includes:

[0016] Obtain a preset deep learning model, which is a long short-term memory network model obtained by training based on training data consisting of parameters corresponding to multiple time difference information and multiple time indicators.

[0017] The initial time index is obtained by the preset deep learning model based on the receiving time difference information, the decoding time difference information, the encoding time difference information, and the sending time difference information;

[0018] The initial time index is rounded to obtain the target time index;

[0019] The predicted on-screen time is determined based on the target time metric and the current display refresh rate.

[0020] In one embodiment, the step of obtaining an initial time index based on the reception time difference information, the decoding time difference information, the encoding time difference information, and the transmission time difference information using the preset deep learning model includes:

[0021] Obtain the training data of the preset deep learning model, wherein the training data includes a first type of training data corresponding to the received time difference information, a second type of training data corresponding to the decoded time difference information, a third type of training data corresponding to the encoded time difference information, and a fourth type of training data corresponding to the transmitted time difference information;

[0022] The receiving time difference information is standardized and normalized based on the mean and variance of the first type of training data to obtain the receiving time difference parameters;

[0023] The decoding time difference information is standardized and normalized based on the mean and variance of the second type of training data to obtain the decoding time difference parameters;

[0024] The encoded time difference information is standardized and normalized based on the mean and variance of the third type of training data to obtain the encoded time difference parameters;

[0025] The transmission time difference information is standardized and normalized based on the mean and variance of the fourth type of training data to obtain the transmission time difference parameters;

[0026] The receiving time difference parameter, the decoding time difference parameter, the encoding time difference parameter, and the transmitting time difference parameter are input into the preset deep learning model to obtain the initial time index.

[0027] In one embodiment, after the step of rounding the initial time index to obtain the target time index, the method further includes:

[0028] Determine whether the target time indicator has reached a preset threshold.

[0029] If so, the target time indicator is set as the preset indicator threshold.

[0030] In one embodiment, the step of outputting the display image based on the predicted screen time includes:

[0031] Generate attitude prediction data based on the predicted screen display time;

[0032] The attitude prediction data is sent to a computing device so that the computing device can feed back rendering data based on the attitude prediction data.

[0033] The output display screen is based on the rendered data.

[0034] Furthermore, to achieve the above objectives, this application also proposes a head-mounted display device, the head-mounted display device comprising:

[0035] The time analysis module is used to determine time difference information based on the time difference when two adjacent frames of data are processed;

[0036] The time prediction module is used to determine the predicted on-screen time based on the time difference information;

[0037] The display module is used to output the display screen based on the predicted on-screen time.

[0038] In addition, to achieve the above objectives, this application also proposes an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the display method as described above.

[0039] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the display method described above.

[0040] One or more technical solutions proposed in this application have at least the following technical effects:

[0041] This application determines time difference information based on the time difference between the processing of two adjacent frames of data; determines the predicted on-screen time based on the time difference information; and outputs the display screen based on the predicted on-screen time. This application uses the time difference between the processing of two adjacent frames of data to determine time difference information, and then uses the predicted on-screen time output by the time difference information to output the display screen. Therefore, it can ensure that the actual display time of the display screen is consistent with the expected display time, avoiding the display screen from being displayed too early or too late, thereby improving the display effect. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0043] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 is a flowchart illustrating the first embodiment of the method of this application;

[0045] Figure 2 is a flowchart illustrating the second embodiment of the method of this application;

[0046] Figure 3 is an example diagram of the modular structure of the split AR device of this application;

[0047] Figure 4 is a flowchart illustrating the third embodiment of the method of this application;

[0048] Figure 5 is a schematic diagram of the basic architecture of the preset deep learning model in the third embodiment of the method shown in this application;

[0049] Figure 6 is a schematic diagram of the module structure of the head-mounted display device of this application;

[0050] Figure 7 is a schematic diagram of the hardware operating environment involved in the display method in the embodiments of this application.

[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0053] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of this application embodiment is: the head-mounted display device determines the time difference information based on the time difference when two adjacent frames of data are processed; determines the predicted screen display time based on the time difference information; and outputs the display screen based on the predicted screen display time.

[0055] In this embodiment, for ease of description, the following description uses a head-mounted display device as the execution subject.

[0056] Existing XR devices suffer from a time delay between the generation and display of the image. When the image does not appear at the expected actual display time, it may appear earlier or later, affecting the display effect. For example, if the display time obtained using attitude prediction data is 10 seconds without delay, and the optimal display time to reduce the impact of delay is 20 seconds, then using the existing method, the image may appear 10 seconds earlier, resulting in a poor display effect.

[0057] This application provides a solution in which a head-mounted display device determines time difference information by using the time difference between the processing of two adjacent frames of data, and then uses the predicted on-screen time output by the time difference information to output the display image. Therefore, it can ensure that the actual display time of the display image is consistent with the expected display time, avoid the display image being displayed too early or too late, and thus improve the display effect.

[0058] Based on this, this application provides a display method. Referring to FIG1, FIG1 is a flowchart of the first embodiment of the display method of this application.

[0059] In this embodiment, the display method includes steps S10 to S30:

[0060] Step S10: Determine the time difference information based on the time difference when two adjacent frames of data are processed.

[0061] It should be noted that the executing entity in this embodiment can be a head-mounted display device with screen display, network communication, and program execution functions that applies XR technology, such as an AR device, VR device, MR device, etc., or an XR electronic device capable of realizing the above functions. The following description uses a head-mounted AR device as an example, employing a head-mounted display device within an AR device to illustrate this embodiment and the subsequent embodiments.

[0062] This split-type AR device also includes a computing device. This computing device can be a device with display, network communication, and program execution capabilities, and applies rendering technology, such as a mobile phone, tablet, or personal computer. It can also be other electronic devices or terminals that perform the same or similar functions. The head-mounted display in the AR device can form a split structure with the computing device. The head-mounted display and the computing device can interact with each other. The computing device uses the data sent by the head-mounted display to render data, generating rendered data. The head-mounted display can then correct the rendered data sent by the computing device and output the displayed image on the screen.

[0063] Understandably, the two adjacent frames mentioned above can be two temporally consecutive image frames used to achieve dynamic effects and interactions in AR scenes. When the image is displayed on the head-mounted display device, each frame is generated immediately after the previous frame.

[0064] In practical implementation, when a user uses a split-type AR device consisting of a head-mounted display and a computing device, the head-mounted display can establish a communication connection with the computing device. Then, the head-mounted display can interact with the computing device, transmitting image data frame by frame. The head-mounted display can record the time taken by each of its internal modules to process the previous frame of data, and the time taken by that internal module to process the next frame. It then calculates the difference between the time taken by the same internal module to process the previous frame and the time taken to process the next adjacent frame, using this difference as the time difference information on the head-mounted display side. Similarly, the computing device can record the time taken by each of its internal modules to process the previous frame of data, and the time taken by that internal module to process the next frame. It then calculates the difference between the time taken by the same internal module to process the previous frame and the next frame, using this difference as the time difference information on the computing device side. The head-mounted display and the computing device can then combine their time difference information into the overall time difference information for the split-type AR device.

[0065] It should be understood that the time difference between two consecutive frames of data processed by a split AR device is the total time required from processing the first frame of data to processing the second frame of data. This time difference information includes the time required for all steps such as data reading, processing, and possible storage or transmission. Therefore, this time difference information can characterize the network latency generated by the split AR device when processing these two frames of data.

[0066] Step S20: Determine the predicted screen display time based on the time difference information.

[0067] It should be noted that the predicted on-screen time mentioned above represents the optimal time for the displayed image to finally appear on the screen of the head-mounted display device. Outputting the displayed image on the display interface when the preset on-screen time is reached can reduce the impact of network latency and prevent the displayed image from appearing too early or too late.

[0068] In practical implementation, the optimal screen-on time under different time zone information can be predetermined, and a mapping relationship can be established between the optimal screen-on time and the corresponding time zone information. After determining the current time zone information of the split AR device, the head-mounted display device can determine the corresponding optimal screen-on time from the preset relationship as the current predicted screen-on time.

[0069] Step S30: Output the display screen based on the predicted on-screen time.

[0070] In its implementation, the aforementioned head-mounted display device can interact with a computing device to render virtual content, generate corresponding display images, and output the display images when the predicted on-screen time is reached.

[0071] In one feasible implementation, step S30 includes steps S301 to S303:

[0072] Step S301: Generate attitude prediction data based on the predicted on-screen time.

[0073] It should be noted that the aforementioned attitude prediction data can be data characterizing the position and orientation of the head-mounted display device in three-dimensional space. Specifically, the attitude prediction data involves the accurate position of the head-mounted display device relative to its surrounding environment, as well as attitude changes such as tilt, rotation, and yaw of the head-mounted display device.

[0074] In its implementation, the aforementioned head-mounted display device can track and determine its attitude in space using built-in sensors, such as an inertial measurement unit (IMU), generating attitude prediction data. The sensors measure the acceleration and angular velocity of the head-mounted display device, and algorithms process this data to estimate its position and attitude. Using the predicted on-screen time as input, attitude prediction data containing the predicted on-screen time is generated to ensure that the head-mounted display device can accurately overlay the displayed image onto the user's real-world field of view when the predicted on-screen time is reached. A feasible predicted on-screen time is the next vertical synchronization cycle of the head-mounted display device plus N vertical synchronization cycles. This vertical synchronization cycle can be the time interval between two consecutive refreshes of the head-mounted display device's screen, and can be the reciprocal of the screen's refresh rate. For example, on a 60Hz screen, the vertical synchronization cycle can be 1s / 60, or 16.67ms.

[0075] Step S302: The attitude prediction data is sent to the computing device so that the computing device can feed back rendering data based on the attitude prediction data.

[0076] In its implementation, the aforementioned head-mounted display device acquires attitude prediction data, including the predicted on-screen time, at fixed intervals (e.g., 1 second or 5 seconds) and sends this attitude prediction data to the computing device. Upon receiving the attitude prediction data, the computing device uses it to render a virtual image. The rendering process includes calculating the correct position, size, and orientation of the virtual image within the user's field of vision using the attitude prediction data. After completing the calculation, rendering data is generated and sent to the head-mounted display device.

[0077] Step S303: Output the display screen based on the rendered data.

[0078] In practice, the head-mounted display device can compensate and correct the rendered data again based on the posture prediction data to form the final display image. When the predicted on-screen time is reached, the display image is accurately output on the display interface for the user to view.

[0079] The display method provided in this embodiment determines time difference information based on the time difference between the processing of two adjacent frames of data; determines the predicted on-screen time based on the time difference information; and outputs the display screen based on the predicted on-screen time. This application uses the time difference between the processing of two adjacent frames of data to determine time difference information, and then uses the predicted on-screen time output by the time difference information to output the display screen. Therefore, it can ensure that the actual display time of the display screen is consistent with the expected display time, avoiding the display screen from being displayed too early or too late, thereby improving the display effect.

[0080] Based on the first embodiment of this application, a second embodiment of this application is proposed. In this second embodiment, any content that is the same as or similar to that in the first embodiment can be referred to the above description and will not be repeated hereafter. Based on this, please refer to Figure 2, which is a flowchart illustrating the second embodiment of the method of this application.

[0081] In this embodiment, the time difference information includes encoded time difference information, transmission time difference information, and display time difference information. Step S10 may include steps S101 to S102:

[0082] Step S101: Receive the encoding time difference information and the transmission time difference information sent by the computing device. The encoding time difference information is determined by the time difference between two adjacent frames of data in the computing device when they are processed by the encoding module, and the transmission time difference information is determined by the time difference between two adjacent frames of data in the computing device when they are processed by the communication transmission module.

[0083] It should be noted that the aforementioned encoding time difference information can be information characterizing the network latency of the encoding module in the computing device. This encoding module can be a module in the computing device that has data encoding capabilities.

[0084] Understandably, the aforementioned transmission time difference information can be used to characterize the network latency of the communication transmission module in the computing device. This communication transmission module can be a module that interacts with the head-mounted display device.

[0085] In its implementation, the computing device renders the attitude prediction data sent by the head-mounted display device. After generating rendered data, it sends this data to its internal encoding module. The encoding module encodes the rendered data and generates encoded data, which is then sent to the head-mounted display device via a communication module. The encoding module records the time taken to encode the previous frame of attitude prediction data and the time taken to encode the next adjacent frame. It then calculates the difference in time taken to process the two adjacent frames of attitude prediction data and uses this difference as the encoding time difference information. The communication module records the time taken to send the previous frame of encoded data to the head-mounted display device and the time taken to send the next adjacent frame of encoded data. It then calculates the difference in time taken to send the two adjacent frames of encoded data and uses this difference as the transmission time difference information.

[0086] Accordingly, the encoding module can add encoding time difference information to the next frame of encoded data, and the communication transmission module can add transmission time difference information to the next frame of encoded data, so that the next frame of encoded data sent by the computing device to the head-mounted display device through the communication transmission module contains encoding time difference information and transmission time difference information.

[0087] Step S102: Determine the display terminal time difference information based on the time difference when two adjacent frames of data are processed.

[0088] In a specific implementation, the aforementioned head-mounted display device can receive encoded data sent by the computing device through the communication sending module through its internal module, and record the time taken to process the previous frame of encoded data and the time taken to process the adjacent next frame of encoded data. Then, it can calculate the difference in the time taken to process the two adjacent frames of encoded data and use this difference as the display end time difference information.

[0089] In one feasible implementation, the display terminal time difference information includes received time difference information and decoded time difference information, and step S102 includes steps S1021 to S1022:

[0090] Step S1021: Determine the receiving time difference information based on the time difference between two adjacent frames of data when they are processed by the communication receiving module.

[0091] It should be noted that the aforementioned reception time difference information can be used to characterize the communication latency of the communication receiving module in the head-mounted display device. This communication receiving module can be a module that interacts with the communication sending module in the computing device.

[0092] Understandably, the aforementioned decoding time difference information can be used to characterize the network latency of the decoding module in a head-mounted display device. This decoding module can be a module in the head-mounted display device that has data decoding capabilities.

[0093] In a specific implementation, the communication receiving module of the aforementioned head-mounted display device can record the time it takes to send the previous frame of encoded data to the decoding module, and record the time it takes to send the adjacent next frame of encoded data to the decoding module. Then, it calculates the difference between the time it takes to send the two adjacent frames of encoded data to the decoding module, and uses this difference as the reception time difference information.

[0094] Step S1022: Determine the decoding time difference information based on the time difference between the processing of two adjacent frames of data by the decoding module.

[0095] In its implementation, the decoding module of the aforementioned head-mounted display device decodes the encoded data sent by the communication receiving module to obtain rendering data, and then forwards this rendering data to the Asynchronous Timewarp (ATW) module. During this process, the decoding module records the time it takes to decode the previous frame of encoded data and the time it takes to decode the adjacent next frame of encoded data. It then calculates the difference in time taken to process the two adjacent frames of encoded data, using this difference as the decoding time difference information.

[0096] Accordingly, the communication receiving module can add the receiving time difference information to the next frame of encoded data, and the decoding module can add the decoding time difference information to the next frame of rendered data, so that the next frame of rendered data sent by the decoding device to the ATW module contains encoding time difference information, transmission time difference information, receiving time difference information, and decoding time difference information. The head-mounted display device can extract the next frame of rendered data from the ATW module, and extract the encoding time difference information, transmission time difference information, receiving time difference information, and decoding time difference information from the next frame of rendered data, and use the sum of the time differences corresponding to the encoding time difference information, transmission time difference information, receiving time difference information, and decoding time difference information as the time difference information of the split AR device.

[0097] For ease of understanding, Figure 3 is used as a reference, but it does not limit the scope of this solution. Figure 3 is an example diagram of the modular structure of the split AR device of this application. Figure 3 shows a feasible example structure and does not limit the scope of this application. In Figure 3, the split AR device includes a head-mounted display device and a computing device. The head-mounted display device includes a time analysis module, a communication receiving module, a decoding module, an ATW module, a posture generation module, a posture prediction module, and a display module. The computing device's application runtime environment includes a posture usage module. The computing device also includes an encoding module and a communication transmitting module. The communication receiving module of the head-mounted display device and the communication transmitting module in the computing device communicate via a network, involving protocols such as Transmission Control Protocol (TCP) and User Datagram Protocol (UDP).

[0098] As shown in Figure 3, the attitude generation module in the head-mounted display device generates attitude prediction data containing the predicted on-screen time. The attitude usage module retrieves the attitude prediction data from the attitude generation module at fixed intervals (e.g., 1 second or 5 seconds). The attitude usage module can also send the attitude prediction data to the attitude usage module in the computing device. The attitude usage module uses the attitude prediction data for rendering. After generating the rendering data, the encoding module encodes the rendering data. After obtaining the encoded data, the encoding module adds its encoding time difference information to the encoded data and sends it to the communication sending module. The communication sending module adds its sending time difference information to the encoded data and sends it to the communication receiving module of the head-mounted display device. The communication receiving module adds its receiving time difference information to the encoded data and forwards it to the decoding module. The decoding module decodes the encoded data to obtain the rendering data. After obtaining the rendering data, the decoding module adds its decoding time difference information to the rendering data and sends it to the ATW module. When processing the rendering data of the next frame, the time analysis module can extract the encoding time difference information, transmission time difference information, reception time difference information and decoding time difference information from the rendering end of the next frame from the ATW module, in order to determine the time difference information that characterizes the latency of the split AR device.

[0099] This embodiment receives encoded time difference information and transmitted time difference information from a computing device; determines received time difference information based on the time difference between two adjacent frames of data processed by the communication receiving module; and determines decoded time difference information based on the time difference between two adjacent frames of data processed by the decoding module. This allows for accurate determination of time difference information characterizing the latency of a split-type AR device based on encoded time difference information, transmitted time difference information, received time difference information, and decoded time difference information, indirectly improving the display accuracy of the split-type AR device.

[0100] Based on the second embodiment of this application, a third embodiment of this application is proposed. In the third embodiment of this application, content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, please refer to Figure 4, which is a flowchart illustrating the third embodiment of the method of this application.

[0101] In this embodiment, step S20 includes steps S201 to S204:

[0102] Step S201: Obtain a preset deep learning model. The preset deep learning model is a long short-term memory network model obtained by training based on training data consisting of parameters corresponding to multiple time difference information and multiple time indicators.

[0103] It should be noted that the above time indicators can be parameters representing the number of vertical synchronization cycles.

[0104] In the specific implementation, referring to Figure 5, which is a schematic diagram of the basic architecture of the preset deep learning model in the third embodiment of the method of this application, X t For frame t input, h t For frame t output, X t-1 For frame t-1 input, h t-1 For frame t-1 output, X t+1 For frame t+1 input, h t+1 The output is for frame t+1. Taking frame t as the input, the forgetting gate in the Long Short-Term Memory (LSTM) model includes a left-hand σ function, which processes the input X at frame t. t The forgetting gate value is calculated by combining the hidden state from the previous time step. This value is between 0 and 1, where 0 represents complete forgetting and 1 represents complete retention. The input gate of the Long Short-Term Memory (LSTM) model includes a σ function and a tanh layer on the middle side. The σ function controls the input X of frame t. t The information needed to update the memory cells is used to create a new candidate value vector through a tanh layer, which may be added to the memory cells. The output gate of the Long Short-Term Memory (LSTM) model consists of a σ function on the right and a tanh layer. The output gate value is calculated by σ on the input Xt of frame t and the hidden state of the previous time step. This value is between 0 and 1, where 0 represents no output and 1 represents full output. Then, the cell state is processed by the tanh function, outputting a result between -1 and 1, which is multiplied by the output gate value to obtain the final output h of frame t. t The descriptions of frames t-1 and t+1 can be found in the description of frame t above, and will not be repeated here.

[0105] It should be understood that when constructing a pre-defined deep learning model based on a long short-term memory network model, the parameters corresponding to the time difference information in the training data can be used as inputs for each frame, and the time indicators can be used as outputs for each frame.

[0106] Step S202: The initial time index is obtained by the preset deep learning model based on the receiving time difference information, the decoding time difference information, the encoding time difference information, and the sending time difference information.

[0107] In a specific implementation, the aforementioned head-mounted display device can use the currently determined reception time difference information, decoding time difference information, encoding time difference information, and transmission time difference information as input to a preset deep learning model, and receive the initial time index output by the preset deep learning model based on the reception time difference information, decoding time difference information, encoding time difference information, and transmission time difference information.

[0108] In one feasible implementation, step S202 includes steps S2021 to S2026:

[0109] Step S2021: Obtain the training data of the preset deep learning model. The training data includes a first type of training data corresponding to the received time difference information, a second type of training data corresponding to the decoded time difference information, a third type of training data corresponding to the encoded time difference information, and a fourth type of training data corresponding to the transmitted time difference information.

[0110] It should be noted that the first type of training data mentioned above can be parameters corresponding to historical reception time difference information generated before the current time. The second type of training data mentioned above can be parameters corresponding to historical decoding time difference information generated before the current time. The third type of training data mentioned above can be parameters corresponding to historical encoding time difference information generated before the current time. The fourth type of training data mentioned above can be parameters corresponding to historical transmission time difference information generated before the current time.

[0111] In a specific implementation, the aforementioned head-mounted display device can acquire locally stored historical data, and determine from the historical data the first type of training data corresponding to the received time difference information, the second type of training data corresponding to the decoded time difference information, the third type of training data corresponding to the encoded time difference information, and the fourth type of training data corresponding to the transmitted time difference information, and construct training data for a preset deep learning model based on the first type of training data, the second type of training data, the third type of training data, and the fourth type of training data.

[0112] Step S2022: Standardize and normalize the receiving time difference information based on the mean and variance of the first type of training data to obtain the receiving time difference parameters.

[0113] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the received time difference information to obtain the received time difference parameters:

[0114]

[0115] In the formula, To receive the time difference parameter, x ij1 To receive time difference information, μ1 is the mean of each parameter in the first type of training data, and σ1 is the variance of each parameter in the first type of training data.

[0116] Step S2023: Standardize and normalize the decoding time difference information based on the mean and variance of the second type of training data to obtain the decoding time difference parameters.

[0117] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the decoding time difference information to obtain the decoding time difference parameters:

[0118]

[0119] In the formula, For decoding time difference parameters, x ij2 To decode the time difference information, μ2 is the mean of each parameter in the second type of training data, and σ2 is the variance of each parameter in the second type of training data.

[0120] Step S2024: Standardize and normalize the encoded time difference information based on the mean and variance of the third type of training data to obtain the encoded time difference parameters.

[0121] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the encoded time difference information to obtain the encoded time difference parameters:

[0122]

[0123] In the formula, x is the encoding time difference parameter. ij3 To encode time difference information, μ3 represents the mean of each parameter in the third class of training data, and σ3 represents the variance of each parameter in the third class of training data.

[0124] Step S2025: Standardize and normalize the transmission time difference information based on the mean and variance of the fourth type of training data to obtain the transmission time difference parameters.

[0125] In practical implementation, the aforementioned head-mounted display device can use the following formula to standardize and normalize the transmission time difference information to obtain the transmission time difference parameter:

[0126]

[0127] In the formula, For the transmission time difference parameter, x ij4 To send time difference information, μ4 is the mean of each parameter in the fourth type of training data, and σ4 is the variance of each parameter in the fourth type of training data.

[0128] Step S2026: Input the receiving time difference parameter, the decoding time difference parameter, the encoding time difference parameter, and the sending time difference parameter into the preset deep learning model to obtain the initial time index.

[0129] In its specific implementation, the aforementioned head-mounted display device standardizes and normalizes the received time difference information, decoded time difference information, encoded time difference information, and transmitted time difference information to obtain the corresponding received time difference parameters, decoded time difference parameters, encoded time difference parameters, and transmitted time difference parameters. Then, it inputs the received time difference parameters, decoded time difference parameters, encoded time difference parameters, and transmitted time difference parameters into a preset deep learning model to obtain the initial time index.

[0130] Step S203: Round the initial time index to obtain the target time index.

[0131] In one feasible implementation, step S203 includes steps S2031 to S2032:

[0132] Step S2031: Determine whether the target time indicator has reached the preset indicator threshold.

[0133] It should be noted that the aforementioned preset threshold values ​​can be used as calibration values ​​to determine whether the target time indicator is too large.

[0134] In a specific implementation, the aforementioned head-mounted display device can obtain the target time indicator through a preset deep learning model, and then compare the target time indicator with a preset indicator threshold to determine whether the target time indicator has reached the preset indicator threshold.

[0135] Step S2032: If yes, then set the target time indicator to the preset indicator threshold.

[0136] In practical implementation, when the aforementioned head-mounted display device detects that the target time indicator has reached a preset threshold, in order to avoid an excessively large target time indicator leading to a greater computational burden for subsequent predictions of the on-screen time, the value of the target time indicator can be reset to the preset threshold to limit the target time indicator. Conversely, when the target time indicator is detected to have not reached the preset threshold, the current value of the target time indicator can be retained for subsequent calculations of the predicted on-screen time.

[0137] Step S204: Determine the predicted on-screen time based on the target time indicator and the current display refresh rate.

[0138] In its implementation, the aforementioned head-mounted display device can calculate the reciprocal of the current display refresh rate to obtain the vertical synchronization period. Then, it uses the target time index as the target quantity and sets the predicted screen display time to the next vertical synchronization period plus the target number of vertical synchronization periods. For example, using vsync as the vertical synchronization period and N as the target time index, the predicted screen display time can be the next vsync of the head-mounted display device + N vsync cycles.

[0139] This embodiment uses a preset deep learning model, taking the received time difference information, decoded time difference information, encoded time difference information, and transmitted time difference information as model inputs to obtain an initial time index. At the same time, the initial time index is rounded to obtain a target time index. Thus, based on the target time index and the current display refresh rate, the on-screen time can be predicted more accurately, thereby improving the display accuracy of the head-mounted display device.

[0140] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the method shown in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0141] This application also provides a head-mounted display device. Please refer to Figure 6, which is a schematic diagram of the module structure of the head-mounted display device of this application. In Figure 6, the head-mounted display device includes:

[0142] The time analysis module 10 is used to determine the time difference information based on the time difference when two adjacent frames of data are processed.

[0143] The time prediction module 20 is used to determine the predicted screen display time based on the time difference information.

[0144] Display module 30 is used to output the display screen based on the predicted on-screen time.

[0145] The head-mounted display device provided in this application, employing the display method described in the above embodiments, can solve the technical problem in the prior art where the process from image generation to display is delayed, causing the image to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the head-mounted display device provided in this application are the same as those of the display method provided in the above embodiments, and other technical features in the head-mounted display device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0146] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the display method in Embodiment 1 or 4 described above.

[0147] Referring to Figure 7, which is a schematic diagram of the hardware operating environment involved in the display method in this embodiment, the electronic device in this embodiment may be, but is not limited to, a split-type XR device combining a head-mounted display device such as AR glasses, VR glasses, or MR glasses with a terminal such as a mobile phone, laptop computer, in-vehicle terminal, or desktop computer. The electronic device shown in Figure 7 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments in this application.

[0148] As shown in Figure 7, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. While electronic devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0149] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0150] The electronic device provided in this application, employing the display method described in the above embodiments, can solve the technical problem in the prior art where the process from generating to displaying a screen involves a time delay, causing the screen to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the electronic device provided in this application are the same as those of the display method provided in the above embodiments, and other technical features of this electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0151] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0153] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the display method in the above embodiments.

[0154] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0155] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.

[0156] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0158] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0159] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described display method. This solves the technical problem in the prior art where the process from image generation to display of the screen involves a time delay, causing the screen to be displayed prematurely or delayed, thus affecting the display effect. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the display method provided in the above embodiments, and will not be repeated here.

[0160] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A display method, characterized in that, The display method includes: determining time difference information based on the time difference when two adjacent frames of data are processed; determining the predicted on-screen time based on the time difference information; and outputting the display screen based on the predicted on-screen time.

2. The display method as described in claim 1, characterized in that, The time difference information includes encoding time difference information, transmission time difference information, and display time difference information. The step of determining the time difference information based on the time difference when two adjacent frames of data are processed includes: receiving the encoding time difference information and the transmission time difference information sent by the computing device, wherein the encoding time difference information is determined by the time difference when two adjacent frames of data in the computing device are processed by the encoding module, and the transmission time difference information is determined by the time difference when two adjacent frames of data in the computing device are processed by the communication transmission module; and determining the display time difference information based on the time difference when two adjacent frames of data are processed.

3. The display method as described in claim 2, characterized in that, The display time difference information includes reception time difference information and decoding time difference information. The step of determining the display time difference information based on the time difference when two adjacent frames of data are processed includes: determining the reception time difference information based on the time difference when two adjacent frames of data are processed by the communication receiving module; and determining the decoding time difference information based on the time difference when two adjacent frames of data are processed by the decoding module.

4. The display method as described in claim 3, characterized in that, The step of determining the predicted screen display time based on the time difference information includes: acquiring a preset deep learning model, wherein the preset deep learning model is a long short-term memory network model trained on training data consisting of parameters corresponding to multiple time difference information and multiple time indicators; obtaining an initial time indicator based on the receiving time difference information, the decoding time difference information, the encoding time difference information, and the sending time difference information through the preset deep learning model; rounding the initial time indicator to obtain a target time indicator; and determining the predicted screen display time based on the target time indicator and the current display refresh rate.

5. The display method as described in claim 4, characterized in that, The step of obtaining an initial time index based on the received time difference information, the decoded time difference information, the encoded time difference information, and the transmitted time difference information using the preset deep learning model includes: acquiring the training data of the preset deep learning model, wherein the training data includes a first type of training data corresponding to the received time difference information, a second type of training data corresponding to the decoded time difference information, a third type of training data corresponding to the encoded time difference information, and a fourth type of training data corresponding to the transmitted time difference information; and standardizing and normalizing the received time difference information according to the mean and variance of the first type of training data to obtain... The receiving time difference parameter is obtained by standardizing and normalizing the decoding time difference information based on the mean and variance of the second type of training data; the encoding time difference information is obtained by standardizing and normalizing the encoding time difference information based on the mean and variance of the third type of training data; the transmitting time difference information is obtained by standardizing and normalizing the transmitting time difference information based on the mean and variance of the fourth type of training data; the receiving time difference parameter, the decoding time difference parameter, the encoding time difference parameter, and the transmitting time difference parameter are input into the preset deep learning model to obtain the initial time index.

6. The display method as described in claim 4, characterized in that, After the step of rounding the initial time index to obtain the target time index, the method further includes: determining whether the target time index reaches a preset index threshold; if so, setting the target time index as the preset index threshold.

7. The display method as described in claim 1, characterized in that, The step of outputting the display screen based on the predicted on-screen time includes: generating attitude prediction data based on the predicted on-screen time; sending the attitude prediction data to a computing device so that the computing device can feed back rendering data based on the attitude prediction data; and outputting the display screen based on the rendering data.

8. A head-mounted display device, characterized in that, The head-mounted display device includes: a time analysis module for determining time difference information based on the time difference when two adjacent frames of data are processed; a time prediction module for determining the predicted on-screen time based on the time difference information; and a display module for outputting the display image based on the predicted on-screen time.

9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the display method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the display method as described in any one of claims 1 to 7.

Images