Image rendering method and apparatus

Abstract

This disclosure relates to an image rendering method and apparatus. A server receives an image request from a terminal, including first pose information acquired by the terminal at the current moment and the terminal's latency. Based on the first pose information, a second pose information for a target moment is predicted. The time difference between the target moment and the current moment is the total latency, which includes both terminal latency and server latency. Image rendering is performed based on the second pose information to obtain a rendered image for the target moment. The rendered image is returned to the terminal, which decodes, renders, and displays the image at the target moment. The entire process considers the total latency of both the terminal and the server. Furthermore, using the server for pose prediction makes the predicted second pose information more accurate, reduces performance requirements on the terminal, and minimizes screen jitter.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image rendering method and apparatus. Background Technology

[0002] In Virtual Reality (VR) technology, the VR terminal predicts the user's future pose based on the user's current pose information and provides this predicted pose information to the cloud VR server. The cloud VR server then renders an image based on the predicted pose information, encodes the rendered image, and sends it back to the VR terminal. The VR terminal decodes the rendered image and displays it on the screen.

[0003] The main purpose of pose prediction is to ensure consistency between the actual pose of the VR terminal and the content displayed on the screen at the same time, reducing display latency caused by rendering time. However, since rendering and video encoding are performed on the cloud VR server, the overall time required for pose prediction on the VR terminal cannot be accurately calculated, leading to inaccurate pose prediction and resulting in jitter in the final VR terminal display. VR terminal display jitter is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the present disclosure proposes an image rendering method and apparatus.

[0005] According to one aspect of this disclosure, an image rendering method is provided, applied on a server, the method comprising:

[0006] Receive an image request from the terminal, the image request including the first pose information of the current moment obtained by the terminal and the terminal delay;

[0007] The second pose information of the target time is predicted based on the first pose information. The time difference between the target time and the current time is the total delay, which includes the terminal delay and the server delay.

[0008] Image rendering is performed based on the second pose information to obtain the rendered image at the target time.

[0009] The rendered image is returned to the terminal so that the terminal can decode and render the rendered image and display it at the target time.

[0010] In one possible implementation, predicting the second pose information at the target time based on the first pose information includes:

[0011] Based on the first pose information and the target prediction equation, pose prediction is performed to obtain the predicted pose information;

[0012] The first pose information is used as the observation value to update the Kalman filter, and the Kalman gain is obtained.

[0013] The predicted pose information is estimated by performing state estimation based on the Kalman gain and the total delay to obtain the second pose information.

[0014] In one possible implementation, if the number of currently recorded first pose information is less than a preset number, the target prediction equation is a preset linear equation.

[0015] If the number of the first pose information already recorded is greater than or equal to a preset number, the target prediction equation is a preset recursive motion function equation.

[0016] In one possible implementation, the terminal delay includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image.

[0017] In one possible implementation, the terminal latency includes the duration of the terminal's estimated latency, and / or the server latency includes the duration of the server's latency estimation.

[0018] In one possible implementation, the first pose information includes position information and / or orientation quaternion information.

[0019] According to another aspect of this disclosure, an image rendering method is provided, applied to a terminal, the method comprising:

[0020] Obtain the first pose information at the current moment;

[0021] An image request is generated based on the first pose information and the terminal delay, wherein the image request includes the first pose information and the terminal delay;

[0022] The image request is sent to the server so that the server can predict the second pose information of the target time with a total delay from the current time and generate a rendered image based on the second pose information. The total delay includes the terminal delay and the server delay.

[0023] Upon receiving a rendered image from the server, the rendered image is decoded, rendered, and displayed at the target time.

[0024] In one possible implementation, the terminal delay includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image.

[0025] In one possible implementation, the server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding.

[0026] In one possible implementation, the terminal latency includes the duration of the terminal's estimated latency, and / or the server latency includes the duration of the server's latency estimation.

[0027] In one possible implementation, the second pose information of the target time, which is delayed by a total time interval from the current time, is predicted, including:

[0028] Based on the first pose information and the target prediction equation, pose prediction is performed to obtain the predicted pose information;

[0029] The first pose information is used as the observation value to update the Kalman filter, and the Kalman gain is obtained.

[0030] The predicted pose information is estimated by performing state estimation based on the Kalman gain and the total delay to obtain the second pose information.

[0031] In one possible implementation, if the number of currently recorded first pose information is less than a preset number, the target prediction equation is a preset linear equation.

[0032] If the number of the first pose information already recorded is greater than or equal to a preset number, the target prediction equation is a preset recursive motion function equation.

[0033] In one possible implementation, the first pose information includes position information and / or orientation quaternion information.

[0034] According to another aspect of this disclosure, an image rendering apparatus is provided for use on a server, the apparatus comprising:

[0035] The request receiving module is used to receive an image request from the terminal, wherein the image request includes the first pose information of the current moment obtained by the terminal and the terminal delay;

[0036] The pose prediction module is used to predict the second pose information of the target time based on the first pose information. The time difference between the target time and the current time is the total delay, which includes terminal delay and server delay.

[0037] An image rendering module is used to render an image based on the second pose information to obtain a rendered image at the target time.

[0038] The information return module is used to return the rendered image to the terminal so that the terminal can decode, render, and display the rendered image at the target time.

[0039] According to another aspect of this disclosure, an image rendering apparatus is provided for use in a client, the apparatus comprising:

[0040] The pose acquisition module is used to acquire the first pose information at the current moment;

[0041] A request generation module is used to generate an image request based on the first pose information and the terminal delay, wherein the image request includes the first pose information and the terminal delay;

[0042] The request sending module is used to send the image request to the server so that the server can predict the second pose information of the target time with a total delay from the current time and generate a rendered image based on the second pose information. The total delay includes the terminal delay and the server delay.

[0043] The image display module is used to decode and render the rendered image received from the server and display it at the target time.

[0044] According to another aspect of this disclosure, an image rendering apparatus is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described method when executing the instructions stored in the memory.

[0045] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided that stores computer program instructions thereon, wherein the computer program instructions, when executed by a processor, implement the above-described method.

[0046] According to another aspect of this disclosure, a computer program product is provided, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0047] The image rendering method and apparatus provided in this disclosure involve a terminal sending a first pose information and terminal latency as an image request to a server. The server, based on the total latency between the terminal and the server, predicts a second pose information at a target time with a total latency interval from the current time, and further renders the image based on the second pose. The rendered image and the second pose information are then returned to the terminal, allowing the terminal to decode and render the image and display it at the target time. The entire process considers the total latency between the terminal and the server. Furthermore, using the server to predict the second pose information makes the predicted second pose information more accurate, reduces performance requirements on the terminal, and minimizes screen jitter.

[0048] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0050] Figure 1 A schematic diagram illustrating an application scenario of an image rendering method according to an embodiment of the present disclosure is shown.

[0051] Figure 2 A schematic flowchart of an image rendering method according to an embodiment of the present disclosure is shown.

[0052] Figure 3 This diagram illustrates terminal latency and server latency in an image rendering method according to an embodiment of the present disclosure.

[0053] Figure 4 A block diagram of an image rendering apparatus according to an embodiment of the present disclosure is shown.

[0054] Figure 5 A block diagram of an image rendering apparatus according to an embodiment of the present disclosure is shown.

[0055] Figure 6 This is a block diagram illustrating an apparatus 800 for image rendering according to an exemplary embodiment.

[0056] Figure 7 This is a block diagram illustrating an apparatus 1900 for image rendering according to an exemplary embodiment. Detailed Implementation

[0057] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0058] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0059] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0060] To address the aforementioned technical problems, this disclosure provides an image rendering method and apparatus. The terminal sends the acquired first pose information and terminal latency as an image request to the server. Based on the total latency between the terminal and the server, the server predicts the second pose information at a target time with a total latency interval from the current time, using the first pose information. The server then performs image rendering based on the second pose and returns the rendered image and the second pose information to the terminal, enabling the terminal to decode and render the image and display it at the target time. The entire process considers the total latency between the terminal and the server. Furthermore, using the server to predict the second pose information makes the predicted second pose information more accurate, reduces the performance requirements of the terminal, and minimizes screen jitter.

[0061] like Figure 1 , Figure 2 As shown, the image rendering method provided in this embodiment is applied to terminal 100 and server 200. Terminal 100 can be used to execute steps S101-S103. Server 200 can be used to execute steps S201-S204. Terminal 100 can be a virtual reality device capable of pose detection and / or image display, including at least one of terminal devices such as VR headsets, VR glasses, VR all-in-one machines, mobile VR boxes, and mobile phones; this disclosure does not limit its use.

[0062] In step S101, the terminal 100 acquires the sensing data detected by the sensor and determines the user's first pose information at the current moment based on the sensing data. In some embodiments, the first pose information may include position information (x, y, z) and orientation quaternion information (rx, ry, rz, rw). The sensor in the terminal 100 may be a gyroscope, gravimeter, or other sensor used for pose detection, and this disclosure does not limit this.

[0063] In one possible implementation, terminal 100 can determine its corresponding terminal latency based on the time it takes to decode and render the image. In some embodiments, the terminal latency includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image. In some embodiments, the terminal latency also includes the time the terminal estimates the duration of the terminal latency.

[0064] The terminal latency corresponds to the speed at which the terminal 100 subsequently decodes and renders the rendered image returned by the server 200. If the terminal latency includes the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image, then, if the speed at which the terminal 100 decodes and renders the image is stable, the terminal 100 can pre-determine the decoding time and rendering time based on its own image decoding and rendering speed, and then directly add the decoding time and rendering time together as the terminal latency.

[0065] In step S102, terminal 100 generates an image request based on the first pose information and terminal delay. The image request includes the first pose information and terminal delay.

[0066] In step S103, terminal 100 sends an image request to server 200 so that server 200 can continue to execute step S201.

[0067] In step S201, after receiving the image request, the server 200 records the first pose information and the acquisition time corresponding to the first pose information (that is, the current time described in step S101), and counts the number n of the first pose information currently recorded.

[0068] In one possible implementation, the server 200 can predetermine the total latency, which includes the terminal latency and the server latency. The server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding. In some embodiments, the server latency may further include the time occupied by the server for server latency estimation.

[0069] In this scenario, where the server-side latency includes the time for pose prediction, image rendering, and image encoding, the server 200 can determine its own latency based on the processing time for these three processes. The sum of the terminal latency and the server-side latency is then used to determine the total latency. Specifically, the server 200 can determine the individual processing times for pose prediction, image rendering, and image encoding based on its own processing speed and computational load, and then sum these times to obtain the server-side latency. This total latency considers both the terminal and server-side latency, resulting in more accurate predicted second pose information and reduced terminal jitter.

[0070] For example, such as Figure 3 As shown, terminal latency may include the time t6 for terminal 100 to perform image decoding and the time t7 for rendering, while server latency may include the time t3 for server 200 to perform pose prediction, the time t4 for rendering the image, and the time t5 for encoding the image. In some embodiments, terminal latency may include, for example, the time t6 for rendering the image and the time t7 for encoding the image. Figure 3 The time t6 for image decoding by the terminal 100, the time t7 for rendering, and the time t1 for terminal latency estimation by the terminal 100 are included. The server latency may include the time t3 for pose prediction by the server 200, the time t4 for image rendering, the time t5 for image encoding, and the time t2 for server latency estimation by the server 200.

[0071] In step S202, the server 200 predicts the second pose information of the target time based on the first pose information, wherein the time difference between the target time and the current time is the total delay.

[0072] In this embodiment, the server 200 can pre-create a model of the pose estimation problem based on a Kalman filter and a recursive motion function. Specifically, the first received first pose information can be used as the state vector of the Kalman filter, and time t can be used as the state estimation time parameter. Then, the first first pose information and its corresponding system time are used to initialize the state vector and state estimation parameters of the Kalman filter.

[0073] The state equation of the Kalman filter is as follows:

[0074] x ′ =f(x) Formula (1)

[0075] P ′ =FPF T +Q formula(2)

[0076] where x ′ is the predicted state vector (predicted pose information). f(x) is the equation for constructing the predicted state. P ′ is the covariance matrix of the state, that is, the covariance matrix of the predicted pose information x ′ . P is the covariance matrix of the previous state. F is the Jacobian matrix of the predicted state equation. Q is the noise matrix, where Q can be the identity matrix (that is, it is considered that there is no noise error in the prediction process).

[0077]

[0078] where n is the number of the first pose information that has been recorded currently. N represents the preset number set in advance, and the specific value of N can be set according to actual needs. v is the speed. is the predicted pose information. is the current state vector, that is, the first pose information. pt is the total delay. K0 is the motion matrix. freq is the state time difference.

[0079] Then, in the process of predicting the second pose information in step S202, after receiving the first pose information, the server 200 can first determine the target prediction equation used for pose prediction according to the number of the first pose information that has been recorded currently. Among them, when the number n of the first pose information that has been recorded currently is less than the preset number N, the preset linear equation such as formula (3) is determined as the target prediction equation. When the number n of the first pose information that has been recorded currently is greater than or equal to the preset number N, the preset recursive motion function equation such as formula (4) is determined as the target prediction equation. Then, pose prediction is performed according to the first pose information and the target prediction equation to obtain the predicted pose information Update the Kalman filter with the first pose information as the observed value to obtain the Kalman gain; perform state estimation on the predicted pose information according to the Kalman gain and the total delay to obtain the second pose information x.

[0080] Among them, when the target prediction equation is formula (4), the recorded nearest N first pose information is used, the dimension of the preset recursive motion function equation (that is, the target prediction equation) is set to 7 (the number of state vectors), the custom retrospective coefficient f (f < N) is set, and the number of iterations can be determined according to the ratio of the total delay to the time difference between states (it can be set to the average time difference of t), and then prediction state estimation is performed to obtain the predicted pose information

[0081] In this embodiment, the observation equation is as follows:

[0082] S = HP ′H T +R Formula (6)

[0083] K = P ′ H T S -1 Formula (7)

[0084] x = x ′ +KΔz formula (8)

[0085] P = P ′ -KHP ′ Formula (9)

[0086] Where x is the final estimated state vector, i.e., the second pose information. P is the covariance matrix of the updated state, and Equation (9) is used to update P. In Equation (9), P... ′ The latest P is calculated using formula (2). After updating P using formula (9), it is then substituted back into formula (2) to calculate the new P. ′ This is achieved through iterative looping. Δz is the error between the observed state vector value (corresponding to the first pose information at the current moment) and the predicted state vector value (corresponding to the predicted pose information). H is the Jacobian matrix of the observed state vector. R is the noise matrix, where R can be the identity matrix (i.e., the observation process is considered to be noise-free).

[0087] In this embodiment, the recursive motion function equation is as follows:

[0088]

[0089] S*K i =L formula (12)

[0090] Among them, l(T) c ) and s(T c ) is the state vector, l(T) c )x i This represents the coordinates of the state vector in dimension i. We use the N most recent recorded first-position information to form L for each dimension i, and use N-1 previously recorded first-position information to form S. We then perform SVD decomposition on S to calculate K for each dimension i. i The value is then used to obtain K (which can be K from various dimensions). i (The matrix formed). Since the total delay will be greater than the state update time, multiple recursive iterations are required to obtain the state vector at the target time.

[0091] In step S203, the server 200 performs image rendering based on the second pose information to obtain the rendered image at the target time.

[0092] In step S204, the server 200 returns the rendered image to the terminal 100, so that the terminal 100 can continue to execute step S103. The server 200 may also return the second pose information along with the rendered image to the terminal 100.

[0093] In step S104, after receiving the rendered image, the terminal 100 decodes and renders the rendered image and displays it at the target time.

[0094] The terminal 100 can first decode and render the received rendering image, and then send the resulting image to the display screen to achieve image display at the target time, so that the user can see the adjusted rendering result.

[0095] like Figure 4 As shown in the embodiments of this disclosure, an image rendering apparatus is also provided, applied to a server, the apparatus comprising:

[0096] The request receiving module 41 is used to receive an image request from the terminal, the image request including the first pose information of the current moment obtained by the terminal and the terminal delay.

[0097] The pose prediction module 42 is used to predict the second pose information at a target time based on the first pose information. The time difference between the target time and the current time is the total delay. The total delay includes terminal delay and server delay.

[0098] The image rendering module 43 is used to perform image rendering based on the second pose information to obtain the rendered image at the target time.

[0099] The information return module 44 is used to return the rendered image to the terminal so that the terminal can decode and render the rendered image and display it at the target time.

[0100] In one possible implementation, the pose prediction module 42 may include:

[0101] The pose prediction submodule is used to perform pose prediction based on the first pose information and the target prediction equation to obtain the predicted pose information.

[0102] The gain determination submodule is used to update the Kalman filter using the first pose information as the observation value to obtain the Kalman gain.

[0103] The pose information determination submodule is used to perform state estimation on the predicted pose information based on the Kalman gain and the total delay to obtain the second pose information.

[0104] In one possible implementation, if the number of currently recorded first pose information is less than a preset number, the target prediction equation is a preset linear equation.

[0105] If the number of the first pose information already recorded is greater than or equal to a preset number, the target prediction equation is a preset recursive motion function equation.

[0106] In one possible implementation, the server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding.

[0107] In one possible implementation, the terminal delay includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image.

[0108] In one possible implementation, the terminal latency includes the duration of the terminal's estimated latency, and / or the server latency includes the duration of the server's latency estimation.

[0109] In one possible implementation, the first pose information includes position information and / or orientation quaternion information.

[0110] like Figure 5 As shown in the embodiments of this disclosure, an image rendering apparatus is also provided, applied to a terminal, the apparatus comprising:

[0111] The pose acquisition module 51 is used to acquire the first pose information at the current moment.

[0112] The request generation module 52 is used to generate an image request based on the first pose information and the terminal delay, wherein the image request includes the first pose information and the terminal delay.

[0113] The request sending module 53 is used to send the image request to the server so that the server can predict the second pose information of the target time with a total delay from the current time and generate a rendered image based on the second pose information. The total delay includes the terminal delay and the server delay.

[0114] The image display module 54 is used to decode and render the rendered image received from the server and display it at the target time.

[0115] In one possible implementation, the terminal delay includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image.

[0116] In one possible implementation, the server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding.

[0117] In one possible implementation, the terminal latency includes the duration of the terminal's estimated latency, and / or the server latency includes the duration of the server's latency estimation.

[0118] In one possible implementation, the second pose information of the target time, which is delayed by a total time interval from the current time, is predicted, including:

[0119] Based on the first pose information and the target prediction equation, pose prediction is performed to obtain the predicted pose information;

[0120] The first pose information is used as the observation value to update the Kalman filter, and the Kalman gain is obtained.

[0121] The predicted pose information is estimated by performing state estimation based on the Kalman gain and the total delay to obtain the second pose information.

[0122] In one possible implementation, if the number of currently recorded first pose information is less than a preset number, the target prediction equation is a preset linear equation.

[0123] If the number of the first pose information already recorded is greater than or equal to a preset number, the target prediction equation is a preset recursive motion function equation.

[0124] In one possible implementation, the first pose information includes position information and / or orientation quaternion information.

[0125] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation and beneficial effects can be referred to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0126] It should be noted that although the image rendering method and apparatus have been described above as examples, those skilled in the art will understand that this disclosure is not limited thereto. In fact, users can flexibly set each step and module according to their personal preferences and / or actual application scenarios, as long as it conforms to the technical solution of this disclosure.

[0127] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0128] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0129] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0130] Figure 6 This is a block diagram illustrating an apparatus 800 for image rendering according to an exemplary embodiment. For example, apparatus 800 may be a virtual reality device.

[0131] Reference Figure 6 The device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output interface 812 (I / O interface), sensor component 814, and communication component 816.

[0132] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0133] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0134] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.

[0135] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0136] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0137] Input / output interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0138] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0139] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0140] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0141] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of the device 800 to perform the above-described method.

[0142] Figure 7 This is a block diagram illustrating an apparatus 1900 for image rendering according to an exemplary embodiment. For example, apparatus 1900 can be provided as a server, serving as the aforementioned server 200. (Refer to...) Figure 7The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0143] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM Unix TM Linux TM FreeBSD TM Or similar.

[0144] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0145] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0146] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0147] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0148] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0149] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0150] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0151] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0153] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image rendering method, characterized in that, Applied to the server side, the method includes: Receive an image request from the terminal, the image request including the first pose information of the current moment obtained by the terminal and the terminal delay; The second pose information of the target time is predicted based on the first pose information. The time difference between the target time and the current time is the total delay, which includes the terminal delay and the server delay. Image rendering is performed based on the second pose information to obtain the rendered image at the target time. The rendered image is returned to the terminal so that the terminal can decode and render the rendered image and display it at the target time. The server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding; the server latency also includes the time occupied by the server to perform server latency estimation. The terminal latency includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image; the terminal latency also includes the time the terminal estimates the time occupied by the terminal latency.

2. The method according to claim 1, characterized in that, The second pose information at the target time is predicted based on the first pose information, including: Based on the first pose information and the target prediction equation, pose prediction is performed to obtain the predicted pose information; The first pose information is used as the observation value to update the Kalman filter, and the Kalman gain is obtained. The predicted pose information is estimated by performing state estimation based on the Kalman gain and the total delay to obtain the second pose information.

3. The method according to claim 2, characterized in that, If the number of the first pose information already recorded is less than a preset number, the target prediction equation is a preset linear equation. If the number of the first pose information already recorded is greater than or equal to a preset number, the target prediction equation is a preset recursive motion function equation.

4. The method according to claim 1, characterized in that, The first pose information includes position information and / or orientation quaternion information.

5. An image rendering method, characterized in that, Applied to a terminal, the method includes: Obtain the first pose information at the current moment; An image request is generated based on the first pose information and the terminal delay, wherein the image request includes the first pose information and the terminal delay; The image request is sent to the server so that the server can predict the second pose information of the target time with a total delay from the current time and generate a rendered image based on the second pose information. The total delay includes the terminal delay and the server delay. Upon receiving a rendered image from the server, the rendered image is decoded, rendered, and displayed at the target time. The terminal latency includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image; the terminal latency also includes the time the terminal estimates the time it will take to delay the terminal. The server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding; the server latency also includes the time occupied by the server to perform server latency estimation.

6. The method according to claim 5, characterized in that, The second pose information of the target time, which is predicted to have a total delay from the current time, includes: Based on the first pose information and the target prediction equation, pose prediction is performed to obtain the predicted pose information; The first pose information is used as the observation value to update the Kalman filter, and the Kalman gain is obtained. The predicted pose information is estimated by performing state estimation based on the Kalman gain and the total delay to obtain the second pose information.

7. The method according to claim 6, characterized in that, If the number of the first pose information already recorded is less than a preset number, the target prediction equation is a preset linear equation. If the number of the first pose information already recorded is greater than or equal to a preset number, the target prediction equation is a preset recursive motion function equation.

8. The method according to claim 5, characterized in that, The first pose information includes position information and / or orientation quaternion information.

9. An image rendering apparatus, characterized in that, Applied to the server side, the device includes: The request receiving module is used to receive an image request from the terminal, wherein the image request includes the first pose information of the current moment obtained by the terminal and the terminal delay; The pose prediction module is used to predict the second pose information of the target time based on the first pose information. The time difference between the target time and the current time is the total delay, which includes terminal delay and server delay. An image rendering module is used to render an image based on the second pose information to obtain a rendered image at the target time. An information return module is used to return the rendered image to the terminal so that the terminal can decode and render the rendered image and display it at the target time. The terminal latency includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image; the terminal latency also includes the time the terminal estimates the time it will take to delay the terminal. The server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding; the server latency also includes the time occupied by the server to perform server latency estimation.

10. An image rendering apparatus, characterized in that, Applied to a client, the device includes: The pose acquisition module is used to acquire the first pose information at the current moment; A request generation module is used to generate an image request based on the first pose information and the terminal delay, wherein the image request includes the first pose information and the terminal delay; The request sending module is used to send the image request to the server so that the server can predict the second pose information of the target time with a total delay from the current time and generate a rendered image based on the second pose information. The total delay includes the terminal delay and the server delay. The image display module is used to decode and render the rendered image received from the server and display it at the target time. The terminal latency includes at least one of the time it takes for the terminal to decode the rendered image and the time it takes for the terminal to render the decoded image; the terminal latency also includes the time the terminal estimates the time it will take to delay the terminal. The server latency includes at least one of the following: the time for the server to perform pose prediction, the time for the server to perform image rendering, and the time for the server to perform image encoding; the server latency also includes the time occupied by the server to perform server latency estimation.

11. An image rendering apparatus, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to, when executing instructions stored in the memory, implement the method of any one of claims 1 to 4, or implement the method of any one of claims 5 to 8.

12. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 4 or the method described in any one of claims 5 to 8.

13. A computer program product comprising a computer program, or a non-volatile computer-readable storage medium carrying a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 4 or the method described in any one of claims 5 to 8.

Images