Image streaming display method and device

Abstract

The present application relates to the field of display technology, and provides an image streaming display method and device, wherein a VR device sends posture data for controlling the display of an image to an ordinary terminal in real time, and receives the real image after the control, and determines the prediction time corresponding to each frame of the image to be compensated according to the time interval between two adjacent frames of real images received and the preset number of compensation frames, and determines the compensation posture data corresponding to each prediction time according to the saved historical posture data set, and predicts and displays the corresponding compensation image in combination with the real image currently displayed. Since the compensation posture data corresponding to each prediction time determined according to the historical posture data set are located between the posture data corresponding to two adjacent frames of real images, the compensation image between the two adjacent frames of real images conforms to the actual motion trajectory, and when the next frame of real picture is displayed, it will not cause the picture to jump, effectively solving the problem of picture jitter during streaming display, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of display technology, and in particular to an image stream display method and device. Background Art

[0002] Virtual Reality (VR) technology uses computer systems and sensor technology to generate a three-dimensional space, providing users with an immersive experience. It has been widely used in various industries, especially in the gaming field.

[0003] In order to bring an immersive experience to users, ordinary terminals (such as mobile phones, TVs, computers, etc.) can stream their own images to VR devices, and the VR devices will display the received images to achieve a 3D effect. In addition, VR devices are generally equipped with an inertial measurement unit (IMU), which can measure the wearer's head posture data. After the VR device sends the posture data to the ordinary terminal, it can control the display image.

[0004] Generally, when a VR device is rendering and displaying, the GPU's rendering frame rate is generally lower than the screen's refresh rate. Therefore, the VR device needs to perform image compensation through time warping to reduce the image jitter caused by the rendering frame rate being lower than the refresh rate.

[0005] At present, when VR devices perform time warping, they use the compensated image predicted based on the previous real image and the current posture. However, when displaying the streamed image, since the VR device transmits the posture data to the ordinary terminal through the network, and the ordinary terminal's image is also transmitted back to the VR device through the network, the head will turn during the transmission process. Therefore, there will be a certain network delay when the VR device receives the next real image. The posture data corresponding to the next real image is the posture data before the compensated image. In this way, when the VR device replaces the compensated image with the received real image, there will be a picture jump, and when it is fed back to the human eye, the picture will jitter, causing dizziness. Summary of the invention

[0006] The embodiments of the present application provide an image stream display method and device to reduce image jitter and thereby alleviate head dizziness.

[0007] In a first aspect, an embodiment of the present application provides a streaming image display method, which is applied to a VR device, comprising:

[0008] Acquire a time interval between two adjacent frames of real images received from a common terminal, where the real image is determined by the common terminal according to the posture data sent by the VR device;

[0009] Determining the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames;

[0010] For each prediction time, a compensation pose data is determined according to the historical pose data set, and the compensation pose data is located between the pose data corresponding to the two adjacent frames of real images;

[0011] According to the currently displayed real image and each compensated posture data, the corresponding compensated image is predicted and displayed;

[0012] When the next frame of real image sent by the ordinary terminal is received, it is directly displayed.

[0013] In a second aspect, an embodiment of the present application provides a VR device, including a processor, a memory, a communication interface, and a display screen, wherein the display screen, the communication interface, the memory, and the processor are connected via a bus:

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Receiving two adjacent frames of real images sent by a common terminal through the communication interface, and determining a time interval between the two adjacent frames of real images, wherein the real images are determined by the common terminal according to the posture data sent by the VR device;

[0016] Determining the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames;

[0017] For each prediction time, a compensation pose data is determined according to the historical pose data set, and the compensation pose data is located between the pose data corresponding to the two adjacent frames of real images;

[0018] According to the real image currently displayed on the display screen and each compensation posture data, a corresponding compensation image is predicted, and displayed on the display screen;

[0019] When the next frame of real image sent by the ordinary terminal is received through the communication interface, it is directly displayed on the display screen.

[0020] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a VR device to execute the image stream display method provided by the embodiment of the present application.

[0021] In the above-mentioned embodiment of the present application, the VR device sends the posture data to the ordinary terminal in real time, and saves the sent historical posture data to obtain the historical posture data set. The ordinary terminal determines the real image based on the posture data, and streams the real image to the VR device. The VR device determines the prediction time corresponding to each frame of the image to be compensated based on the time interval between the two adjacent frames of real images received and the preset number of compensation frames. For each prediction time, the VR device determines a compensation posture data based on the historical posture data set, and predicts and displays the corresponding compensation image in combination with the currently displayed real image, and directly displays it when the next frame of real image is received. Since the compensation posture data corresponding to each prediction time determined according to the historical posture data set are located between the posture data corresponding to the two adjacent frames of real images, the compensation image between the two adjacent frames of real images conforms to the actual motion trajectory and will not cause the picture to jump, which effectively solves the problem of picture jitter during streaming display, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0024] Figure 2 The schematic diagram of screen jitter during streaming display of a VR device provided in an embodiment of the present application is exemplarily shown;

[0025] Figure 3 The flowchart of the method for displaying image streams provided by the embodiment of the present application is exemplified;

[0026] Figure 4 The flowchart of the method for determining the prediction time of the image to be compensated provided by the embodiment of the present application is exemplarily shown;

[0027] Figure 5 The effect diagram of the image stream display provided by the embodiment of the present application is exemplarily shown;

[0028] Figure 6 An interactive flow chart of image stream display provided by an embodiment of the present application is exemplarily shown;

[0029] Figure 7 The structure diagram of the VR device provided in the embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0030] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0031] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0032] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0033] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application; Figure 1 It is shown that the VR device 100 interacts with the ordinary terminal 300 through the network 200. The VR device 100 transmits the posture data of the head to the ordinary terminal 300 through the network 200 to control the image displayed by the ordinary terminal 300, and the ordinary terminal 300 transmits the controlled image to the VR device 100 through the network 200 for rendering and display. During the streaming display process, the VR device 100 predicts at least one compensation image based on the posture data and the last frame image received, and replaces the corresponding compensation image when receiving the real image transmitted by the ordinary terminal 300. Among them, the ordinary terminal 300 includes but is not limited to a desktop computer 301, a laptop computer 302, and a smart TV 303.

[0034] In non-streaming display application scenarios, VR devices use time warping technology to predict and compensate images based on the current posture data and the last real image displayed, thereby increasing the rendering frame rate and avoiding image jitter. However, in streaming display application scenarios, due to network transmission delays, if the compensation image is still predicted based on the current posture data and the last real image, the problem of image jitter cannot be solved.

[0035] In the scenario where the image of an ordinary terminal is streamed to a VR device for display, assuming that the overall time taken by the VR device from sending a posture to receiving the image corresponding to the posture is 30ms, and the VR device refreshes every 16ms, then a compensation image needs to be predicted in this 30ms.

[0036] For example, Figure 2As shown in the figure, at time T0, the VR device transmits the posture POS_0 to the ordinary terminal through the network. The ordinary terminal controls the current screen according to the posture POS_0, obtains the screen A, and transmits the screen A back to the VR device through the network. The VR device receives the screen A at time T1, and renders the screen A on two display screens for stereoscopic display. The display effect is screen ①. At the same time, the VR device sends the posture POS_1 at time T to the ordinary terminal. If the VR device does not receive the screen returned by the ordinary terminal at time T2, it predicts and compensates the screen display according to the posture POS_2 at time T2 and the screen A received at the previous moment, and the display effect is screen ②. At time T3, the VR device receives the screen B corresponding to the posture POS_1 returned by the ordinary terminal, and renders the screen B on two display screens for stereoscopic display. The display effect is screen ③. Ideally, according to the compensation result of screen ②, the screen ④ should be displayed at time T3. The posture corresponding to screen ④ should be after POS_2, while the posture corresponding to screen ③ is before POS_2. In this way, there is a large gap between the real picture ③ and the ideal picture ④, which will cause jumps and cause dizziness.

[0037] In view of this, an embodiment of the present application provides an image streaming display method and device, which determines the predicted time of each frame of the image to be compensated according to the time interval between two adjacent frames of real images received from an ordinary terminal and a preset number of compensation frames, and for each predicted time, determines a compensation posture data using historical posture data, wherein the compensation posture data is located between the posture data corresponding to the two adjacent frames of real images, so that the compensated picture between the two adjacent frames of real images conforms to the actual motion trajectory and does not cause picture jumps, effectively solving the phenomenon of picture jitter during streaming display, thereby improving user experience.

[0038] In an embodiment of the present application, the image received by the VR device from the ordinary terminal is recorded as a real image, and the image predicted by the VR device based on the posture data is recorded as a compensated image.

[0039] See also Figure 3 , is a method flow of an image stream display provided in an embodiment of the present application, the method is executed by a VR device, and mainly includes the following steps:

[0040] S301: Obtain a time interval between two adjacent frames of real images received from a common terminal, wherein the real image is determined by the common terminal according to posture data sent by a VR device.

[0041] When a VR device streams and displays images from an ordinary terminal, it sends the posture data to the ordinary device via the network in real time. The ordinary device controls the image it displays according to the received posture data at the refresh rate of the target video, and sends the controlled real image to the VR device via the network. At the same time, the ordinary terminal also sends the posture data corresponding to the real image to the VR device, which is saved by the VR device.

[0042] Generally, the frequency at which a VR device sends posture data may be inconsistent with the refresh rate of the target video. Therefore, not every posture data has a corresponding real image. In an embodiment of the present application, when a VR device sends posture data to an ordinary terminal, it will record the sent posture data locally and save it as a historical posture data set. Each historical posture data in the historical data set contains a posture (position) and the sending time (time) of the posture.

[0043] In actual applications, since the VR device does not know the moment when the next frame of real image is received, when predicting the compensated posture data of the image to be compensated, the VR device can make a prediction based on the time interval between at least two frames of real images streamed by an ordinary terminal, so as to perform frame compensation before receiving the next frame of real image.

[0044] In specific implementation, when an ordinary terminal sends each frame of real image, it will also send the posture data corresponding to the frame of real image to the VR device. The VR device can determine whether the currently rendered display image is a real image, and record the time of receiving two adjacent frames of real images to obtain the time interval between the two adjacent frames of real images.

[0045] For example, assuming that the time when the VR device receives the first frame of the real image of the ordinary terminal stream is T0, and the time when the second frame of the real image is T1, then the time interval deltaT=T1-T0.

[0046] In an embodiment of the present application, the time interval is affected by the network. In the case of severe frame loss, the time interval between the previous real image and the next real image will exceed the upper threshold, reducing the accuracy of the predicted compensated image. In order to improve the accuracy of the compensated image prediction, it is necessary to eliminate network anomalies. Specifically, when the time interval is less than at least one refresh interval of the VR device screen, it indicates that the time interval is valid and the compensated image can be predicted using the time interval. Otherwise, it indicates that the time interval is invalid and needs to be recalculated.

[0047] S302: Determine the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames.

[0048] In the embodiment of the present application, one or more frames of images are compensated between two adjacent frames of real images to ensure the refresh rate requirement of the VR device. Each frame of the image to be compensated corresponds to a prediction time. For the specific determination process, see Figure 4 :

[0049] S3021: Divide the value interval (0, 1) into a corresponding number of intervals according to the number of compensation frames.

[0050] For example, assuming that the number of compensation frames N=2, the first value interval after equal division is (0, 0.5), and the second value interval after equal division is (0.5, 1).

[0051] For another example, assuming that the number of compensation frames N=3, the first value interval after equal division is (0, 0.3), the second value interval after equal division is (0.3, 0.6), and the third value interval after equal division is (0.6, 1).

[0052] When the number of compensation frames N=1, the corresponding interval is the original value interval (0, 1).

[0053] S3022: Take a target value from each interval, and multiply the target value by the time interval.

[0054] For example, assuming the compensation frame number N = 1, take a target value ΔT from the interval (0, 1) 1 =0.5; Assuming the number of compensation frames N = 2, a target value ΔT is taken from the interval (0, 0.5) 2 =0.3, take a target value ΔT from the interval (0.5, 1) 3 =0.7.

[0055] S3023: Taking the reception time of the first frame of the real image in two adjacent frames of the real image as a reference, add each product respectively to obtain the prediction time corresponding to each frame of the image to be compensated.

[0056] For example, assuming that the time when the VR device receives the first real image of two adjacent real images is T0, the calculation formula for the predicted time is:

[0057] T1 i ′=T0+deltaT*ΔT i , i=N Formula 1

[0058] S303: For each prediction time, determine a compensation pose data according to the historical pose data set, wherein the compensation pose data is located between pose data corresponding to two adjacent frames of real images.

[0059] In the specific implementation, each historical posture data and the sending time of the historical posture data in the historical posture data set are fitted to obtain the objective function, and according to the objective function, a compensation posture data corresponding to each prediction time is determined.

[0060] In an embodiment of the present application, when the VR device sends a real image, the posture data corresponding to the real image is also sent to the VR device, and each prediction time is located between two adjacent frames of real images. Therefore, each compensated posture data determined according to the historical posture data set is located between the posture data corresponding to two adjacent frames of real images.

[0061] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the fitting algorithm. For example, the least squares method can be used for fitting, and the gradient descent method can also be used for fitting.

[0062] In an optional implementation, in order to improve fitting efficiency and accuracy, the VR device may perform fitting based on historical pose data located between two adjacent frames of real images in the historical pose data set.

[0063] S304: According to the currently displayed real image and each compensated posture data, a corresponding compensated image is predicted and displayed.

[0064] In the embodiment of the present application, the time interval can only be calculated after receiving two adjacent frames of real images, and frame compensation may not be performed between the two adjacent frames of real images. Starting from the reception of the second frame of real image, the corresponding compensated image is predicted and displayed based on the currently displayed real image and each compensated posture data. Since each compensated posture data is a point in the motion trajectory corresponding to the previous frame of real image and the next frame of real image, and there is a time sequence between each compensated posture data, the predicted frames of compensated images are compounded with the motion trend between the two frames of real images, and will not cause screen jumps.

[0065] S305: directly display the next frame of real image when it is received.

[0066] For example, assuming that the second frame of the real image is currently being displayed, two frames of compensation images are predicted, which are respectively recorded as compensation image 1 and compensation image 2. When the first refresh time arrives, the third frame of the real image has not been received, so compensation image 1 is displayed. Due to the increase in network speed, when the second refresh time arrives, the third frame of the real image has been received, and the third frame of the real image is used to replace compensation image 2 for display.

[0067] For another example, assuming that the second frame of the real image is currently being displayed, two frames of compensated images are predicted, which are respectively recorded as compensated image 1 and compensated image 2. When the first refresh time arrives, the third frame of the real image has not been received, so compensated image 1 is displayed. When the second refresh time arrives, the third frame of the real image has not been received, so compensated image 2 is displayed. When the third refresh time arrives, the third frame of the real image is directly displayed.

[0068] The following takes compensating one frame of image as an example. Figure 5 , which is a process diagram of the image streaming display method provided by the embodiment of the present application, the VR device records the posture sent to the ordinary terminal at each moment, and saves it locally to obtain a historical posture data set. At time T0, the VR device sends posture POS_0 to the ordinary terminal. The ordinary terminal controls the screen displayed by itself according to the posture POS_0, obtains screen A, and transmits screen A and the posture POS_0 corresponding to screen A back to the VR device through the network; the VR device receives screen A at time T1, and there is no need to perform frame compensation. Screen A is directly rendered on two display screens for stereoscopic display, and the display effect is screen ①. At the same time, the VR device sends the posture POS_1 at time T1 to the ordinary terminal. At time T2, the VR device sends POS_2 to the ordinary terminal. Due to the existence of network delay, the VR device does not receive the screen B corresponding to the posture POS_1 sent by the ordinary terminal, and frame compensation is required. At time T2, the pose of the currently displayed picture ① is POS_0, and the sending time of POS_0 is T0. Based on T0, the predicted time T1′ of the picture to be compensated is determined, where T1′ is between T0 and T1. T1′ is brought into the objective function of the historical pose data set fitting to obtain the pose POS_1′, where POS_1′ is after POS_0 and before POS_1. Further, based on the currently displayed picture ① and the determined pose POS_1′, a frame of compensated picture ② is predicted and displayed at time T2. At time T3, the VR device sends the pose POS_3 to the ordinary terminal, and has received the picture B that the ordinary terminal will control according to the pose POS_1. The VR device renders the picture B on two display screens for stereoscopic display, and the display effect is picture ③.

[0069] like Figure 5 As shown, the compensation picture ② is the picture predicted by the compensation posture after T0 and before T2. In this way, when the real picture corresponding to the posture POS_1 sent at time T1 is received, since the compensation picture is the picture predicted by the compensation posture between the picture ① and the T1 picture ③, it conforms to the actual motion trajectory and will not produce the following Figure 1 The picture jumps shown will not cause discomfort to the user's head.

[0070] In an embodiment of the present application, during the streaming display process, the VR device sends the posture data to the ordinary terminal, and obtains a historical posture data set based on the sent posture data. The ordinary terminal returns the image controlled by the posture data of the VR device and the posture data corresponding to the image to the VR device. The VR device determines the prediction time corresponding to each frame of the image to be compensated according to the time interval between the two adjacent real images received and the preset number of compensation frames, and fits an objective function according to the historical posture data between the two display moments. After each prediction time is brought into the objective function, the compensation posture corresponding to each prediction time is obtained, and the corresponding compensation image is predicted and displayed in combination with the currently displayed real image, and is directly displayed when the next frame of the real image is received. Since each prediction time is between two adjacent frames of real images displayed, the compensation posture data corresponding to each prediction time determined according to the historical posture data set are all between the posture data corresponding to the two frames of real images, so that the compensation image between the two adjacent frames of real images conforms to the actual motion trajectory and does not cause the picture to jump, effectively solving the problem of picture jitter during streaming display, thereby improving the user experience.

[0071] In some embodiments, when the encoding frame rate of an ordinary terminal is N times the decoding frame rate of a VR device, where N is an integer greater than or equal to 2, the VR device can directly select historical pose data for which no corresponding real image exists from the historical pose data set as the compensation pose data corresponding to the corresponding prediction time.

[0072] For example, assuming that the frame rate of the target video displayed by an ordinary terminal is 60 frames per second, and the encoding frame rate of the ordinary terminal is equal to the frame rate of the target video, that is, the number of pose data sent by the VR device to the ordinary terminal per second is greater than or equal to 60, and the decoding frame rate of the VR device is 30 frames per second, which is lower than the encoding frame rate of the ordinary terminal. In order to ensure that the VR device displays the latest image frame each time, half of the 60 frames of images encoded by the ordinary terminal need to be discarded. In an ideal case, it can be understood as discarding every other frame, that is, discarding one frame of image for every other pose data. In this way, assuming that the pose data corresponding to the image frame is 0, 2, 4, 6..., and the pose data corresponding to the image frame is 1, 3, 5..., at this time, under the condition that one frame of image needs to be compensated, the VR device can directly obtain poses 1, 3, 5 from the historical pose data set as the compensation pose data of the compensation image.

[0073] In practical applications, by using historical posture data that has not actually generated an image to perform frame compensation, it is only necessary to use the posture of the historical posture data before the posture data corresponding to the next frame of real image displayed to compensate. This can ensure that the image displayed by the VR device is compensated in a smooth manner during streaming display.

[0074] It should be noted that during streaming display, there is a certain delay between the image displayed by the VR device and the image displayed by the ordinary terminal. However, since the compensation images are generated based on the posture data between the two frames of real images, when the delay at both ends is small, the rendering frame rate of the VR device can be improved and will not affect the VR experience.

[0075] The method for displaying image streams provided in the embodiments of the present application can be applied to various virtual reality scenarios. For example, the images that can be displayed can be games developed with Unity, teaching videos, or film and television dramas.

[0076] Taking the game scene as an example, the method for displaying image streams provided by the embodiment of the present application is described from the interaction process between the TV and the VR glasses. Figure 6 , the method mainly includes the following steps:

[0077] S601: The VR glasses establish a connection with the TV.

[0078] In an optional implementation, the VR glasses and the television may be connected by wire via a communication interface.

[0079] S602: The TV starts a target game.

[0080] Among them, the game interface of the target game can be controlled by VR glasses.

[0081] S603: The VR glasses send the posture data to the TV in real time and store the local historical posture data set.

[0082] The user changes the posture of the VR glasses by moving his head. The VR glasses use IMU to measure the posture data of the user's head in real time and send it to the TV through the network, and record the sent posture data in the historical posture data set.

[0083] S604: The TV uses the received posture data to control its own game interface according to the frame rate of the target game.

[0084] After receiving the posture data sent by the VR glasses, the TV controls the game interface according to the posture data.

[0085] S605: The TV sends the controlled game interface and the posture data corresponding to the game interface to the VR glasses.

[0086] During streaming display, the VR glasses do not know which posture data controls the received game interface. Therefore, when the TV sends the game interface through the OpenVR interface, it also sends the posture data corresponding to the game interface to the VR glasses, so that the compensation posture data determined by the VR glasses is located after the posture data, ensuring smooth display of the game interface.

[0087] S606: The VR glasses respond to the refresh request and determine whether the game interface sent by the TV is received. If so, execute S607; otherwise, execute S609.

[0088] S607: The VR glasses directly render the received game interface on two display screens for display.

[0089] VR glasses achieve a 3D stereoscopic effect by rendering the game interface on two displays.

[0090] S608: The VR glasses determine the time interval based on two adjacent frames of the received game interface.

[0091] The method for determining the time interval refers to S301 and will not be repeated here.

[0092] S609: The VR glasses determine whether the time interval is less than at least one refresh interval of the screen. If so, execute S610; otherwise, return to S608 to recalculate the time interval.

[0093] In order to avoid the impact of network anomalies, when using the time interval for frame compensation, it is necessary to ensure that the time interval is less than at least one refresh interval of the screen.

[0094] S610: The VR glasses determine at least one compensated posture data according to the time interval, the preset number of compensation frames and the historical posture data set, wherein the compensated posture data is located between the posture data corresponding to two adjacent frames of real images.

[0095] The description of this step is referred to S302-S303 and will not be repeated here.

[0096] S611: The VR glasses predict and display the corresponding compensation interface based on the currently displayed game interface and various compensation posture data.

[0097] VR glasses achieve a 3D stereoscopic effect by rendering the compensation interface on two displays.

[0098] Based on the same technical concept, an embodiment of the present application provides a VR device, which can implement the method steps of image streaming display in the above embodiment and achieve the same technical effect.

[0099] See also Figure 7 The VR device includes a processor 701, a memory 702, a communication interface 703, and a display screen 704. The display screen 704, the communication interface 703, the memory 702 and the processor 701 are connected via a bus 705:

[0100] The memory 702 stores a computer program, and the processor 701 performs the following operations according to the computer program:

[0101] Receiving two adjacent frames of real images sent by a common terminal through the communication interface 703, and determining a time interval between the two adjacent frames of real images, wherein the real images are determined by the common terminal according to the posture data sent by the VR device;

[0102] Determining the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames;

[0103] For each prediction time, a compensation pose data is determined according to the historical pose data set, and the compensation pose data is located between the pose data corresponding to the two adjacent frames of real images;

[0104] Predicting a corresponding compensated image based on the real image currently displayed on the display screen 704 and each compensated posture data, and displaying the image on the display screen;

[0105] When the next frame of real image sent by the common terminal is received through the communication interface 703 , it is directly displayed on the display screen 704 .

[0106] Optionally, the processor 701 determines the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames, and the specific operation is:

[0107] According to the compensation frame number, the value interval (0, 1) is equally divided into a corresponding number of intervals;

[0108] Taking a target value from each interval, and multiplying the target value by the time interval;

[0109] The reception time of the first frame of the two adjacent frames of real images is taken as a reference, and each product is added respectively to obtain the prediction time corresponding to each frame of the image to be compensated.

[0110] Optionally, the processor 701 determines a compensation posture data according to the historical posture data set for each prediction time, and the specific operation is:

[0111] Fitting each historical posture data and the sending time of the historical posture data in the historical posture data set to obtain an objective function;

[0112] According to the objective function, a compensation posture data corresponding to the predicted time is determined.

[0113] Optionally, when the encoding frame rate of the common terminal is N times the decoding frame rate of the VR device, N is an integer greater than or equal to 2;

[0114] The processor 701 determines a compensation posture data according to the historical posture data set for each prediction time, and the specific operation is:

[0115] From the historical pose data set, historical pose data for which there is no corresponding real image is selected as the compensation pose data corresponding to the corresponding prediction time.

[0116] Optionally, the time interval is smaller than at least one refresh interval of the VR device screen.

[0117] Optionally, the real image is a game screen.

[0118] It should be noted that Figure 7 This is only an example, and provides the necessary hardware for a VR device to execute the steps of the image streaming display method provided in the embodiments of the present application. Not shown, the VR device also includes common hardware of a display device, such as a speaker, a microphone, left and right eyeglasses, etc.

[0119] Embodiments of the present application Figure 7 The processor involved may be a central processing unit (CPU), a general purpose processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The memory may be integrated in the processor or may be provided separately from the processor.

[0120] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0121] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0122] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0123] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for displaying an image stream, It is characterized in that Applied to VR devices, including: Acquire a time interval between two adjacent frames of real images received from a common terminal, where the real image is determined by the common terminal according to the posture data sent by the VR device; Determining the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames; For each prediction time, a compensation pose data is determined according to the historical pose data set, and the compensation pose data is located between the pose data corresponding to the two adjacent frames of real images; According to the currently displayed real image and each compensated posture data, the corresponding compensated image is predicted and displayed; When the next frame of real image sent by the ordinary terminal is received, it is directly displayed.

2. The method according to claim 1, It is characterized in that Determining the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames includes: According to the compensation frame number, the value interval (0, 1) is equally divided into a corresponding number of intervals; Taking a target value from each interval, and multiplying the target value by the time interval; The reception time of the first frame of the two adjacent frames of real images is taken as a reference, and each product is added respectively to obtain the prediction time corresponding to each frame of the image to be compensated.

3. The method according to claim 1, It is characterized in that The method of determining a compensation posture data according to the historical posture data set for each prediction time includes: Fitting each historical posture data and the sending time of the historical posture data in the historical posture data set to obtain an objective function; According to the objective function, a compensation posture data corresponding to the predicted time is determined.

4. The method according to claim 1, It is characterized in that When the encoding frame rate of the common terminal is N times the decoding frame rate of the VR device, N is an integer greater than or equal to 2; The method of determining a compensation posture data according to the historical posture data set for each prediction time includes: From the historical pose data set, historical pose data for which there is no corresponding real image is selected as the compensation pose data corresponding to the corresponding prediction time.

5. The method according to any one of claims 1 to 4, It is characterized in that The time interval is smaller than at least one refresh interval of the VR device screen.

6. The method according to any one of claims 1 to 4, It is characterized in that The real image is a game screen.

7. A VR device, It is characterized in that It includes a processor, a memory, a communication interface, and a display screen, wherein the display screen, the communication interface, the memory and the processor are connected via a bus: The memory stores a computer program, and the processor performs the following operations according to the computer program: Receiving two adjacent frames of real images sent by a common terminal through the communication interface, and determining a time interval between the two adjacent frames of real images, wherein the real images are determined by the common terminal according to the posture data sent by the VR device; Determining the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames; For each prediction time, a compensation pose data is determined according to the historical pose data set, and the compensation pose data is located between the pose data corresponding to the two adjacent frames of real images; According to the real image currently displayed on the display screen and each compensation posture data, a corresponding compensation image is predicted, and displayed on the display screen; When the next frame of real image sent by the ordinary terminal is received through the communication interface, it is directly displayed on the display screen.

8. The VR device according to claim 7, It is characterized in that The processor determines the prediction time corresponding to each frame of the image to be compensated according to the time interval and the preset number of compensation frames, and the specific operation is: According to the compensation frame number, the value interval (0, 1) is equally divided into a corresponding number of intervals; Taking a target value from each interval, and multiplying the target value by the time interval; The reception time of the first frame of the two adjacent frames of real images is taken as a reference, and each product is added respectively to obtain the prediction time corresponding to each frame of the image to be compensated.

9. The VR device according to claim 7, It is characterized in that The processor determines a compensation posture data according to the historical posture data set for each prediction time, and the specific operation is: Fitting each historical posture data and the sending time of the historical posture data in the historical posture data set to obtain an objective function; According to the objective function, a compensation posture data corresponding to the predicted time is determined.

10. The VR device according to claim 7, It is characterized in that When the encoding frame rate of the common terminal is N times the decoding frame rate of the VR device, N is an integer greater than or equal to 2; The processor determines a compensation posture data according to the historical posture data set for each prediction time, and the specific operation is: From the historical pose data set, historical pose data for which there is no corresponding real image is selected as the compensation pose data corresponding to the corresponding prediction time.

Images