Display method, device, equipment and storage medium

Abstract

The present application relates to the field of picture display technology, and in particular to a display method, apparatus, device and storage medium for solving the problem of picture delay in traditional solutions. The method comprises the following steps: obtaining a perspective change signal, and based on the perspective information included in the perspective change signal, obtaining an original picture frame corresponding to the perspective information, wherein the original picture frame is the next display frame to be displayed; obtaining a target picture frame based on the current display frame, the original picture frame and the delay duration, wherein the delay duration is the duration from obtaining any original picture frame to directly displaying the original picture frame on the display panel of the display device, and the current display frame is the picture frame currently displayed on the display panel; finally, replacing the original picture frame with the target picture frame for display; in this way, the picture delay caused by the software and hardware defects of the display device is effectively reduced without increasing the hardware computing power cost of the display device.

Description

Technical Field

[0001] The present application relates to the field of screen display technology, and in particular to a display method, device, equipment and storage medium. Background Art

[0002] With the continuous development of virtual reality and augmented reality technologies, people are provided with better visual experiences. Therefore, various virtual reality display devices and / or augmented reality display devices are gradually gaining popularity among the public.

[0003] However, due to limitations in screen characteristics, transmission speeds, and other hardware and software factors, existing virtual reality or augmented reality display devices are unable to provide real-time feedback on screen content based on user actions. They often require a certain amount of time after receiving a perspective change signal before presenting the current image to the user. For example, when a user wears a VR helmet and turns their head, the VR helmet collects the perspective change signal through a motion sensor, renders the image corresponding to the perspective change signal, and transmits the rendered image to the display panel, which then displays the rendered image. Therefore, there is a certain delay between the image displayed on the VR helmet's display panel after the user turns their head and their own movements.

[0004] Studies have shown that when the above delay is longer than 20ms, users will feel obvious image stickiness or dragging, and may also experience dizziness because the displayed image lags behind their own movements, which greatly affects the user experience.

[0005] Under existing technologies, the delay is usually reduced by increasing the refresh rate of the display device. However, a high refresh rate requires a processor with higher computing power to be deployed for the display device and more picture frames need to be rendered. This greatly increases the hardware cost of the display device, thereby reducing the universality of this solution. Summary of the Invention

[0006] Embodiments of the present application provide a display method, apparatus, device, and storage medium for improving the image display speed of a virtual reality or augmented reality display device, thereby effectively reducing image delays caused by software and hardware defects of the display device without increasing the hardware computing power cost of the display device.

[0007] The specific technical solutions provided in the embodiments of this application are as follows:

[0008] In a first aspect, an embodiment of the present application provides a display method, applied to a display device having a virtual reality or augmented reality display function, the method comprising:

[0009] Acquire a viewing angle change signal, and based on viewing angle information included in the viewing angle change signal, acquire an original picture frame corresponding to the viewing angle information, wherein the original picture frame is a next display frame to be displayed;

[0010] Obtaining a target frame based on the current display frame, the original frame, and a delay duration, wherein the delay duration is the duration from acquiring any of the original frames to directly displaying the original frame on a display panel of the display device, and the current display frame is the frame currently displayed on the display panel;

[0011] The target frame is used to replace the original frame for display.

[0012] By adopting the display method provided in the embodiment of the present application, since the target picture frame is obtained based on the current display frame, the original picture frame and the delay duration, the accuracy of the picture prediction is improved. Since the picture finally displayed on the display panel takes into account the delay duration when the display device displays the picture, the target picture frame finally displayed can be closer to the expected image after the user's actual action, thereby effectively reducing the picture delay problem caused by software and hardware defects without increasing the hardware computing power cost.

[0013] In one possible implementation, the delay duration is determined as follows:

[0014] Invoking a monitoring tool to obtain a first moment and a second moment, wherein the first moment is a moment when the original picture frame is acquired, and the second moment is a moment when the original picture frame is directly displayed on a display panel of the display device;

[0015] The difference between the second moment and the first moment is determined as the delay duration.

[0016] The above method uses the time difference between two frames to determine the delay duration, so as to accurately determine the motion vector corresponding to the target picture frame based on the delay ratio of the delay duration to the time interval between the two frames.

[0017] In a possible implementation, obtaining the target frame based on the current display frame, the original frame, and the delay duration includes:

[0018] determining a motion estimation vector between the current display frame and the original picture frame based on the current display frame and the original picture frame;

[0019] Determining a motion vector corresponding to the target picture frame based on the delay duration, the time interval, and the motion estimation vector;

[0020] The target picture frame is generated based on the motion vector.

[0021] The above method obtains the motion vector of the target frame based on the current display frame and the original frame through motion estimation technology and motion estimation compensation technology, and then generates the target frame based on the motion vector. In this way, the delay duration is compensated in the target frame to achieve the purpose of making the final displayed target frame closer to the expected image after the user's actual action.

[0022] In a possible implementation, after acquiring the original picture frame corresponding to the viewing angle information and before obtaining the target picture frame, the method further includes:

[0023] The time interval is determined based on a first moment when the original picture frame is acquired and a third moment corresponding to the currently displayed frame.

[0024] In the above method, since the frame rate of the video source may change, the time interval between the two current frames of the video, that is, the frame rate, can be calculated in real time, thereby facilitating the subsequent adaptive calculation of the motion vector corresponding to the target picture frame after the delay time at different frame rates, so as to ensure the accuracy of the target picture frame generated based on the motion vector.

[0025] In a possible implementation, determining the motion vector corresponding to the target picture frame based on the delay duration, the time interval, and the motion estimation vector includes:

[0026] Determining, based on the delay duration and the time interval, a delay ratio of the delay duration to the time interval;

[0027] Based on the delay ratio and the motion estimation vector, a motion vector corresponding to the target picture frame is determined.

[0028] In a possible implementation, determining the motion vector corresponding to the target picture frame based on the delay ratio and the motion estimation vector includes:

[0029] Obtaining a motion vector corresponding to the target picture frame based on the motion estimation vector and a first ratio, wherein the first ratio is a sum of a preset value and the delay ratio;

[0030] The generating the target picture frame based on the motion vector includes:

[0031] The target picture frame is generated based on the current display frame and the motion vector using a motion estimation and compensation algorithm.

[0032] In a possible implementation, determining the motion vector corresponding to the target picture frame based on the delay ratio and the motion estimation vector includes:

[0033] Obtaining a motion vector corresponding to the target picture frame based on the motion estimation vector and a second ratio, wherein the second ratio is equal to the delay ratio;

[0034] The generating the target picture frame based on the motion vector includes:

[0035] The target picture frame is generated based on the original picture frame and the motion vector using a motion estimation and compensation algorithm.

[0036] In a second aspect, an embodiment of the present application provides a display device, which is applied to a display device having a virtual reality or augmented reality display function, and the device includes:

[0037] an acquisition module, configured to acquire a viewing angle change signal, and based on viewing angle information included in the viewing angle change signal, acquire an original picture frame corresponding to the viewing angle information, wherein the original picture frame is a next display frame to be displayed;

[0038] an obtaining module, configured to obtain a target frame based on a current display frame, the original frame, and a delay duration, wherein the delay duration is the duration from obtaining any of the original frame to directly displaying the original frame on a display panel of the display device, and the current display frame is the frame currently displayed on the display panel;

[0039] The replacement display module is used to replace the original picture frame with the target picture frame for display.

[0040] In one possible implementation, the delay duration is determined as follows:

[0041] Invoking a monitoring tool to obtain a first moment and a second moment, wherein the first moment is a moment when the original picture frame is acquired, and the second moment is a moment when the original picture frame is directly displayed on a display panel of the display device;

[0042] The difference between the second moment and the first moment is determined as the delay duration.

[0043] In a possible implementation, the obtaining module is specifically configured to:

[0044] determining a motion estimation vector between the current display frame and the original picture frame based on the current display frame and the original picture frame;

[0045] Determining a motion vector corresponding to the target picture frame based on the delay duration, the time interval, and the motion estimation vector;

[0046] The target picture frame is generated based on the motion vector.

[0047] In a possible implementation, the obtaining module is further configured to:

[0048] After acquiring the original picture frame corresponding to the viewing angle information and before obtaining the target picture frame, the time interval is determined based on a first moment when the original picture frame is acquired and a third moment corresponding to the currently displayed frame.

[0049] In a possible implementation, the obtaining module is specifically configured to:

[0050] Determining, based on the delay duration and the time interval, a delay ratio of the delay duration to the time interval;

[0051] Based on the delay ratio and the motion estimation vector, a motion vector corresponding to the target picture frame is determined.

[0052] In a possible implementation, the obtaining module is specifically configured to:

[0053] Obtaining a motion vector corresponding to the target picture frame based on the motion estimation vector and a first ratio, wherein the first ratio is a sum of a preset value and the delay ratio;

[0054] The target picture frame is generated based on the current display frame and the motion vector using a motion estimation and compensation algorithm.

[0055] In a possible implementation, the obtaining module is specifically configured to:

[0056] Obtaining a motion vector corresponding to the target picture frame based on the motion estimation vector and a second ratio, wherein the second ratio is equal to the delay ratio;

[0057] The target picture frame is generated based on the original picture frame and the motion vector using a motion estimation and compensation algorithm.

[0058] In a third aspect, an embodiment of the present application provides a display device, including:

[0059] A display panel for displaying objects in a virtual reality or augmented reality scene;

[0060] Motion sensor, used to collect perspective change signals;

[0061] Memory, used to store computer programs or instructions;

[0062] A processor is configured to execute the computer program or instructions in the memory so that the method according to any one of the first aspects described above is performed.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which, when instructions in the storage medium are executed by a processor, enables the processor to execute any one of the methods described in the first aspect above.

[0064] In addition, the technical effects brought about by any implementation method in the second to fourth aspects can refer to the technical effects brought about by different implementation methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a schematic diagram of an application scenario in an embodiment of the present application;

[0066] Figure 2 This is a schematic diagram of another application scenario in an embodiment of the present application;

[0067] Figure 3 This is a schematic diagram of the logical structure of a wearable device in an embodiment of the present application;

[0068] Figure 4 This is a flow chart of a display method in an embodiment of the present application;

[0069] Figure 5 Schematic diagram of a flow chart of a method for determining a delay time in an embodiment of the present application;

[0070] Figure 6 Schematic diagram of a flow chart of a method for generating a target picture frame in an embodiment of the present application;

[0071] Figure 7 Schematic diagram of a flow chart of a method for determining a motion vector corresponding to a target picture frame in an embodiment of the present application;

[0072] Figure 8 Schematic diagram of the interaction process between components included in a wearable device in an embodiment of the present application;

[0073] Figure 9 This is a schematic diagram of a user rotating their head viewing angle in an embodiment of the present application;

[0074] Figure 10 This is a schematic diagram of the logical architecture of a display device according to an embodiment of the present application;

[0075] Figure 11 Schematic diagram of the physical structure of the display device in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] It should be noted that the terms "first," "second," "third," etc. in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0078] The following is a brief introduction to the design concept of the embodiments of this application.

[0079] The present application relates to the field of screen display technology, and mainly to a display method, device, equipment and storage medium.

[0080] Virtual Reality (VR) is a new technology that leverages computer graphics, simulation, multimedia, artificial intelligence, computer networking, parallel processing, and multi-sensor technologies to simulate the visual, auditory, and tactile functions of a target subject (such as a user). This allows the subject to immerse themselves in a computer-generated virtual world and interact with virtual objects in real time through natural means such as speech, gestures, and head movements. Augmented Reality (AR) is a technology that calculates the position and angle of a camera image in real time and adds corresponding graphics. It seamlessly integrates real-world and virtual world information.

[0081] The continuous development of virtual reality and augmented reality technologies has provided people with a better visual experience. As a result, various virtual reality and augmented reality display devices have gradually gained popularity. However, in actual use, due to limitations in display screen characteristics, transmission speed, image rendering speed, and other hardware and software factors, existing virtual reality and augmented reality display devices are unable to provide real-time feedback on the screen content based on user actions. It often takes a certain amount of time after receiving the signal before the image is presented to the user. Research has shown that when the delay time is greater than 20ms, users will experience noticeable image lag or dragging. Users may also experience dizziness due to the lag between the displayed image and their own movements, which greatly affects the user experience.

[0082] However, for virtual reality display devices or augmented reality display devices, it is very difficult to reduce the delay time to less than 20ms. The device must obtain user motion information through sufficiently accurate sensors, and timely obtain and render the picture content based on the motion information, and also need to display the picture on the display panel in a timely manner. Therefore, under the existing technology, one solution to reduce picture delay is to increase the refresh rate of the display device. However, general research shows that only when the refresh rate of the device reaches 120Hz can it stably avoid dizziness caused by delay. However, a high refresh rate requires the deployment of a processor with higher computing power and the rendering of more picture frames, which greatly increases the hardware cost. Therefore, the above solution is not universal.

[0083] In view of this, in order to effectively reduce the picture delay problem caused by the software and hardware defects of the display device without increasing the hardware computing power cost of the display device, an embodiment of the present application provides a display method, which is suitable for a display device with virtual reality or augmented reality display function. In the embodiment of the present application, a perspective change signal is obtained, and based on the perspective information included in the perspective change signal, the original picture frame corresponding to the perspective information is obtained, wherein the original picture frame is the next display frame to be displayed; based on the current display frame, the original picture frame and the delay time, the target picture frame is obtained, wherein the delay time is the time from obtaining any original picture frame to directly displaying the original picture frame on the display panel of the display device, and the current display frame is the picture frame currently displayed on the display panel; finally, the target picture frame replaces the original picture frame for display.

[0084] By adopting the display method provided by this application, since the target picture frame is obtained based on the current display frame, the original picture frame and the delay duration, the delay duration when the display device displays the picture is taken into account, which can effectively improve the virtual reality or augmented reality picture display, so that the final displayed target picture frame is closer to the expected image after the user's actual action, and effectively reduces the picture delay problem caused by software and hardware defects without increasing the hardware computing power cost.

[0085] The display method provided in the embodiment of the present application is applicable to a display device having a virtual reality or augmented reality display function.

[0086] In the embodiments of the present application, the display device can operate independently; illustratively, the display device can be a smart mobile terminal, tablet computer, laptop computer, smart handheld device, personal computer (PC), computer, smart screen, various wearable devices (including but not limited to VR glasses, VR helmets, VR gaming devices, AR devices, etc.), personal digital assistant (PDA), smart TV, etc. equipped with a motion sensor. It should be noted that the above-mentioned motion sensor can be integrated into the display device or can be connected to the display device wirelessly or wirelessly.

[0087] The following is a schematic diagram of an application scenario in an embodiment of the present application, taking the display device as a wearable device as an example. Figure 1 See Figure 1 As shown, the application scenario includes a wearable device 100. The wearable device 100 can be worn on the user's head and can execute the display process provided in the embodiment of the present application to provide the user with an immersive experience. In a specific implementation, the wearable device 100 acquires a perspective change signal through a motion sensor, and obtains the corresponding original picture frame based on the perspective information included in the perspective change signal, and obtains the target picture frame based on the current display frame, the original picture frame and the delay time, and then replaces the above-mentioned original picture frame with the target picture frame for display.

[0088] In an embodiment of the present application, the display device can also be connected to other smart display devices as an external device; illustratively, the display device can also be a server connected to a device with virtual reality or augmented reality display functions (such as various wearable devices), for example, it can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; wherein the server can be connected to the device with virtual reality or augmented reality display functions by wired or wireless means, and the present application does not limit the connection method.

[0089] The following is a schematic diagram of another application scenario in the embodiment of the present application, taking the display device as a server connected to a device with virtual reality or augmented reality display function (such as various wearable devices) as an example. Figure 2 See Figure 2As shown, the application scenario includes a server 200 and a wearable device 100 with a virtual reality or augmented reality display function, and the server 200 communicates with the wearable device 100 via the Internet. In the embodiment of the present application, after the wearable device 100 obtains the perspective change signal, it sends the perspective change signal to the server 200 via the Internet; the server 200 executes the display process provided by the embodiment of the present application, that is, obtains the original picture frame corresponding to the perspective information included in the perspective change signal, and obtains the target picture frame based on the current display frame, the original picture frame and the delay time. Then, the target picture frame replaces the original picture frame and is transmitted to the wearable device 100 via the Internet, so that the wearable device 100 displays the target picture frame.

[0090] It will be understood that the specific type of the display device is not limited in the embodiments of the present application.

[0091] After introducing the application scenarios of the embodiments of the present application, the preferred implementation methods of the present application are further described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments of the present application and the features in the embodiments can be combined with each other if there is no conflict.

[0092] The subsequent content of this application explains the technical solution using the display device as a wearable device as an example. It should be understood that the provided technical solution can also be applied to the other types of display devices mentioned above.

[0093] Figure 3 The schematic diagram of the architecture of the wearable device provided by the embodiment of the present application is exemplarily shown. Figure 3 As shown, the wearable device includes a motion sensor, a central processing unit (CPU), an image processing module and a display panel, wherein the motion sensor is used to collect a perspective change signal according to a preset frequency and send the perspective change signal to the image processing module, where the perspective change signal includes the perspective information of the current user; the image processing module includes a motion estimation and compensation module, which is used to call a motion estimation and compensation algorithm to determine the target picture frame.

[0094] The following is based on Figure 3 The structure of the wearable device shown is used to specifically describe the technical solution of this application. Figure 4 As shown, the embodiment of the present application provides a display method, the specific process of the method is as follows:

[0095] Step 300: Obtain a viewing angle change signal, and based on the viewing angle information included in the viewing angle change signal, obtain an original picture frame corresponding to the viewing angle information, wherein the original picture frame is the next display frame to be displayed.

[0096] In an embodiment of the present application, when executing step 300, the wearable device can obtain a perspective change signal through a motion sensor, and then obtain the original picture frame corresponding to the perspective information based on the perspective information included in the perspective change signal through an image processing module, wherein the original picture frame is the next display frame to be displayed.

[0097] In a specific implementation, after the user wears the wearable device, the motion sensor will collect a perspective change signal including the user's perspective information according to the display refresh rate of the display panel, and send the perspective change signal to the image processing module; then, after receiving the perspective change signal, the image processing module obtains the corresponding original picture frame based on the perspective information included in the perspective change signal, wherein the original picture frame can be a video frame in the virtual reality video stream corresponding to the perspective information; it can also be a video frame in the augmented reality video stream corresponding to the perspective information.

[0098] Step 310: Based on the current display frame, the original picture frame and the delay duration, a target picture frame is obtained, wherein the delay duration is the duration from obtaining any original picture frame to directly displaying the original picture frame on the display panel of the display device, and the current display frame is the picture frame currently displayed on the display panel.

[0099] In the embodiment of the present application, before executing step 310, refer to Figure 5 As shown, the above delay duration is obtained by executing the following steps:

[0100] Step 400: calling a monitoring tool to obtain a first moment and a second moment, wherein the first moment is the moment when the original picture frame is acquired, and the second moment is the moment when the original picture frame is directly displayed on the display panel of the display device.

[0101] In an embodiment of the present application, a monitoring tool is pre-configured in the CPU for monitoring the actual delay time from when the display device acquires the original picture frame to when the display device directly displays the original picture frame on the display panel of the display device, that is, the above-mentioned delay time.

[0102] Exemplarily, after the wearable device receives the first perspective change signal through the image processing module, step 400 is executed, and the monitoring tool is called through the CPU to monitor the moment when the image processing module obtains the original picture frame, that is, the first moment, and monitor the moment when the display panel directly displays the original picture frame, that is, the second moment.

[0103] Step 410: The difference between the second moment and the first moment is determined as the delay duration.

[0104] In the embodiment of the present application, when executing step 410, the CPU determines the difference between the second moment and the first moment as the above-mentioned delay duration.

[0105] In some feasible embodiments, if the display panel of the wearable device does not have a variable refresh rate, the above-mentioned delay duration can also be obtained in advance by analyzing the manufacturing process of the wearable device.

[0106] In the embodiment of the present application, after obtaining the above delay time through the CPU, refer to Figure 6 As shown, when executing step 310, the above function is realized by executing the following steps:

[0107] Step 3101: Based on the current display frame and the original picture frame, determine a motion estimation vector between the current display frame and the original picture frame.

[0108] In an embodiment of the present application, when executing step 3101, the wearable device calculates the motion estimation vector between the current display frame and the original picture frame based on the current display frame and the original picture frame using a motion estimation algorithm through the image processing module.

[0109] In the specific implementation, the block matching method in the motion estimation algorithm is taken as an example. In the embodiment of the present application, each frame is divided into a certain number of macroblocks by the image processing module, and any macroblock of the current display frame is matched one by one with each macroblock of the original picture frame, wherein each macroblock of the original picture frame can be regarded as a macroblock after the original macroblock (i.e., the macroblock of the current display frame) corresponds to a different displacement, thereby selecting a macroblock whose matching degree reaches the matching degree threshold, recording it as the target macroblock, and determining the motion estimation sub-vector corresponding to the macroblock based on the target macroblock and the macroblock of the current display frame; the same method as above is used to obtain the motion estimation sub-vector corresponding to each macroblock contained in the current display frame, and then, each obtained motion estimation sub-vector is spliced ​​according to the position of the corresponding macroblock in the current display frame to obtain the motion estimation vector between the current display frame and the original picture frame.

[0110] Step 3102: Determine the motion vector corresponding to the target picture frame based on the delay duration, the time interval and the motion estimation vector.

[0111] In the embodiment of the present application, before executing step 3102, it is necessary to determine the time interval based on the first moment when the original picture frame is acquired and the third moment corresponding to the currently displayed frame.

[0112] In a specific implementation, after the wearable device determines through the CPU that the original picture frame is obtained, the moment when the original picture frame is obtained is recorded as the first moment. Since the original picture frame is the original picture frame corresponding to the perspective information included in the received perspective conversion signal, the original picture frame is the next display frame to be displayed. Therefore, based on the third moment and the first moment corresponding to the obtained current display frame, the above-mentioned time interval can be obtained. Specifically, the difference between the third moment and the first moment is determined as the above-mentioned time interval.

[0113] It should be noted that the above-mentioned current display frame may be a picture frame that is determined by the image processing module based on the display method provided in the embodiment of the present application to replace the original picture frame for display (that is, the picture frame after motion estimation compensation), or it may be a picture frame that is directly displayed by the acquired original picture frame. Therefore, when the current display frame is a picture frame that is displayed by replacing the original picture frame with the target picture frame, the third moment corresponding to the above-mentioned current display frame is the moment of the original picture frame corresponding to the acquired current display frame, that is, the time interval is the time difference between the two acquired original picture frames, that is, the frame rate of the current video stream.

[0114] It should be noted that the above time interval is the time interval between two consecutive video frames obtained by the image processing module. If there is frame loss in the video stream, the above time interval is not the time interval between two consecutive video frames in the video stream.

[0115] In the embodiment of the present application, after the CPU determines the delay time and time interval, the delay time and time interval are sent to the image processing module, and then, referring to Figure 7 As shown, when the image processing module executes step 3102, it specifically implements the process of determining the motion vector corresponding to the target picture frame based on the delay duration, the time interval and the motion estimation vector by executing the following steps:

[0116] Step 31021: Based on the delay duration and the time interval, determine the delay ratio of the delay duration to the time interval.

[0117] In the embodiment of the present application, it is assumed that the delay duration is D and the time interval is T. Then, when the image processing module executes step 31021, the delay ratio S of the delay duration to the time interval can be obtained by the following formula:

[0118]

[0119] Step 31022: Determine the motion vector corresponding to the target picture frame based on the delay ratio and the motion estimation vector.

[0120] Step 3103: Generate a target picture frame based on the motion vector.

[0121] In the embodiment of the present application, assuming that the current display frame is an I frame, the image processing module obtains the original picture frame as a P frame based on the perspective information included in the received perspective conversion signal; the motion estimation vector between the current display frame (I frame) and the original picture frame (P frame) is determined to be And the latency ratio determined by the CPU is S.

[0122] Then, when the image processing module executes step 31022, it can obtain the motion vector corresponding to the target picture frame based on the motion estimation vector and the first ratio, where the first ratio is the sum of the preset value and the delay ratio; and send the motion vector and the current display frame to the motion estimation compensation module. The motion estimation compensation module executes step 3103 and can generate the above-mentioned target picture frame based on the current display frame and the motion vector using the motion estimation compensation algorithm.

[0123] In a specific implementation, the above preset value can be set to 1. Then, in the embodiment of the present application, the image processing module can obtain the motion vector corresponding to the target picture frame by the following formula:

[0124]

[0125] In some feasible embodiments, when executing step 31022, the image processing module can also obtain the motion vector corresponding to the target picture frame based on the motion estimation vector and the second ratio, where the second ratio is equal to the delay ratio; and send the motion vector and the original picture frame to the motion estimation compensation module. The motion estimation compensation module executes step 3103 and can generate the above-mentioned target picture frame based on the original picture frame and the motion vector using the motion estimation compensation algorithm.

[0126] In a specific implementation, the image processing module can obtain the motion vector corresponding to the target image frame through the following formula:

[0127]

[0128] In some preferred embodiments, when executing step 31022, the image processing module may further obtain a first motion vector based on the motion estimation vector and a third ratio, and a second motion vector based on the motion estimation vector and a fourth ratio, wherein the third ratio is the sum of a preset value and a delay ratio, and the fourth ratio is equal to the delay ratio. The current display frame and the first motion vector, as well as the original image frame and the second motion vector, are then sent to a motion estimation and compensation module. The motion estimation and compensation module executes step 3103, employing a motion estimation and compensation algorithm based on the current display frame and the first motion vector to obtain a first image frame, and employing a motion estimation and compensation algorithm based on the original image frame and the second motion vector to obtain a second image frame. The better image frame is then selected as the target image frame based on the first and second images. Exemplarily, the motion estimation and compensation module may select the image frame with the smallest sum of absolute errors as the target image frame.

[0129] Step 320: Replace the original frame with the target frame for display.

[0130] In a specific implementation, after the motion estimation and compensation module obtains the target picture frame, the target picture frame is sent to the image processing module. Before executing step 320, the image processing module renders the target picture frame. Then, step 320 is executed to replace the above-mentioned original picture frame with the rendered target picture frame and send it to the display panel. After receiving the rendered target picture frame, the display panel displays the rendered target picture frame. In this way, since the target picture frame is obtained based on the current display frame, the original picture frame and the delay duration, the delay duration when the display device directly displays the picture is taken into account, and the motion estimation compensation algorithm is used to compensate for the motion compensation corresponding to the above-mentioned delay duration in the target picture frame. Therefore, the target picture frame finally displayed can be closer to the expected image after the user's actual action, effectively shortening or avoiding the delay between the displayed picture and the user's actual action.

[0131] After introducing the display method provided by the embodiment of the present application, refer to Figure 8 As shown, Figure 8 The following figure shows the interaction process between the components of the wearable device:

[0132] Step 800: The motion sensor sends a first viewing angle change signal to the image processing module.

[0133] In an embodiment of the present application, after a user wears the wearable device, they press a power button to activate the wearable device. After the wearable device is activated, the motion sensor acquires a first perspective change signal, which includes the user's initial perspective information. The motion sensor then transmits the first perspective change signal including the initial perspective information to the image processing module.

[0134] Step 801: After receiving the first viewing angle change signal, the image processing module obtains an original picture frame corresponding to the initial viewing angle information contained in the first viewing angle change signal.

[0135] Step 802: The CPU calls the monitoring tool to obtain the first moment and the second moment, wherein the first moment is the moment when the original picture frame corresponding to the initial perspective information is obtained, and the second moment is the moment when the original picture frame corresponding to the initial perspective information is directly displayed on the display panel.

[0136] In the embodiment of the present application, after the user puts on the wearable device, he or she usually does not move immediately, but waits until the first picture frame is displayed in front of the user before moving to obtain more picture frames and enjoy an immersive experience.

[0137] Then, after the image processing module of the wearable device receives the first perspective change signal containing the initial perspective information, it obtains the original picture frame corresponding to the initial perspective information. The CPU can obtain the behavior of the image processing module obtaining the original picture frame corresponding to the initial perspective information through the bus connected to the image processing module, and obtain the behavior of the display panel directly displaying the original picture frame corresponding to the initial perspective information through the bus connected to the display panel; then, call the monitoring tool to monitor the moment when the image processing module obtains the original picture frame corresponding to the initial perspective information (i.e., the first moment), and monitor the moment when the display panel directly displays the original picture frame corresponding to the initial perspective information on the display panel (i.e., the second moment).

[0138] Step 803: The CPU determines the difference between the second moment and the first moment as the delay duration of the wearable device.

[0139] The delay time is recorded as D.

[0140] Step 804: The CPU sends the delay duration to the image processing module.

[0141] It should be noted that the above steps 801 to 804 are only examples. In actual applications, based on actual conditions, the acquisition and display process of the first video frame or the first few video frames after the wearable device is turned on can be selected to determine the above-mentioned delay duration of the wearable device; given that under normal circumstances, the above-mentioned delay duration of the device is fixed, it is not necessary to repeat the calculation in the subsequent execution process.

[0142] In the embodiment of the present application, in the subsequent process, only any one perspective change signal after the wearable device determines the above-mentioned delay time is used as an example to explain in detail the display method of the embodiment of the present application. The execution process for other perspective change signals is similar to this and will not be repeated here.

[0143] Assume that the user turns his head after seeing the image on the display panel, and the display refresh rate of the wearable device is 60 Hz.

[0144] Then, the motion sensor collects each perspective change signal during the user's movement at a frequency of 60 frames / s.

[0145] Step 805: The motion sensor sends a viewing angle change signal to the image processing module.

[0146] Step 806: After receiving the perspective change signal, the image processing module obtains the original picture frame corresponding to the target perspective information based on the target perspective information included in the perspective change signal.

[0147] In an embodiment of the present application, after receiving the perspective change signal, the image processing module obtains the original picture frame corresponding to the target perspective information included in the perspective change signal based on the perspective change signal, that is, the original picture frame obtained by the virtual perspective change, which is recorded as a P frame.

[0148] Assume that the picture frame currently displayed on the display panel of the wearable device is an I-frame. It should be noted that the I-frame can be an original picture frame corresponding to the initial viewing angle information, or an original picture frame corresponding to the target viewing angle information included in any previous viewing angle conversion signal, or a target picture frame determined based on the display method provided in the embodiment of the present application to replace the original picture frame corresponding to the target viewing angle information included in any previous viewing angle conversion signal.

[0149] Step 807: The image processing module determines a motion estimation vector between the current display frame and the original image frame corresponding to the target viewing angle information.

[0150] In the embodiment of the present application, the image processing module uses a motion estimation algorithm to determine the motion estimation vector between the current display frame and the original picture frame, which is recorded as

[0151] Step 808: The CPU determines the time interval between the two frames based on the original picture frame and the current display frame corresponding to the target viewing angle information.

[0152] In a specific implementation, the CPU can monitor the fourth moment (equivalent to the aforementioned first moment) when the original picture frame (P frame) is obtained, and the third moment corresponding to the current display frame (I frame). Then, a subtraction operation is performed on these two moments, that is, the fourth moment is subtracted from the third moment, so as to obtain the time interval between the original picture frame corresponding to the above-mentioned target perspective information and the current display frame or the original picture frame corresponding to the current display frame, which is recorded as T.

[0153] It should be noted that the above time interval is the time interval between two consecutive video frames obtained by the image processing module. If there is frame loss in the video stream, the above time interval is not the time interval between two consecutive video frames in the video stream.

[0154] In actual applications, since the frame rate of the video source may change, by executing step 808, the CPU can calculate in real time the time interval between the current two video frames in the video stream received by the image processing module, that is, the current frame rate of the video stream, thereby facilitating the subsequent adaptive calculation of the motion vector corresponding to the target picture frame after the delay time at different frame rates, so as to ensure the accuracy of the target picture frame generated based on the motion vector.

[0155] Step 809: The CPU sends the time interval to the image processing module.

[0156] Step 810: The image processing module determines the delay ratio of the delay duration to the time interval based on the delay duration and the time interval.

[0157] In the embodiment of the present application, when executing step 810, the image processing module divides the delay duration by the time interval to obtain the delay ratio, where the delay ratio is recorded as S, then

[0158] Step 811: The image processing module determines a motion vector corresponding to the target image frame based on the delay ratio and the current display frame.

[0159] In the embodiment of the present application, when executing step 811, the image processing module determines the motion vector corresponding to the target image frame by the following formula:

[0160]

[0161] In this way, by executing step 811, the current frame rate of the video can be automatically calculated in real time, so as to adaptively calculate the motion vector corresponding to the target picture frame after the delay time under different frame rates.

[0162] Step 812: The image processing module sends the motion vector and the current display frame to the motion estimation and compensation module.

[0163] Step 813: The motion estimation and compensation module generates a target picture frame based on the motion vector and the current display frame.

[0164] In the embodiment of the present application, when executing step 813, the motion estimation compensation module is based on And I frame, use motion estimation compensation algorithm to calculate the target picture frame, recorded as P_new frame.

[0165] Step 814: The motion estimation and compensation module sends the target picture frame to the image processing module.

[0166] Step 815: The image processing module replaces the original frame corresponding to the target viewing angle information with the target frame for rendering.

[0167] In an embodiment of the present application, when executing step 815, the image processing module replaces the original picture frame (P frame) with the target picture frame (P_new frame) for rendering processing, rather than rendering the original picture frame (P frame) corresponding to the target perspective information. In this way, the number of images that the image processing module needs to render is not increased, and the image processing capability of the wearable device is not greatly changed. Therefore, the universality of the display method provided in the embodiment of the present application is improved.

[0168] Step 816: The image processing module sends the rendered target image frame to the display panel.

[0169] In the embodiment of the present application, when executing step 816, the rendered target picture frame (P_new frame) is sent to the display panel instead of the original picture frame (P frame) originally corresponding to the target perspective information, so that the display panel can display it.

[0170] Step 817: The display panel displays the rendered target image frame.

[0171] In an embodiment of the present application, the picture contents of the above-mentioned I frame and P frame are used, and reference is made to the delay duration S of the wearable device and the time interval T between the current two video frames (i.e., the frame rate) to predict the next frame, and the predicted next frame (i.e., the target picture frame - P_new frame) replaces the P frame for display. Without increasing the frame rate, it can be ensured that the picture finally displayed on the display panel is more in line with the expected picture after the user performs the action.

[0172] It should be noted that, in the embodiment of the present application, there is no limitation on the order of executing the above-mentioned steps 808 and 807, and steps 808 to 809 can be executed before or at the same time as step 807. Therefore, steps 808 to 809 can be executed at any time before executing step 810 after executing step 806, that is, after obtaining the target picture frame corresponding to the target perspective information, and the technical solutions of the present application can be executed smoothly.

[0173] In order to facilitate further understanding of the display method provided in the embodiment of the present application, a simple example is used below to compare the difference in the images presented in the display panel between the traditional solution and the solution of the present application.

[0174] In order to make the results more intuitive, the relevant parameters of this example are specially specified. Assume that the user is wearing a head-mounted display device with virtual reality or augmented reality display function, and rotates 90 degrees horizontally to the right from the front, that is, the screen angle is rotated 90 degrees uniformly, such as Figure 9 shown.

[0175] Assume that the delay time D of the display device is 8 ms, and the time interval T between every two frames is equal to the delay time D, that is, 8 ms.

[0176] When using the traditional solution, since the delay duration D in this example is consistent with the time interval T between each two frames, when the user's head rotates 30°, the display device cannot present the 30° viewing angle picture in real time; and when the user's head rotates to 60°, the display device will fully display the picture obtained by the display device based on the 30° viewing angle. In this way, as the user's head continues to rotate, the picture presented on the display panel of the display device will always be delayed, that is, the picture presented in the display panel is the picture corresponding to the viewing angle that lags behind the user's current viewing angle by 30°.

[0177] However, when the display method provided in the present application is used, when the user's head rotates 30°, the display device does not display the original picture frame with a 30° viewing angle; the display device obtains the target picture frame based on the original picture frame with a 30° viewing angle, the current display frame with a 0° viewing angle, and the delay duration D.

[0178] In the specific implementation, based on the current display frame of 0° viewing angle and the original picture frame of 30° viewing angle, the motion estimation vector between the two frames is obtained. Then, based on the delay duration D and the time interval T, the delay ratio of the display device is determined, that is,

[0179] Then, based on the delay ratio and the motion estimation vector Determine the motion vector corresponding to the target frame, that is Then, based on the motion vector and the current display frame with a viewing angle of 0° to obtain a target picture frame, and the target picture frame replaces the original picture frame with a viewing angle of 30° and is transmitted to the display panel, so that the display panel displays the target picture frame.

[0180] At this time, since the delay duration D is consistent with the time interval T between each two frames, when the target picture frame is presented on the display panel, the user's head also happens to rotate to a 60° viewing angle position. In this way, using the display method provided in the embodiment of the present application, the display device only has a mismatch between the picture seen by the user and the action performed by the user when displaying the first two frames. During the subsequent user movement process, the consistency between the picture displayed by the display device and the actual action performed by the user can be ensured, offsetting the picture stickiness and dragging caused by the delay duration of the display device.

[0181] Based on the same inventive concept, see Figure 10 As shown, an embodiment of the present application provides a display device, which is applied to a display device with a virtual reality or augmented reality display function, and the device includes:

[0182] An acquisition module 1010 is configured to acquire a viewing angle change signal, and based on viewing angle information included in the viewing angle change signal, acquire an original picture frame corresponding to the viewing angle information, wherein the original picture frame is a next display frame to be displayed;

[0183] an obtaining module 1020 configured to obtain a target frame based on a current display frame, the original frame, and a delay duration, wherein the delay duration is a duration from when any original frame is acquired to when the original frame is directly displayed on a display panel of the display device, and the current display frame is a frame currently displayed on the display panel;

[0184] The replacement display module 1030 is configured to replace the original frame with the target frame for display.

[0185] In one possible implementation, the delay duration is determined as follows:

[0186] Invoking a monitoring tool to obtain a first moment and a second moment, wherein the first moment is a moment when the original picture frame is acquired, and the second moment is a moment when the original picture frame is directly displayed on a display panel of the display device;

[0187] The difference between the second moment and the first moment is determined as the delay duration.

[0188] In a possible implementation, the obtaining module 1020 is specifically configured to:

[0189] determining a motion estimation vector between the current display frame and the original picture frame based on the current display frame and the original picture frame;

[0190] Determining a motion vector corresponding to the target picture frame based on the delay duration, the time interval, and the motion estimation vector;

[0191] The target picture frame is generated based on the motion vector.

[0192] In a possible implementation, the obtaining module 1020 is further configured to:

[0193] After acquiring the original picture frame corresponding to the viewing angle information and before obtaining the target picture frame, the time interval is determined based on a first moment when the original picture frame is acquired and a third moment corresponding to the currently displayed frame.

[0194] In a possible implementation, the obtaining module 1020 is specifically configured to:

[0195] Determining, based on the delay duration and the time interval, a delay ratio of the delay duration to the time interval;

[0196] Based on the delay ratio and the motion estimation vector, a motion vector corresponding to the target picture frame is determined.

[0197] In a possible implementation, the obtaining module 1020 is specifically configured to:

[0198] Obtaining a motion vector corresponding to the target picture frame based on the motion estimation vector and a first ratio, wherein the first ratio is a sum of a preset value and the delay ratio;

[0199] The target picture frame is generated based on the current display frame and the motion vector using a motion estimation and compensation algorithm.

[0200] In a possible implementation, the obtaining module 1020 is specifically configured to:

[0201] Obtaining a motion vector corresponding to the target picture frame based on the motion estimation vector and a second ratio, wherein the second ratio is equal to the delay ratio;

[0202] The target picture frame is generated based on the original picture frame and the motion vector using a motion estimation and compensation algorithm.

[0203] Based on the same inventive concept, a display device is provided in an embodiment of the present application. The structure of the display device can be as follows: Figure 11 Shown, including:

[0204] Display panel 1110, for displaying objects in a virtual reality or augmented reality scene;

[0205] Motion sensor 1120, used to collect perspective change signals;

[0206] Memory 1130, for storing computer programs or instructions;

[0207] The processor 1140 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1140 is configured to call the program code stored in the memory 1130 to implement the following process: obtaining a perspective change signal, and based on the perspective information included in the perspective change signal, obtaining an original picture frame corresponding to the perspective information, wherein the original picture frame is the next display frame to be displayed; obtaining a target picture frame based on the current display frame, the original picture frame, and a delay duration, wherein the delay duration is the duration from obtaining any of the original picture frames to directly displaying the original picture frame on the display panel of the display device, and the current display frame is the picture frame currently displayed on the display panel; and replacing the original picture frame with the target picture frame for display.

[0208] The specific connection medium between the display panel 1110, the motion sensor 1120, the memory 1130 and the processor 1140 is not limited in the embodiment of the present application. Figure 11 In FIG, the display panel 1110, the motion sensor 1120, the memory 1130 and the processor 1140 are connected via a bus 1150. The connection between other components is only for schematic illustration and is not intended to be limiting. The bus 1150 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0209] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments.

[0210] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0211] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each flow and / or box in the flow chart and / or block diagram, as well as the combination of the flow chart and / or box in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one flow chart or multiple flows and / or one box or multiple boxes in the block diagram.

[0212] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0213] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0214] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A display method, characterized in that: Applied to a display device having a virtual reality or augmented reality display function, the method includes: Acquire a viewing angle change signal, and based on viewing angle information included in the viewing angle change signal, acquire an original picture frame corresponding to the viewing angle information, wherein the original picture frame is a next display frame to be displayed; Determining a motion estimation vector between the current display frame and the original picture frame based on a current display frame and the original picture frame, wherein the current display frame is a picture frame currently displayed on a display panel of the display device; Determining a motion vector based on a ratio of the delay duration to the time interval and the motion estimation vector; generating a target picture frame based on the motion vector, wherein the delay duration is a time period from when any one of the original picture frames is acquired to when the original picture frame is directly displayed on a display panel of the display device, and the time interval is a difference between a third moment corresponding to the currently displayed frame and a first moment when the original picture frame is acquired; The target frame is used to replace the original frame for display.

2. The method according to claim 1, wherein The delay duration is determined as follows: Invoking a monitoring tool to obtain a first moment and a second moment, wherein the first moment is a moment when the original picture frame is acquired, and the second moment is a moment when the original picture frame is directly displayed on a display panel of the display device; The difference between the second moment and the first moment is determined as the delay duration.

3. The method according to claim 1 or 2, wherein: The determining of the motion vector based on the delay ratio of the delay duration to the time interval and the motion estimation vector includes: Determining, based on the delay duration and the time interval, a delay ratio of the delay duration to the time interval; The motion vector is determined based on the delay ratio and the motion estimation vector.

4. The method according to claim 3, wherein Determining the motion vector based on the delay ratio and the motion estimation vector includes: Obtaining the motion vector based on the motion estimation vector and a first ratio, wherein the first ratio is a sum of a preset value and the delay ratio; Generating a target picture frame based on the motion vector includes: The target picture frame is generated based on the current display frame and the motion vector using a motion estimation and compensation algorithm.

5. The method according to claim 3, wherein Determining the motion vector based on the delay ratio and the motion estimation vector includes: Obtaining the motion vector based on the motion estimation vector and a second ratio, wherein the second ratio is equal to the delay ratio; Generating a target picture frame based on the motion vector includes: The target picture frame is generated based on the original picture frame and the motion vector using a motion estimation and compensation algorithm.

6. A display device, characterized in that: Applicable to a display device having a virtual reality or augmented reality display function, the device comprises: an acquisition module, configured to acquire a viewing angle change signal, and based on viewing angle information included in the viewing angle change signal, acquire an original picture frame corresponding to the viewing angle information, wherein the original picture frame is a next display frame to be displayed; an obtaining module, configured to determine, based on a current display frame and the original picture frame, a motion estimation vector between the current display frame and the original picture frame, wherein the current display frame is a picture frame currently displayed on a display panel of the display device; determine a motion vector based on a delay ratio of a delay duration to a time interval and the motion estimation vector; and generate a target picture frame based on the motion vector, wherein the delay duration is a time duration from when any one of the original picture frames is acquired to when the original picture frame is directly displayed on the display panel of the display device, and the time interval is a difference between a third moment corresponding to the current display frame and a first moment when the original picture frame is acquired; The replacement display module is used to replace the original picture frame with the target picture frame for display.

7. A display device, characterized in that: include: A display panel for displaying objects in a virtual reality or augmented reality scene; Motion sensor, used to collect perspective change signals; Memory, used to store computer programs or instructions; A processor is configured to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 5 is performed.

8. A computer-readable storage medium, characterized in that When the instructions in the storage medium are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 5.

Images