A frame rate adjustment method, device, apparatus and storage medium

Abstract

The application relates to the technical field of image processing, in particular to a frame rate adjusting method and device, equipment and a storage medium, which are used for solving the problem that the frame rate of an existing video cannot adapt to a display device with a high refresh rate and a variable refresh rate in a traditional scheme. The method is as follows: upon receiving each original video frame in a to-be-displayed video, a motion vector between the original video frame and a target video frame is obtained based on the original video frame and the target video frame, wherein the target video frame is a previous frame of the original video frame in the to-be-displayed video; then, when it is determined that a motion scene corresponding to the original video frame and the target video frame is a preset scene based on the motion vector, a target refresh rate of the display device is determined based on the motion vector, and an interpolation operation is performed between the original video frame and the target video frame based on the target refresh rate and a frame rate of the to-be-displayed video, so as to adjust the frame rate of the to-be-displayed video and adapt to the display requirement of the display device with the high refresh rate and the variable refresh rate.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a frame rate adjustment method, apparatus, device and storage medium. Background Technology

[0002] The refresh rate of display devices remains a significant cause of dizziness during user experience. To reduce this, more and more manufacturers are increasing the refresh rates of their displays and incorporating variable refresh rate technology.

[0003] However, existing video frame rates remain at 24fps or 30fps. To achieve higher frame rates, the hardware performance of the shooting equipment needs to be improved. But the hardware update cycle of shooting equipment is usually long, which cannot meet actual needs. Therefore, there is an urgent need for a frame rate adjustment method to dynamically adjust the video frame rate. Summary of the Invention

[0004] This application provides a frame rate adjustment method, apparatus, device, and storage medium for dynamically adjusting the frame rate of a video to adapt to display devices with variable refresh rates.

[0005] The specific technical solutions provided in this application are as follows:

[0006] In a first aspect, embodiments of this application provide a frame rate adjustment method, applied to a display device with a variable refresh rate, the method comprising:

[0007] For each original video frame received in the video to be displayed, a motion vector between the original video frame and the target video frame is obtained based on the original video frame and the target video frame, wherein the target video frame is the frame preceding the original video frame in the video to be displayed;

[0008] Based on the motion vector, determine whether the motion scene corresponding to the original video frame and the target video frame is a preset scene;

[0009] If so, the target refresh rate of the display device is determined based on the motion vector, and frame interpolation is performed between the original video frame and the target video frame based on the target refresh rate and the frame rate of the video to be displayed, so as to adjust the frame rate of the video to be displayed.

[0010] The frame rate adjustment method provided in this application determines motion vectors based on the original video frames and historical video frames (i.e., the target video frame, the frame preceding the original video frame) in the video to be displayed. Then, when it is determined that the current motion scene is a preset scene based on the motion vectors, frame interpolation is performed between the original video frames and historical video frames based on the motion vectors, thereby dynamically adjusting the video frame rate to adapt to the display requirements of devices with high refresh rates and / or variable refresh rates. Furthermore, for the above-mentioned display devices with virtual reality or augmented reality, the frame rate adjustment method provided in this application can also reduce dizziness during user use and provide a better user experience.

[0011] In one possible implementation, after receiving each original video frame from the video to be displayed, the method further includes:

[0012] Determine whether the target video frame exists in the cache space;

[0013] If not, the original video frame is cached in the cache space as the target video frame;

[0014] If so, the target video frame is retrieved from the cache space, and the original video frame is cached in the cache space as the new target video frame.

[0015] By caching a historical video frame in the cache space, motion vectors can be determined subsequently based on the historical frame information in the video to be displayed.

[0016] In one possible implementation, determining whether the motion scene corresponding to the original video frame and the target video frame is a preset scene based on the motion vector includes:

[0017] Based on the motion vector, determine the magnitude of the vector for each pixel block;

[0018] Based on the magnitude of each pixel block vector and a preset grouping rule, the multiple pixel block vectors included in the motion vector are grouped.

[0019] Count the number of pixel block vectors contained in each group, and determine the group with the most pixel block vectors as the target group;

[0020] If the number of pixel block vectors contained in the target group is greater than a preset value, then the motion scene corresponding to the original video frame and the target video frame is determined to be the preset scene;

[0021] If the number of pixel block vectors contained in the target group is not greater than the preset value, then it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene.

[0022] By determining whether the motion scene corresponding to the currently received original video frame and the target video frame is a preset scene, it is determined whether to perform frame interpolation operation, thereby dynamically adjusting the frame rate in the video to be displayed.

[0023] In one possible implementation, determining the target refresh rate of the display device based on the motion vector includes:

[0024] Based on the pixel block vectors contained in the target group, determine the target vector corresponding to the target group;

[0025] Based on the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, the target refresh rate of the display device corresponding to the target vector is determined, wherein the amplitude of the target vector is positively correlated with the target refresh rate of the display device.

[0026] Since the display device has a variable refresh rate, by configuring the correspondence between the vector amplitude range and the target refresh rate of the display device, after determining that the motion scene corresponding to the original video frame and the target video frame is a preset scene, the target refresh rate of the display device is determined based on the size of the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, so as to adjust the frame rate of the video to be displayed.

[0027] In one possible implementation, the frame interpolation operation between the original video frame and the target video frame based on the target refresh rate and the frame rate of the video to be displayed includes:

[0028] Based on the target refresh rate and the frame rate of the video to be displayed, determine the number of interpolation frames that need to be inserted between the original video frame and the target video frame;

[0029] Based on the number of frames, the original video frames, and / or the target video frames, an interpolation algorithm is invoked to obtain each interpolated frame;

[0030] The interpolated frames are inserted between the original video frame and the target video frame.

[0031] By determining the target refresh rate and the frame rate of the video to be displayed, the number of interpolation frames to be inserted between the original video frame and the target video frame is determined. Then, the frame interpolation algorithm is called to obtain each interpolation frame to be inserted between the original video frame and the target video frame. Each interpolation frame is then inserted between the original video frame and the target video frame, thus adjusting the frame rate of the video to be displayed.

[0032] In one possible implementation, after inserting the interpolated frames between the original video frame and the target video frame, the method further includes:

[0033] According to the playback order of the video frames of the video to be displayed, the target video frame, each interpolated frame and the original video frame are rendered and displayed sequentially.

[0034] In one possible implementation, the method further includes:

[0035] If it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene, then the target video frame and the original video frame are rendered and displayed sequentially according to the playback order of the video frames to be displayed.

[0036] In this embodiment of the application, if it is determined that the motion scene corresponding to the original video frame and the target video frame is not a preset scene, the frame rate of the video to be displayed will not be adjusted, thereby realizing dynamic adjustment of the frame rate of the video to be displayed.

[0037] Secondly, embodiments of this application provide a frame rate adjustment device, applied to a display device with a variable refresh rate, the device comprising:

[0038] The module is configured to, upon receiving an original video frame from the video to be displayed, obtain a motion vector between the original video frame and the target video frame based on the original video frame and the target video frame, wherein the target video frame is the frame preceding the original video frame in the video to be displayed;

[0039] The determination module is used to determine, based on the motion vector, whether the motion scene corresponding to the original video frame and the target video frame is a preset scene;

[0040] The frame interpolation module is configured to, if so, determine the target refresh rate of the display device based on the motion vector, and perform frame interpolation between the original video frame and the target video frame based on the target refresh rate and the frame rate of the video to be displayed, so as to adjust the frame rate of the video to be displayed.

[0041] In one possible implementation, the obtaining module is further used for:

[0042] After receiving each original video frame from the video to be displayed, it is determined whether the target video frame is in the buffer space;

[0043] If not, the original video frame is cached in the cache space as the target video frame;

[0044] If so, the target video frame is retrieved from the cache space, and the original video frame is cached in the cache space as the new target video frame.

[0045] In one possible implementation, when performing the step of determining whether the motion scene corresponding to the original video frame and the target video frame is a preset scene based on the motion vector, the determining module is specifically used for:

[0046] Based on the motion vector, determine the magnitude of the vector for each pixel block;

[0047] Based on the magnitude of each pixel block vector and a preset grouping rule, the multiple pixel block vectors included in the motion vector are grouped.

[0048] Count the number of pixel block vectors contained in each group, and determine the group with the most pixel block vectors as the target group;

[0049] If the number of pixel block vectors contained in the target group is greater than a preset value, then the motion scene corresponding to the original video frame and the target video frame is determined to be the preset scene;

[0050] If the number of pixel block vectors contained in the target group is not greater than the preset value, then it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene.

[0051] In one possible implementation, when performing the determination of the target refresh rate of the display device based on the motion vector, the frame interpolation module is specifically used for:

[0052] Based on the pixel block vectors contained in the target group, determine the target vector corresponding to the target group;

[0053] Based on the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, the target refresh rate of the display device corresponding to the target vector is determined, wherein the amplitude of the target vector is positively correlated with the target refresh rate of the display device.

[0054] In one possible implementation, the frame interpolation module is specifically used for:

[0055] Based on the target refresh rate and the frame rate of the video to be displayed, determine the number of interpolation frames that need to be inserted between the original video frame and the target video frame;

[0056] Based on the number of frames, the original video frames, and / or the target video frames, an interpolation algorithm is invoked to obtain each interpolated frame;

[0057] The interpolated frames are inserted between the original video frame and the target video frame.

[0058] In one possible implementation, the frame interpolation module is further configured to:

[0059] After the interpolated frames are inserted between the original video frame and the target video frame, the target video frame, the interpolated frames, and the original video frame are rendered and displayed sequentially according to the playback order of the video frames to be displayed.

[0060] In one possible implementation, the determining module is further configured to:

[0061] If it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene, then the target video frame and the original video frame are rendered and displayed sequentially according to the playback order of the video frames to be displayed.

[0062] Thirdly, embodiments of this application provide a display device, including:

[0063] A display screen is used to display images.

[0064] Memory is used to store computer programs or instructions;

[0065] A processor for executing a computer program or instructions in the memory, such that the method described in any of the first aspects above is performed.

[0066] Fourthly, embodiments of this application provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first aspects above.

[0067] Furthermore, the technical effects of any of the implementation methods in the second to fourth aspects can be found in the technical effects of different implementation methods in the first aspect, and will not be repeated here. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of an application scenario in the embodiments of this application;

[0069] Figure 2 This is a schematic diagram of the hardware structure of the VR head-mounted display device in the embodiments of this application;

[0070] Figure 3 This is a schematic diagram of the logical architecture of a VR head-mounted display device according to an embodiment of this application;

[0071] Figure 4 This is a flowchart illustrating a frame rate adjustment method according to an embodiment of this application;

[0072] Figure 5 This is a flowchart illustrating a method for determining or acquiring a target video frame according to an embodiment of this application.

[0073] Figure 6 This is a flowchart illustrating a method for determining whether a motion scene is a preset scene in an embodiment of this application.

[0074] Figure 7 This is a schematic diagram illustrating the process of determining whether a motion scene is a preset scene in an embodiment of this application;

[0075] Figure 8 This is a schematic flowchart illustrating a method for determining the target refresh rate of a display device according to an embodiment of this application.

[0076] Figure 9 This is a flowchart illustrating a frame interpolation operation in an embodiment of this application;

[0077] Figure 10 This is a schematic diagram of the logical architecture of a frame rate adjustment device in an embodiment of this application;

[0078] Figure 11 This is a schematic diagram of the physical architecture of the display device in the embodiments of this application. Detailed Implementation

[0079] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0080] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in sequences other than those illustrated or described herein.

[0081] The design concept of the embodiments of this application will be briefly introduced below.

[0082] In the display device industry, refresh rate remains a significant cause of dizziness for users. To reduce this, more and more manufacturers are increasing the refresh rate of their displays and incorporating variable refresh rate technology.

[0083] Currently, video frame rates remain at 24fps and 30fps. To adapt to high refresh rate devices and variable refresh rate devices, higher frame rates are needed, such as 60fps, 120fps, and 240fps. However, achieving higher frame rates places higher demands on the hardware performance of the shooting equipment. Since the hardware update cycle for shooting equipment is typically long, it's impossible to obtain shooting equipment with superior hardware performance in a relatively short time. Therefore, a frame rate adjustment method is urgently needed to dynamically adjust the video frame rate.

[0084] In view of this, in order to solve the problem that the frame rate of existing videos cannot adapt to display devices with high refresh rates and variable refresh rates, this application provides a frame rate adjustment method applied to display devices with variable refresh rates. In this application embodiment, each time an original video frame of a video to be displayed is received, a motion vector between the original video frame and the target video frame is obtained based on the original video frame and the target video frame, wherein the target video frame is the frame preceding the original video frame in the video to be displayed; then, when it is determined based on the motion vector that the motion scene corresponding to the original video frame and the target video frame is a preset scene, the target refresh rate of the display device is determined based on the motion vector, and a frame interpolation operation is performed between the original video frame and the target video frame based on the target refresh rate and the frame rate of the video to be displayed, so as to adjust the frame rate of the video to be displayed.

[0085] The frame rate adjustment method provided in this application determines motion vectors based on the original video frames and historical video frames (i.e., the target video frame, the frame preceding the original video frame). Then, when the current motion scene is determined to be a preset scene based on the motion vectors, frame interpolation is performed between the original video frames and historical video frames to dynamically adjust the video frame rate, thereby adapting to the display needs of devices with high refresh rates and variable refresh rates. Furthermore, for the aforementioned display devices with virtual reality or augmented reality, the frame rate adjustment method provided in this application can also reduce dizziness during use, providing a better user experience.

[0086] With the rapid development of computer technology and the increase in wireless data transmission, Virtual Reality (VR) and Augmented Reality (AR) are being applied to more and more fields. VR is a new technology that utilizes computer graphics, simulation technology, multimedia technology, artificial intelligence technology, computer network technology, parallel processing technology, and multi-sensor technology to simulate the visual, auditory, and tactile sensory functions of a target object (such as a user), allowing the target object to immerse itself in a computer-generated virtual world and interact with virtual objects in real time through natural means such as language, gestures, and head movements. AR is a technology that calculates the position and angle of camera images in real time and adds corresponding images, representing a new technology that seamlessly integrates real-world and virtual-world information.

[0087] In this embodiment of the application, only a VR head-mounted display device with a variable refresh rate is used as an example to describe the above frame rate adjustment method in detail. However, it should be noted that the technical solution provided in this application can also be applied to other types of display devices with variable refresh rates, such as smartphones, tablets, desktop computers, laptops, smart TVs and other devices that can play videos and interact.

[0088] Figure 1 An exemplary diagram illustrating an application scenario provided by an embodiment of this application is shown. For example... Figure 1 As shown, this application scenario includes a VR head-mounted display device 100. The VR head-mounted display device 100 can be worn on a user's head and can execute the frame rate adjustment process provided in this application embodiment, thereby dynamically adjusting the video frame rate to provide the user with a better immersive experience.

[0089] Figure 2 An exemplary schematic diagram of the hardware structure of a VR head-mounted display device 100 provided in an embodiment of this application is shown. Figure 2 As shown, the VR head-mounted display device 100 includes a lens group 101 and a two-dimensional display screen 102 disposed directly in front of the lens group 101. The lens group 101 consists of a left display lens 101_1 and a right display lens 101_2. When using the VR head-mounted display device 100, the user's eyes can view the video displayed on the display screen 102 through the lens group 101 and experience the VR effect.

[0090] After introducing the application scenarios of the embodiments of this application and the structural schematic diagram of the VR head-mounted display device, the preferred embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

[0091] Figure 3 An exemplary schematic diagram of the logical architecture of a VR head-mounted display device provided in an embodiment of this application is shown. (See also...) Figure 3 As shown, the VR head-mounted display device includes an image processing module and a display screen. The processing module includes a frame rate detection module, a motion estimation module, a scene recognition module, a frame interpolation module, and an image rendering module.

[0092] The following is based on Figure 3 The logical architecture of the VR head-mounted display device shown herein provides a detailed explanation of the technical solution of this application. (See also...) Figure 4 As shown in the figure, this application provides a frame rate adjustment method applied to the above-mentioned VR head-mounted display device. The specific process of the method is as follows:

[0093] Step 400: For each original video frame received from the video to be displayed, obtain the motion vector between the original video frame and the target video frame based on the original video frame and the target video frame, where the target video frame is the frame preceding the original video frame in the video to be displayed.

[0094] In this embodiment, the video to be displayed is input into the VR head-mounted display device, and the VR head-mounted display device performs step 400, receiving each original video frame of the video to be displayed through the image processing module in the VR head-mounted display device.

[0095] In practice, the frame rate detection module receives each video frame from the video to be displayed, detects the frame rate of the video, and after receiving each original video frame from the video to be displayed, refers to... Figure 5 As shown, further perform the following steps:

[0096] Step 4001: Determine if the target video frame exists in the buffer space. If not, proceed to step 4002; if yes, proceed to step 4003.

[0097] In this embodiment of the application, when performing step 4001, the image processing module obtains the target video frame from the cache space of the frame rate detection module. If the target video frame is not obtained from the cache space, it is determined that there is no target video frame in the cache space, indicating that the original video frame is the first video frame in the video to be displayed, and step 4002 is executed. If the target video frame is obtained from the cache space, it is determined that there is a target video frame in the cache space, indicating that the original video frame is not the first video frame in the video to be displayed, and step 4003 is executed.

[0098] Step 4002: Cache the original video frame as the target video frame into the cache space.

[0099] In this embodiment of the application, since there is no target video frame in the cache space, the received original video frame is the first video frame in the video to be displayed. Therefore, step 4002 is executed to cache the original video frame as the target video frame in the cache space and the process ends.

[0100] Step 4003: Obtain the target video frame from the cache space and cache the original video frame as the new target video frame in the cache space.

[0101] In this embodiment of the application, since there is a target video frame in the cache space, the received original video frame is not the first video frame in the video to be displayed. Therefore, step 4003 is executed, the target video frame is obtained from the cache space through the frame rate detection module, and the original video frame is cached as a new target video frame in the cache space of the frame rate detection module. The original video frame and the target video frame obtained from the cache space are sent together to the motion estimation module.

[0102] Then, in this embodiment of the application, after the motion estimation module receives the original video frame and the target video frame, it executes step 400 to obtain the motion vector between the original video frame and the target video frame based on the original video frame and the target video frame. In specific implementation, when executing step 400, the motion estimation module calls the motion estimation algorithm based on the original video frame and the target video frame to obtain the motion vector between the original video frame and the target video frame.

[0103] To facilitate understanding of motion estimation algorithms, a brief introduction is given below using block matching as an example. Motion estimation refers to block-based motion estimation. The basic idea is to divide each frame of an image sequence into many non-overlapping blocks, assuming that all pixels within a block have the same displacement. Then, each macroblock in the current frame is matched one-to-one with candidate macroblocks within a specified range in the reference frame. Based on the block matching criteria, the block in the reference frame that is most similar to the current block is found; this is the matching block. The relative displacement between the matching block and the current block is the motion vector. It should be noted that the above example only uses block matching; in this embodiment, other methods in motion estimation algorithms can also be used.

[0104] In this embodiment of the application, after obtaining the motion vector between the original video frame and the target video frame in step 400, the motion vector, the original video frame, and the target video frame are sent to the scene recognition module through the motion estimation module.

[0105] Step 410: Based on motion vectors, determine whether the motion scene corresponding to the original video frame and the target video frame is a preset scene.

[0106] In this embodiment of the application, when performing step 410, refer to... Figure 6 As shown, the scene recognition module performs the following steps:

[0107] Step 4101: Determine the magnitude of the vector for each pixel block based on the motion vector.

[0108] Step 4102: Based on the magnitude of each pixel block vector and the preset grouping rules, group the multiple pixel block vectors included in the motion vector.

[0109] Step 4103: Count the number of pixel block vectors contained in each group, and determine the group containing the most pixel block vectors as the target group.

[0110] Step 4104: If the number of pixel block vectors contained in the target group is greater than the preset value, then the motion scene corresponding to the original video frame and the target video frame is determined to be the preset scene.

[0111] Step 4105: If the number of pixel block vectors contained in the target group is not greater than the preset value, then it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene.

[0112] See Figure 7 As shown, taking the histogram statistics of the currently estimated motion vector as an example, steps 4101 to 4105 in the embodiments of this application will be introduced.

[0113] First, during step 4101, the magnitude of each pixel block vector can be calculated using the following formula:

[0114]

[0115] Where m represents any pixel block vector; |m| represents the magnitude of that pixel block vector.

[0116] In this embodiment of the application, assuming that the preset grouping rule is to divide the magnitude of each pixel block vector into M groups, then the starting horizontal coordinate of any group can be represented by the following formula:

[0117]

[0118] Where k is the sequence number, and X0 = 0.

[0119] In this way, we can obtain the intervals, and the size of each interval is...

[0120] Suppose a video frame image is divided into 16 pixel blocks, and the magnitudes of the vectors of the 16 pixel blocks are |m1|, |m2|, ..., |m15|, |m16| respectively.

[0121] Therefore, during step 4102, based on the magnitude of each pixel block vector and the interval corresponding to each group, the magnitude of each pixel block vector is divided into different groups, such as... Figure 7 As shown.

[0122] Then, step 4103 is executed to count the number of pixel block vectors contained in each group, and the group containing the most pixel block vectors is determined as the target group.

[0123] See Figure 7 As shown, assuming that group 4 contains the most pixel block vectors, which is 5, then group 4 is determined as the target group.

[0124] Assume the preset value is 4, and the preset scene is a translational motion scene.

[0125] Since 5 > 4, step 4104 is executed to determine that the motion scene corresponding to the original video frame and the target video frame is a preset scene, i.e., a translational motion scene. Then, the scene recognition module sends the pixel block vectors contained in the target group, the original video frame, and the target video frame to the frame interpolation module.

[0126] In some feasible embodiments, assuming the preset value is 8 and the number of pixel block vectors contained in group 4 is the largest, which is still 5, then since 5 < 8, step 4105 is executed to determine that the motion scene corresponding to the original video frame and the target video frame is not the preset scene, i.e., it is not a translational motion scene. Then, the scene recognition module renders and displays the target video frame and the original video frame sequentially according to the playback order of the video frames to be displayed.

[0127] In practice, the scene recognition module sends the target video frame and the original video frame to the image rendering module in the order of video frame playback. The image rendering module renders the target video frame or the original video frame and sends the rendered target video frame or the original video frame to the display screen. Finally, the rendered target video frame and the original video frame are displayed on the display screen in sequence.

[0128] Step 420: If yes, then determine the target refresh rate of the display device based on the motion vector.

[0129] In this embodiment, after executing step 410, based on motion vectors, if the motion scene corresponding to the original video frame and the target video frame is determined to be a preset scene, then step 420 is executed to determine the target refresh rate of the display device. Specifically, when executing step 420, refer to... Figure 8 As shown, this can be achieved by performing the following steps:

[0130] Step 4201: Determine the target vector corresponding to the target group based on the pixel block vectors contained in the target group.

[0131] In this embodiment of the application, when performing step 4201, the average vector of each pixel block contained in the target group is obtained by averaging the vectors of each pixel block contained in the target group, and this average vector is determined as the target vector corresponding to the target group, denoted as m. avg .

[0132] Step 4202: Based on the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, determine the target refresh rate of the display device corresponding to the target vector, wherein the amplitude of the target vector is positively correlated with the target refresh rate of the display device.

[0133] In this embodiment of the application, it is assumed that the variable refresh rates of the display device are F0, F1, F2, F3, and F4 in sequence, and the correspondence between the vector amplitude range and the target refresh rate of the display device is shown in the table below, where F0 < F1 < F2 < F3 < F4:

[0134] A table summarizing the correspondence between vector amplitude ranges and the target refresh rates of display devices.

[0135] <![CDATA[(0,th0)]]> F0 <![CDATA[[th0,th1)]]> F1 <![CDATA[[th1,th2)]]> F2 <![CDATA[[th2,th3)]]> F3 <![CDATA[[th3,∞)]]> F4

[0136] Then, during step 4202, the magnitude of the target vector is compared with the magnitude intervals of each vector in the above correspondence. If the magnitude of the target vector is |m avg If |∈(0, th0), then the target refresh rate of the display device corresponding to the target vector is F0; if the magnitude of the target vector is |m avg If |∈[th0, th1), then the target refresh rate of the display device corresponding to the target vector is F1; if the magnitude of the target vector is |m avg If |∈[th1, th2), then the target refresh rate of the display device corresponding to the target vector is F2; ​​if the magnitude of the target vector is |m avg If |∈[th2, th3), then the target refresh rate of the display device corresponding to the target vector is F3; if the magnitude of the target vector is |m avg If |∈[th3,∞), then the target refresh rate of the display device corresponding to the target vector is F4.

[0137] Step 430: Based on the target refresh rate and the frame rate of the video to be displayed, perform frame interpolation between the original video frames and the target video frames to adjust the frame rate of the video to be displayed.

[0138] In this embodiment of the application, when performing step 430, refer to... Figure 9 As shown, the frame interpolation module performs the following steps:

[0139] Step 4301: Based on the target refresh rate and the frame rate of the video to be displayed, determine the number of interpolation frames that need to be inserted between the original video frames and the target video frames.

[0140] In this embodiment of the application, when performing step 4301, the total number of video frames that need to be inserted into the video to be displayed is obtained based on the target refresh rate and the frame rate of the video to be displayed. Then, based on the ratio of the total number of video frames to the frame rate of the video to be displayed, the number of interpolation frames that need to be inserted between the original video frames and the target video frames is determined.

[0141] For example, suppose the target refresh rate is 36Hz and the frame rate of the video to be displayed is 24fps.

[0142] Therefore, since the target refresh rate of the display device is 36Hz, which means that 36 frames can be refreshed per second, and the frame rate of the video to be displayed is 24fps, which means that 24 frames can be refreshed per second.

[0143] Therefore, based on the target refresh rate and the frame rate of the video to be displayed, the total number of video frames that need to be inserted into the video to be displayed is 36-24=12. Then, based on the ratio of the total number of video frames to the frame rate of the video to be displayed, i.e. 12 / 24=0.5, one video frame needs to be inserted between every two video frames, thus determining that the number of interpolated frames that need to be inserted between the original video frame and the target video frame is 1.

[0144] Step 4302: Based on the frame number, the original video frame, and / or the target video frame, call the frame interpolation algorithm to obtain each interpolated frame.

[0145] In this embodiment of the application, when performing step 4302, an existing frame interpolation algorithm can be called to obtain each interpolated frame based on the number of frames and the original video frame; an existing frame interpolation algorithm can also be called to obtain each interpolated frame based on the number of frames and the target video frame; or an existing frame interpolation algorithm can be called to obtain each interpolated frame based on the number of frames, the original video frame, and the target video frame. This application does not limit the scope of the application.

[0146] Step 4303: Insert each interpolated frame between the original video frame and the target video frame.

[0147] In this embodiment of the application, after executing step 4303, the frame interpolation module further renders and displays the target video frame, each interpolated frame, and the original video frame in the order of video frame playback of the video to be displayed.

[0148] In practice, the frame interpolation module sends the target video frame, each interpolated frame, and the original video frame to the image rendering module in the order of video frame playback. The image rendering module renders the target video frame, each interpolated frame, or the original video frame and sends the rendered target video frame, each interpolated frame, or the original video frame to the display screen. Finally, the rendered target video frame, each interpolated frame, and the original video frame are displayed sequentially on the display screen.

[0149] Thus, by employing the frame rate adjustment method provided in this application embodiment, motion estimation can be performed based on historical frame information (e.g., target video frame) in the video to be displayed, thereby determining whether the motion scene corresponding to the original video frame and the target video frame is a preset scene; and after determining that the motion scene is a preset scene, the target refresh rate of the display device is determined based on the motion vector between the original video frame and the target video frame, and then, based on the target refresh rate and the frame rate of the video to be displayed, a frame interpolation algorithm is used to perform frame interpolation operations between the original video frame and the target video frame, so as to adaptively adjust the frame rate of the video to be displayed, adapting it to display devices with variable refresh rates, thereby improving the user's immersive experience.

[0150] Based on the same inventive concept, see [reference] Figure 10As shown in the figure, this application provides a frame rate adjustment device, applied to a display device with a variable refresh rate, the device comprising:

[0151] The module 1010 is used to obtain a motion vector between the original video frame and the target video frame based on the original video frame and the target video frame for each original video frame received in the video to be displayed, wherein the target video frame is the frame preceding the original video frame in the video to be displayed;

[0152] The determining module 1020 is used to determine, based on the motion vector, whether the motion scene corresponding to the original video frame and the target video frame is a preset scene;

[0153] The frame interpolation module 1030 is configured to, if so, determine the target refresh rate of the display device based on the motion vector, and perform frame interpolation between the original video frame and the target video frame based on the target refresh rate and the frame rate of the video to be displayed, so as to adjust the frame rate of the video to be displayed.

[0154] In one possible implementation, the obtaining module 1010 is further configured to:

[0155] After receiving each original video frame from the video to be displayed, it is determined whether the target video frame is in the buffer space;

[0156] If not, the original video frame is cached in the cache space as the target video frame;

[0157] If so, the target video frame is retrieved from the cache space, and the original video frame is cached in the cache space as the new target video frame.

[0158] In one possible implementation, when performing the step of determining whether the motion scene corresponding to the original video frame and the target video frame is a preset scene based on the motion vector, the determining module 1020 is specifically used for:

[0159] Based on the motion vector, determine the magnitude of the vector for each pixel block;

[0160] Based on the magnitude of each pixel block vector and a preset grouping rule, the multiple pixel block vectors included in the motion vector are grouped.

[0161] Count the number of pixel block vectors contained in each group, and determine the group with the most pixel block vectors as the target group;

[0162] If the number of pixel block vectors contained in the target group is greater than a preset value, then the motion scene corresponding to the original video frame and the target video frame is determined to be the preset scene;

[0163] If the number of pixel block vectors contained in the target group is not greater than the preset value, then it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene.

[0164] In one possible implementation, when performing the determination of the target refresh rate of the display device based on the motion vector, the frame interpolation module 1030 is specifically used for:

[0165] Based on the pixel block vectors contained in the target group, determine the target vector corresponding to the target group;

[0166] Based on the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, the target refresh rate of the display device corresponding to the target vector is determined, wherein the amplitude of the target vector is positively correlated with the target refresh rate of the display device.

[0167] In one possible implementation, the frame interpolation module 1030 is specifically used for:

[0168] Based on the target refresh rate and the frame rate of the video to be displayed, determine the number of interpolation frames that need to be inserted between the original video frame and the target video frame;

[0169] Based on the number of frames, the original video frames, and / or the target video frames, an interpolation algorithm is invoked to obtain each interpolated frame;

[0170] The interpolated frames are inserted between the original video frame and the target video frame.

[0171] In one possible implementation, the frame interpolation module 1030 is further configured to:

[0172] After the interpolated frames are inserted between the original video frame and the target video frame, the target video frame, the interpolated frames, and the original video frame are rendered and displayed sequentially according to the playback order of the video frames to be displayed.

[0173] In one possible implementation, the determining module 1020 is further configured to:

[0174] If it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene, then the target video frame and the original video frame are rendered and displayed sequentially according to the playback order of the video frames to be displayed.

[0175] Based on the same inventive concept, this application provides a display device, the structure of which can be as follows: Figure 11 As shown, it includes:

[0176] Display screen 1110 is used to display images;

[0177] Memory 1120 is used to store computer programs or instructions;

[0178] Processor 1130 may include one or more Central Processing Units (CPUs) or Digital Processing Units, etc. Processor 1130 is used to call program code stored in memory 1120 to implement the following process: Upon receiving an original video frame from a video to be displayed, based on the original video frame and a target video frame, obtain a motion vector between the original video frame and the target video frame, wherein the target video frame is the frame preceding the original video frame in the video to be displayed; based on the motion vector, determine whether the motion scene corresponding to the original video frame and the target video frame is a preset scene; if so, based on the motion vector, determine the target refresh rate of the display device, and based on the target refresh rate and the frame rate of the video to be displayed, perform frame interpolation between the original video frame and the target video frame to adjust the frame rate of the video to be displayed.

[0179] This application embodiment does not limit the specific connection medium between the display screen 1110, memory 1120, and processor 1130. This application embodiment... Figure 11 In this diagram, the display screen 1110, memory 1120, and processor 1130 are connected via a bus 1140. The connections between other components are merely illustrative and not intended to be limiting. The bus 1140 can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0180] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor, enables the processor to perform any of the methods described in the above embodiments.

[0181] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0182] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0183] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes in a flowchart and / or one or more blocks in a block diagram.

[0184] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0185] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A frame rate adjustment method, characterized in that, Applied to a display device with a variable refresh rate, the method includes: For each original video frame received in the video to be displayed, a motion vector between the original video frame and the target video frame is obtained based on the original video frame and the target video frame, wherein the target video frame is the frame preceding the original video frame in the video to be displayed; Based on the motion vector, the amplitude of each pixel block vector is determined; based on the amplitude of each pixel block vector and a preset grouping rule, the multiple pixel block vectors included in the motion vector are grouped, wherein the preset grouping rule is that the amplitude range of each group is the same size; Count the number of pixel block vectors contained in each group, and determine the group with the most pixel block vectors as the target group; If the number of pixel block vectors contained in the target group is greater than a preset value, then the motion scene corresponding to the original video frame and the target video frame is determined to be a preset scene; Based on the pixel block vectors contained in the target group, a target vector corresponding to the target group is determined; based on the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, the target refresh rate of the display device corresponding to the target vector is determined, wherein the amplitude of the target vector is positively correlated with the target refresh rate of the display device; Based on the target refresh rate and the frame rate of the video to be displayed, a frame interpolation operation is performed between the original video frame and the target video frame to adjust the frame rate of the video to be displayed.

2. The method as described in claim 1, characterized in that, After receiving each original video frame from the video to be displayed, the process further includes: Determine whether the target video frame exists in the cache space; If not, the original video frame is cached in the cache space as the target video frame; If so, the target video frame is retrieved from the cache space, and the original video frame is cached in the cache space as the new target video frame.

3. The method as described in claim 1 or 2, characterized in that, The step of interpolating frames between the original video frames and the target video frames based on the target refresh rate and the frame rate of the video to be displayed includes: Based on the target refresh rate and the frame rate of the video to be displayed, determine the number of interpolation frames that need to be inserted between the original video frame and the target video frame; Based on the number of frames, the original video frames, and / or the target video frames, an interpolation algorithm is invoked to obtain each interpolated frame; The interpolated frames are inserted between the original video frame and the target video frame.

4. The method as described in claim 3, characterized in that, After inserting the interpolated frames between the original video frame and the target video frame, the method further includes: According to the playback order of the video frames of the video to be displayed, the target video frame, each interpolated frame and the original video frame are rendered and displayed sequentially.

5. The method as described in claim 1, characterized in that, After determining the group containing the most pixel block vectors as the target group, the process also includes: If the number of pixel block vectors contained in the target group is not greater than the preset value, then it is determined that the motion scene corresponding to the original video frame and the target video frame is not the preset scene; The target video frame and the original video frame are rendered and displayed sequentially according to the playback order of the video frames to be displayed.

6. A frame rate adjustment device, characterized in that, An apparatus for use in a display device with a variable refresh rate, the apparatus comprising: The module is configured to, upon receiving an original video frame from the video to be displayed, obtain a motion vector between the original video frame and the target video frame based on the original video frame and the target video frame, wherein the target video frame is the frame preceding the original video frame in the video to be displayed; The determination module is used to determine the amplitude of each pixel block vector based on the motion vector; group the multiple pixel block vectors included in the motion vector based on the amplitude of each pixel block vector and a preset grouping rule, wherein the preset grouping rule is that the amplitude range of each group is the same size; count the number of pixel block vectors contained in each group, and determine the group containing the most pixel block vectors as the target group; if the number of pixel block vectors contained in the target group is greater than a preset value, then the motion scene corresponding to the original video frame and the target video frame is determined to be a preset scene; The frame interpolation module is configured to: determine a target vector corresponding to the target group based on the pixel block vectors contained in the target group; determine the target refresh rate of the display device corresponding to the target vector based on the target vector and the correspondence between the vector amplitude range and the target refresh rate of the display device, wherein the amplitude of the target vector is positively correlated with the target refresh rate of the display device; and perform frame interpolation between the original video frame and the target video frame based on the target refresh rate and the frame rate of the video to be displayed, so as to adjust the frame rate of the video to be displayed.

7. A display device, characterized in that, include: A display screen is used to display images. Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-5 is performed.

8. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor, the processor is able to perform the method as described in any one of claims 1-5.

Images