Motion vector correction method and device and electronic equipment

Abstract

The invention discloses a motion vector correction method and device and electronic equipment, and belongs to the technical field of image processing. The method comprises the following steps: obtaining a motion vector consistency graph and a color consistency graph, wherein the motion vector consistency graph and the color consistency graph are obtained according to a first image frame and a second image frame; based on the motion vector consistency graph and the color consistency graph, a to-be-corrected area in a target image frame is determined, and the target image frame is at least one of the first image frame or the second image; and correcting the motion vector corresponding to the to-be-corrected area to obtain a corrected target image frame.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to a motion vector correction method, device and electronic equipment. Background Art

[0002] Currently, when rendering game images, in order to reduce the rendering pressure of a graphics processing unit (GPU) and at the same time increase the game frame rate as much as possible, an interpolation algorithm is usually used to perform interpolation processing on the game images.

[0003] Typically, when using an interpolation algorithm to perform interpolation processing on a game screen, the electronic device can obtain a motion vector of the object based on the depth information of the object in the image frame, such as the motion vector of the object in the current image frame (hereinafter referred to as the first motion vector) and the motion vector of the object in the previous image frame (hereinafter referred to as the second motion vector), and obtain a new image frame based on the first motion vector, the second motion vector, the color map of the current image frame, and the color map of the previous image frame, and insert the new image frame between the current image frame and the previous image frame to perform interpolation processing on the game screen.

[0004] However, when an image frame contains a virtual object (such as a person's shadow or special effects), the electronic device cannot obtain the motion vector of the virtual object because the virtual object has no depth information. Therefore, when performing interpolation processing, the motion vector of the physical object (such as the floor or a person) at the corresponding position of the virtual object is used as the motion vector of the virtual object. In this way, the motion vector of the physical object is reused at this position, which may cause ghosting in the image interpolated by the electronic device. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a motion vector correction method, device and electronic device, which can reduce the ghosting phenomenon existing in the image of the electronic device's interpolated frame.

[0006] In a first aspect, an embodiment of the present application provides a motion vector correction method, which includes: obtaining a motion vector consistency map and a color consistency map, wherein the motion vector consistency map and the color consistency map are obtained based on a first image frame and a second image frame; based on the motion vector consistency map and the color consistency map, determining an area to be corrected in a target image frame, wherein the target image frame is at least one of the first image frame or the second image; and correcting the motion vector corresponding to the area to be corrected to obtain a corrected target image frame.

[0007] In a second aspect, an embodiment of the present application provides a motion vector correction device, which includes: an acquisition module, a determination module and a correction module. The acquisition module is used to acquire a motion vector consistency map and a color consistency map, and the motion vector consistency map and the color consistency map are obtained according to the first image frame and the second image frame. The determination module is used to determine the area to be corrected in the target image frame based on the motion vector consistency map and the color consistency map acquired by the acquisition module, and the target image frame is at least one of the first image frame or the second image. The correction module is used to correct the motion vector corresponding to the area to be corrected determined by the determination module to obtain a corrected target image frame.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In an embodiment of the present application, the electronic device can obtain a motion vector consistency map and a color consistency map; then, based on the motion vector consistency map and the color consistency map, determine the area to be corrected in the target image frame; finally, correct the motion vector corresponding to the area to be corrected to obtain the corrected target image frame. In this scheme, since the motion vector consistency map of the image frame can characterize the degree of motion vector deviation of the image frame and the color consistency map of the image frame can characterize the degree of deviation of the color value of the image frame. Therefore, through the motion vector consistency map and the color consistency map of the image frame, the area where the motion vector of the object in the image frame is inaccurate, that is, the area to be corrected, can be determined; then, by correcting the motion vector corresponding to the area to be corrected, the motion vectors of the objects in the image frame can be made more accurate (such as each object has a corresponding motion vector), so that when performing interpolation processing, each object in the image frame generated based on these motion vectors (i.e., the interpolated image) can correspond to its own motion vector, without the need to reuse the motion vectors of other objects, thereby reducing the ghosting phenomenon of the image interpolated by the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is one of the flow charts of a motion vector correction method provided in an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a mapping method provided in an embodiment of the present application;

[0015] Figure 3 is a structural diagram of a motion vector correction method provided by an embodiment of the present application;

[0016] Figure 4 This is the second flowchart of a motion vector correction method provided in an embodiment of the present application;

[0017] Figure 5 This is a third flowchart of a motion vector correction method provided in an embodiment of the present application;

[0018] Figure 6 This is a fourth flowchart of a motion vector correction method provided in an embodiment of the present application;

[0019] Figure 7 is an example diagram of motion vector correction provided by an embodiment of the present application;

[0020] Figure 8 It is one of the structural schematic diagrams of a motion vector correction device provided in an embodiment of the present application;

[0021] Fig. 9 This is a second structural schematic diagram of a motion vector correction device provided in an embodiment of the present application;

[0022] Fig.10 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0023] Fig.11 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0027] The motion vector correction method, device and electronic device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0028] The professional terms provided in the embodiments of the present application are explained below.

[0029] Motion vector consistency: Taking a pixel in an image frame as an example, motion vector consistency refers to the constraint condition for the displacement vector between the pixel and another pixel. For example, the displacement vector between a pixel and another pixel should have the same length and opposite direction. This constraint condition is usually defined as motion vector consistency. The other pixel is a pixel in the other image frame of two adjacent image frames that matches the position of the pixel.

[0030] It can be understood that the motion vector consistency map provided in the embodiment of the present application refers to whether the degree of offset of the motion vector between the pixel points in the previous image frame mapped to the current image frame and mapped from the current image frame back to the previous image frame is within a threshold; or, whether the degree of offset of the motion vector between the pixel points in the current image frame mapped to the previous image frame and mapped from the previous image frame back to the current image frame is within a threshold.

[0031] Color consistency: Color consistency is used to measure whether two images are similar.

[0032] It can be understood that the color consistency map provided in the embodiment of the present application refers to whether the color difference of the pixel point in the previous image frame is mapped to the current image frame within a threshold; or, whether the color difference of the pixel point in the current image frame is mapped to the first image frame within a threshold.

[0033] Binarization: It is to set the grayscale value of the pixel in the image to the first value (such as 0) or the second value (such as 1), that is, to present the entire image with an obvious visual effect of only black and white. For example, an image includes a target object, background and noise. To directly extract the target object from a multi-valued digital image, the most common method is to set a global threshold T, and use T to divide the pixels in the image into two parts, namely, a pixel group greater than T and a pixel group less than T. The pixel values ​​of the pixel group greater than T are set to white (or black), and the pixel values ​​of the pixel group less than T are set to black (or white).

[0034] It can be understood that the binarized image provided in the embodiments of the present application refers to an image that has been binarized.

[0035] Motion vector: A vector that has a change in position in time and space, that is, a vector with both magnitude and direction is called a motion vector.

[0036] The motion vector correction method, device and electronic device provided in the embodiment of the present application can be applied to the interpolation scene of the game screen. Of course, the embodiment of the present application can also be applied to the interpolation scene of other possible pictures such as video pictures, which can be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0037] It should be noted that, in order to more clearly describe the motion vector correction method provided in the embodiments of the present application, the following embodiments are illustrative in the context of applying the method to an interpolation scene of a game screen.

[0038] In recent years, with the development of mobile platforms, the content and experience of digital interactive entertainment have been greatly improved, especially in mobile games. Using mobile platforms to render high-refresh-rate, high-quality real-time game images is crucial to providing users with a smoother visual experience. Although mainstream electronic devices can support 1.5K or 2K image resolution and 120HZ or 144HZ screen refresh rate, large-scale open 3D games are usually limited to 60HZ or even lower refresh rates due to the computing power of the GPU of electronic devices, which seriously affects the user's gaming experience. Therefore, it is necessary to develop a low-power rendering algorithm to reduce the GPU rendering pressure while increasing the game frame rate as much as possible. Based on this, the interpolation technology of the game screen came into being.

[0039] Interpolation technology refers to the process of calculating the color map of the intermediate frame based on the color map of the current image frame and the previous image frame, combined with information such as motion vector (MV) and depth (D) that are easily obtained in the game rendering pipeline. Taking the interpolation technology of the game screen as an example, there is no need to consider various complex calculations in the three-dimensional scene when interpolating frames. The entire calculation process is performed in the two-dimensional image space. Therefore, compared with native rendering of a frame of color map, the process of obtaining the intermediate frame using interpolation technology can effectively reduce the amount of calculation of the GPU, allowing electronic devices to obtain higher frame rate game screens with limited computing resources.

[0040] In the interpolation technology of game screens, the accuracy of the motion vector map plays a vital role in the quality of the final interpolation. However, in the game engine, usually for physical objects with depth (such as floors or characters), electronic devices (such as GPUs) can calculate the motion vector of each pixel; but for virtual objects, such as character shadows, special effects and other areas, it is difficult to calculate their motion vectors, and these areas are usually located above the layers of physical objects, which will lead to the phenomenon that the motion vector does not match the color map. The color map of most floors and characters corresponds to the motion vector map, while there is no corresponding motion vector for character shadows, special effects and other parts. In the motion vector map, this part still belongs to the floor part, so the motion vector of this part does not match the color map. If the motion vector of the game engine is directly used to generate interpolation, the motion vector of the physical object at its corresponding position will be used for the virtual object, resulting in the repeated use of the motion vector of the physical object, which in turn leads to a ghosting effect in the virtual object area in the interpolation image. Furthermore, if the interpolated image is a dynamic image, the virtual object area in the dynamic image will also show a periodic jumping phenomenon, which seriously affects the user's gaming experience.

[0041] In order to solve the above problems, in the related art, the motion vector output by the game engine can usually be corrected by a block matching search algorithm (such as a three-dimensional recursive search). Specifically, the block matching search algorithm can be based on the motion vector output by the game engine, starting from the first pixel block in the current image frame, searching for a pixel block that matches the color of the first pixel block in the previous image frame, and then calculating the motion vector offset value between the two pixel blocks, and then adding the motion vector offset value between the two pixel blocks to the motion vector corresponding to each pixel point in the first pixel block. And so on, until a pixel block that matches the color of all pixel blocks in the current image frame is found in the previous image frame, and the motion vector offset value between all matching pixel blocks in the previous image frame and the current image frame is calculated, and the motion vector offset value between all matching pixel blocks is added to the motion vector corresponding to each pixel point in the current image frame; thereby completing the correction of the motion vector of the current image frame.

[0042] However, since the above-mentioned block matching search algorithm performs full-image matching, that is, it is necessary to traverse all pixel blocks in the image frame, and the motion vectors output by the game engine are usually mostly accurate, the block matching search algorithm will bring additional computing overhead, resulting in increased energy consumption of electronic devices.

[0043] In order to solve this technical problem, the embodiments of the present application provide a motion vector correction method, device, and electronic device. Since the motion vector consistency map of the image frame can characterize the degree of motion vector deviation of the image frame and the color consistency map of the image frame can characterize the degree of deviation of the color value of the image frame. Therefore, through the motion vector consistency map and the color consistency map of the image frame, the area where the motion vector of the object in the image frame is inaccurate, that is, the area to be corrected, can be determined; then, by correcting the motion vector corresponding to the area to be corrected, the motion vectors of the objects in the image frame can be made more accurate (such as each object has a corresponding motion vector), so that when performing interpolation processing, each object in the image frame (i.e., the interpolated image) generated based on these motion vectors can correspond to its own motion vector, without the need to reuse the motion vectors of other objects, thereby reducing the ghosting phenomenon in the interpolated image of the electronic device.

[0044] Furthermore, in the embodiment of the present application, only the motion vector corresponding to the area to be corrected can be corrected, and other areas still use the motion vector output by the game engine, so there is no need to perform full-image matching. In this way, the ghosting phenomenon of the image inserted in the electronic device can be reduced, and the power consumption of the electronic device can be ensured to be low.

[0045] The motion vector correction method provided in the embodiment of the present application may be executed by a motion vector correction device, which may be an electronic device, or a functional module or entity module in the electronic device. The technical solution provided in the embodiment of the present application is exemplarily described below using an electronic device as an example.

[0046] The present application embodiment provides a motion vector correction method. Figure 1 FIG. 4 is a flow chart showing a motion vector correction method provided by an embodiment of the present application. Figure 1 As shown, the motion vector correction method provided in the embodiment of the present application may include the following steps 201 to 203.

[0047] Step 201: The electronic device obtains a motion vector consistency map and a color consistency map.

[0048] In the embodiment of the present application, the above-mentioned motion vector consistency map and color consistency map are obtained according to the first image frame and the second image frame.

[0049] In the embodiment of the present application, the second image frame may be an image frame adjacent to the first image frame.

[0050] Optionally, in an embodiment of the present application, the second image frame may be an image frame adjacent to the front of the first image frame; or, the second image frame may be an image frame adjacent to the back of the first image frame.

[0051] It can be understood that the first image frame and the second image frame are adjacent image frames, which can be understood as the first image frame can be the first image frame among the adjacent image frames, that is, the previous image frame, and the second image frame can be the second image frame among the adjacent image frames, that is, the current image frame. Alternatively, the first image frame can be the second image frame among the adjacent image frames, that is, the current image frame, and the second image frame can be the first image frame among the adjacent image frames, that is, the previous image frame.

[0052] In the embodiment of the present application, the size of the motion vector consistency map and the color consistency map is the same as the size of the first image frame.

[0053] In an embodiment of the present application, the above-mentioned motion vector consistency map is used to characterize the degree of position difference between the first pixel point and the third pixel point, where the first pixel point is the pixel point of the first image frame, and the third pixel point is the pixel point when the first pixel point is mapped from the first image frame to the second image frame and then mapped from the second image frame back to the first image frame.

[0054] It should be noted that the mapping of a pixel point in one image frame to a pixel point in another image frame can be understood as searching for a pixel point matching the position of the pixel point in the other image frame.

[0055] Optionally, in an embodiment of the present application, the mapping of the above-mentioned first pixel point from the first image frame to the second image frame may mean that the electronic device can determine a pixel point in the second image frame that matches the position of the first pixel point in the first image frame.

[0056] Exemplarily, the electronic device may determine the second pixel in the second image frame through the position information and motion vector of the first pixel, that is, determine the pixel in the second image frame that matches the position of the first pixel in the first image frame.

[0057] For example, taking the position information of the first pixel as (1, 1) and the motion vector as (0, 5), the electronic device can determine the pixel at the position (1, 6) in the second image frame as the second pixel.

[0058] Optionally, in the embodiment of the present application, mapping the second pixel point from the second image frame to the first image frame may mean that the electronic device may determine the third pixel point in the first image frame.

[0059] Exemplarily, the electronic device may locate the second pixel in the first image frame through the position information and the motion vector of the second pixel.

[0060] For example, if the position information of the second pixel is (6, 1) and the motion vector is (0, 4), the electronic device may determine the pixel at the position (6, 5) in the first image frame as the second pixel.

[0061] In the embodiment of the present application, the color consistency map is used to characterize the degree of color difference between the first pixel and the fifth pixel, where the fifth pixel is the pixel of the first pixel mapped from the first image frame to the second image frame.

[0062] Optionally, in the embodiment of the present application, the first pixel point may be any pixel point in the first image frame.

[0063] Optionally, in an embodiment of the present application, the electronic device may obtain the motion vector consistency map by acquiring a motion vector offset map corresponding to the first image frame and then binarizing the motion vector offset map.

[0064] Optionally, in an embodiment of the present application, the above-mentioned motion vector consistency map may include a forward motion vector consistency map and a backward motion vector consistency map.

[0065] Exemplarily, in an embodiment of the present application, when the first image frame is a previous image frame, the motion vector offset map may be a forward motion vector offset map, and the motion vector consistency map may be a forward motion vector consistency map.

[0066] As another example, when the first image frame is the current image frame, the motion vector offset map may be a backward motion vector offset map, and the motion vector consistency map may be a backward motion vector consistency map.

[0067] In an embodiment of the present application, the electronic device can obtain the first forward motion vector of the first pixel point in the above-mentioned previous image frame and the first backward motion vector of the above-mentioned second pixel point, and then obtain the forward motion vector offset value of the first pixel point based on the first forward motion vector and the first backward motion vector.

[0068] In the embodiment of the present application, the first forward motion vector can be used to represent the motion vector of the first pixel point in the previous image frame moving to the current image frame. The first backward motion vector can be used to represent the motion vector of the second pixel point in the current image frame moving to the previous image frame.

[0069] Exemplarily, the electronic device may perform a reverse spatial transformation operation on the first forward motion vector and the first backward motion vector, that is, the first pixel is mapped from the previous image frame to the current image frame, and then mapped from the current image frame back to the previous image frame; then the sixth pixel is obtained, and the Euclidean distance between the sixth pixel and the first pixel is calculated to obtain the motion vector offset value corresponding to the first pixel in the previous image frame. This can be specifically implemented by the following formula (1).

[0070]

[0071] in, is the motion vector offset value corresponding to the first pixel in the previous image frame, φ 1→0 is the first backward motion vector, is the inverse spatial transformation operation, φ 0→1 is the first forward motion vector.

[0072] For example, Figure 2 As shown, the previous image frame is represented by I0, and the current image frame is represented by I1. Figure 2 The first upper left point in is the first pixel. Figure 2 1 in the figure, the electronic device can determine the second pixel point in I1 through the first forward motion vector through the reverse spatial transformation operation, and then map the second pixel point to I0, that is, the second pixel point in the lower right corner of I0, Figure 2 2 represents the sixth pixel.

[0073] It should be noted that the above is only an exemplary explanation using one pixel as an example. For each pixel in the previous image frame, the electronic device can obtain the motion vector offset value corresponding to each pixel in the previous image frame through the above formula (1). To avoid repetition, it will not be repeated here.

[0074] Optionally, in an embodiment of the present application, after obtaining the motion vector offset value corresponding to each pixel point in the previous image frame, the electronic device may construct a first image based on the position information of each pixel point in the previous image frame and the size of the previous image frame, and add the motion vector offset value corresponding to each pixel point in the previous image frame to the first image to generate a forward motion vector offset map corresponding to the previous image frame.

[0075] Exemplarily, the electronic device may use the size of the previous image frame as the size of the first image, and divide the first image based on the number of pixels to obtain multiple pixel areas in the first image. Then, the electronic device may add the motion vector offset value corresponding to each pixel in the previous image frame to the corresponding pixel area based on the position of each pixel in the previous image frame to obtain a forward motion vector offset map corresponding to the previous image frame.

[0076] Optionally, in an embodiment of the present application, after obtaining the forward motion vector offset map, the electronic device may perform binarization processing on the forward motion vector offset map to obtain a forward motion vector consistency map.

[0077] Exemplarily, the electronic device sets the value of the image area in the forward motion vector offset map where the motion vector offset value is greater than or equal to the first motion vector offset threshold to 1; and sets the value of the image area in the forward motion vector offset map where the motion vector offset value is less than the first motion vector offset threshold to 0, so as to obtain the forward motion vector consistency map. This can be implemented by the following formula (2).

[0078]

[0079] in, is the forward motion vector consistency map, is the forward motion vector offset map, T mv is the first motion vector offset threshold.

[0080] It should be noted that the image area set to 1 can be used to indicate that the motion vector offset value corresponding to the image area exceeds the maximum acceptable motion vector offset value (i.e., the first motion vector offset threshold); the maximum acceptable motion vector offset value is the offset that affects the accuracy beyond the acceptable range when performing interpolation processing, that is, the motion vector corresponding to the image area set to 1 is an inaccurate motion vector, that is, a motion vector that needs to be corrected. The image area set to 0 is used to indicate that the motion vector offset value corresponding to the image area does not exceed the maximum acceptable motion vector offset value; that is, the motion vector corresponding to the image area set to 0 is a relatively accurate motion vector.

[0081] It can be understood that the above is an exemplary description of the image region whose forward motion vector offset value is greater than or equal to the first motion vector offset threshold being set to 1, and the image region whose forward motion vector offset value is less than the first motion vector offset threshold being set to 0. In actual implementation, the image region whose forward motion vector offset value is greater than or equal to the first motion vector offset threshold can also be set to 0, and the image region whose forward motion vector offset value is less than the first motion vector offset threshold can be set to 1. The specific value can be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0082] In an embodiment of the present application, the electronic device can obtain the second backward motion vector of the first pixel point in the above-mentioned current image frame and the second forward motion vector of the above-mentioned second pixel point, and then obtain the backward motion vector offset value of the first pixel point based on the second forward motion vector and the second backward motion vector.

[0083] In the embodiment of the present application, the second backward motion vector can be used to represent the motion vector of the first pixel point in the current image frame moving to the previous image frame. The second forward motion vector can be used to represent the motion vector of the second pixel point in the previous image frame moving to the current image frame.

[0084] Exemplarily, the electronic device may perform a reverse spatial transformation operation on the second forward motion vector and the second backward motion vector, that is, the first pixel is mapped from the current image frame to the previous image frame, and then mapped from the previous image frame back to the current image frame; then the seventh pixel is obtained, and the Euclidean distance between the seventh pixel and the first pixel is calculated to obtain the motion vector offset value corresponding to the first pixel in the current image frame. This can be specifically achieved by the following formula (3).

[0085]

[0086] in, is the motion vector offset value corresponding to the first pixel in the current image frame, φ 0→1' is the second forward motion vector, is the inverse spatial transformation operation, φ 1→0' is the second backward motion vector.

[0087] It should be noted that the above is only an exemplary explanation using one pixel as an example. For each pixel in the current image frame, the electronic device can obtain the motion vector offset value corresponding to each pixel in the current image frame through the above formula (3). To avoid repetition, it will not be repeated here.

[0088] Optionally, in an embodiment of the present application, after obtaining the motion vector offset value corresponding to each pixel point in the current image frame, the electronic device may construct a second image based on the position information of each pixel point in the current image frame and the size of the current image frame, and add the motion vector offset value corresponding to each pixel point in the current image frame to the second image to generate a backward motion vector offset map corresponding to the current image frame.

[0089] Exemplarily, the electronic device may use the size of the current image frame as the size of the second image, and divide the second image based on the number of pixels to obtain multiple pixel areas in the second image. Then, the electronic device may add the motion vector offset value corresponding to each pixel in the current image frame to the corresponding blank pixel area based on the position of each pixel in the current image frame to obtain a backward motion vector offset map corresponding to the current image frame.

[0090] Optionally, in an embodiment of the present application, after obtaining the backward motion vector offset map, the electronic device may perform binarization processing on the backward motion vector offset map to obtain a backward motion vector consistency map.

[0091] Exemplarily, the electronic device sets the value of the image area in the backward motion vector offset map where the motion vector offset value is greater than or equal to the first motion vector offset threshold to 1; and sets the value of the image area in the backward motion vector offset map where the motion vector offset value is less than the first motion vector offset threshold to 0, so as to obtain the backward motion vector consistency map. This can be implemented by the following formula (4).

[0092]

[0093] in, is the backward motion vector consistency map, is the backward motion vector offset map, T mv is the first motion vector offset threshold.

[0094] It should be noted that the image area set to 1 can be used to indicate that the motion vector offset value corresponding to the image area exceeds the maximum acceptable motion vector offset value (i.e., the first motion vector offset threshold); the maximum acceptable motion vector offset value is the offset that affects the accuracy beyond the acceptable range when performing interpolation processing, that is, the motion vector corresponding to the image area set to 1 is an inaccurate motion vector, that is, a motion vector that needs to be corrected. The image area set to 0 is used to indicate that the motion vector offset value corresponding to the image area does not exceed the maximum acceptable motion vector offset value; that is, the motion vector corresponding to the image area set to 0 is a relatively accurate motion vector.

[0095] It can be understood that the above is an exemplary description of the image region where the backward motion vector offset value is greater than or equal to the first motion vector offset threshold value, and the image region where the backward motion vector offset value is less than the first motion vector offset threshold value, which is set to 0. In actual implementation, the image region where the backward motion vector offset value is greater than or equal to the first motion vector offset threshold value may also be set to 0, and the image region where the backward motion vector offset value is less than the first motion vector offset threshold value may be set to 1. The specific value can be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0096] Optionally, in an embodiment of the present application, the electronic device may obtain the color consistency map by acquiring a color difference map corresponding to the first image frame and then performing a binarization process on the color difference map.

[0097] Optionally, in an embodiment of the present application, the color consistency map may include a forward color consistency map and a backward color consistency map.

[0098] Optionally, in an embodiment of the present application, when the first image frame is a previous image frame, the color difference map may be a forward color difference map, and the color consistency map may be a forward color consistency map.

[0099] In an embodiment of the present application, the electronic device can obtain the first forward motion vector of the first pixel point in the previous image frame, the color value of the first pixel point, and the color value of the second pixel point; and then obtain the forward color difference value corresponding to the first pixel point based on the first forward motion vector, the color value of the first pixel point, and the color value of the second pixel point.

[0100] Exemplarily, the electronic device may perform a reverse spatial transformation operation on the first forward motion vector and the color value of the second pixel, that is, the first pixel is mapped from the previous image frame to the current image frame; then the eighth pixel is obtained, and the Euclidean distance between the color value of the eighth pixel and the color value of the first pixel is calculated to obtain the forward color difference value corresponding to the first pixel in the previous image frame. This may be implemented specifically by the following formula (5).

[0101]

[0102] in, is the forward color difference value corresponding to the first pixel in the previous image frame, I1 is the color value of the second pixel, It is an inverse spatial transformation operation, where I0 is the color value of the first pixel.

[0103] It should be noted that the above description is only for one pixel. For each pixel in the previous image frame, the electronic device can obtain the forward color difference value corresponding to each pixel in the previous image frame through the above formula (5). To avoid repetition, it will not be repeated here.

[0104] Optionally, in an embodiment of the present application, after obtaining the forward color difference value corresponding to each pixel in the previous image frame, the electronic device may construct a third image based on the pixel information of each pixel in the previous image frame and the size of the previous image frame, and add the forward color difference value corresponding to each pixel in the previous image frame to the third image to generate a forward color difference map corresponding to the previous image frame.

[0105] Optionally, in an embodiment of the present application, the above-mentioned pixel information includes the position of the pixel points and the number of the pixel points.

[0106] Exemplarily, the electronic device may use the size of the previous image frame as the size of the third image, and divide the third image based on the number of pixels to obtain multiple pixel areas in the third image. Then, the electronic device may add the forward color difference value corresponding to each pixel in the previous image frame to the corresponding pixel area based on the position of each pixel in the previous image frame to obtain a forward color difference map corresponding to the previous image frame.

[0107] Optionally, in the embodiment of the present application, after obtaining the forward color difference map, the electronic device may perform binarization processing on the forward color difference map to obtain a forward color consistency map.

[0108] Exemplarily, the electronic device sets the value of the image area whose color difference value in the forward color difference map is greater than or equal to the first color threshold to 1, and sets the value of the image area whose color difference value in the forward color difference map is less than the first color threshold to 0, so as to obtain the forward color consistency map. This can be implemented by the following formula (6).

[0109]

[0110] in, is the forward color consistency map, is the forward color difference map, T c is the first color threshold.

[0111] It should be noted that the image area set to 1 is used to indicate that the color difference value corresponding to the image area exceeds the maximum acceptable color difference value (i.e., the first color threshold); the maximum acceptable color difference value is a difference value that affects the accuracy beyond the acceptable range when performing interpolation processing, that is, the color value corresponding to the image area set to 1 does not match the motion vector. The image area set to 0 is used to indicate that the color difference value corresponding to the image area does not exceed the maximum acceptable motion vector offset value. That is, the color value corresponding to the image area set to 0 matches the motion vector.

[0112] It can be understood that the above is an exemplary description of the value of the image area where the forward color difference value is greater than or equal to the first color threshold value is set to 1, and the value of the image area where the forward color difference value is less than the first color threshold value is set to 0. In actual implementation, the value of the image area where the forward color difference value is greater than or equal to the first color threshold value can also be set to 0, and the value of the image area where the forward color difference value is less than the first color threshold value can be set to 1. The specific value can be determined according to actual use requirements, and the embodiment of the present application is not limited to this.

[0113] Optionally, in an embodiment of the present application, when the first image frame is the current image frame, the color difference map may be a backward color difference map, and the color consistency map may be a backward color consistency map.

[0114] In an embodiment of the present application, the electronic device can obtain the first backward motion vector of the second pixel point in the above-mentioned current image frame, the color value of the second pixel point and the color value of the first pixel point; and then obtain the backward color difference value corresponding to the second pixel point based on the first backward motion vector, the color value of the second pixel point and the color value of the first pixel point.

[0115] Exemplarily, the electronic device may perform a reverse spatial transformation operation on the first backward motion vector and the color value of the first pixel, that is, the second pixel is mapped from the current image frame to the previous image frame; then the ninth pixel is obtained, and the Euclidean distance between the color value of the ninth pixel and the color value of the second pixel is calculated to obtain the backward color difference value corresponding to the second pixel in the current image frame. This may be implemented specifically by the following formula (7).

[0116]

[0117] in, is the backward color difference value corresponding to the second pixel in the current image frame, I1 is the color value of the second pixel, It is an inverse spatial transformation operation, where I0 is the color value of the first pixel.

[0118] It should be noted that the above description is only for one pixel. For each pixel in the current image frame, the electronic device can obtain the backward color difference value corresponding to each pixel in the current image frame through the above formula (7). To avoid repetition, it will not be repeated here.

[0119] Optionally, in an embodiment of the present application, after obtaining the backward color difference value corresponding to each pixel in the current image frame, the electronic device may construct a fourth image based on the pixel information of each pixel in the current image frame and the size of the current image frame, and add the backward color difference value corresponding to each pixel in the current image frame to the fourth image to generate a backward color difference map corresponding to the current image frame.

[0120] Exemplarily, the electronic device may use the size of the current image frame as the size of the fourth image, and divide the fourth image based on the number of pixels to obtain multiple pixel areas in the fourth image. Then, the electronic device may add the backward color difference value corresponding to each pixel in the current image frame to the corresponding pixel area based on the position of each pixel in the current image frame to obtain a backward color difference map corresponding to the current image frame.

[0121] Optionally, in the embodiment of the present application, after obtaining the backward color difference map, the electronic device may perform binarization processing on the backward color difference map to obtain a backward color consistency map.

[0122] Exemplarily, the electronic device sets the value of the image area whose color difference value in the backward color difference map is greater than or equal to the first color threshold to 1, and sets the value of the image area whose color difference value in the backward color difference map is less than the first color threshold to 0, so as to obtain the backward color consistency map. This can be implemented by the following formula (8).

[0123]

[0124] in, is the backward color consistency map, is the backward color difference map, T c is the first color threshold.

[0125] It should be noted that the image area set to 1 is used to indicate that the color difference value corresponding to the image area exceeds the maximum acceptable color difference value (i.e., the first color threshold); the maximum acceptable color difference value is a difference value that affects the accuracy beyond the acceptable range when performing interpolation processing, that is, the color value corresponding to the image area set to 1 does not match the motion vector. The image area set to 0 is used to indicate that the color difference value corresponding to the image area does not exceed the maximum acceptable motion vector offset value. That is, the color value corresponding to the image area set to 0 matches the motion vector.

[0126] It can be understood that the above is an exemplary description of the value of the image area where the backward color difference value is greater than or equal to the first color threshold value is set to 1, and the value of the image area where the backward color difference value is less than the first color threshold value is set to 0. In actual implementation, the value of the image area where the backward color difference value is greater than or equal to the first color threshold value can also be set to 0, and the value of the image area where the backward color difference value is less than the first color threshold value can be set to 1. The specific value can be determined according to actual use requirements, and the embodiment of the present application is not limited to this.

[0127] Step 202: The electronic device determines a region to be corrected in the target image frame based on the motion vector consistency map and the color consistency map.

[0128] In the embodiment of the present application, the target image frame is at least one of the first image frame or the second image frame.

[0129] Optionally, in the embodiment of the present application, the above-mentioned area to be corrected may be an area in the first image frame where the motion vector is inaccurate.

[0130] Optionally, in the embodiment of the present application, the above-mentioned area to be corrected may be one or more areas.

[0131] Optionally, in the embodiment of the present application, the above-mentioned area to be corrected may be a pixel point or an area consisting of multiple pixel points.

[0132] Optionally, in the embodiment of the present application, the first area to be corrected may be a pixel area composed of a plurality of continuous pixels; or, the first area to be corrected may be a plurality of pixel areas composed of a plurality of discrete pixels. The specific details may be determined according to actual conditions and are not limited in the embodiment of the present application.

[0133] Optionally, in an embodiment of the present application, the above-mentioned area to be corrected may include a forward area to be corrected and a backward area to be corrected.

[0134] Exemplarily, when the first image frame is the previous image frame, the electronic device can determine the area to be corrected of the previous image frame through a forward motion vector consistency map and a forward color consistency map; or, when the first image frame is the current image frame, the electronic device can determine the area to be corrected of the current image frame through a backward motion vector consistency map and a backward color consistency map.

[0135] It should be noted that the specific process of determining the area to be corrected can be found in the following embodiments, and will not be described again here to avoid repetition.

[0136] Step 203: The electronic device corrects the motion vector corresponding to the area to be corrected to obtain a corrected first image frame.

[0137] In the embodiment of the present application, the electronic device may correct the motion vector corresponding to the area to be corrected by using a block matching search algorithm to obtain a corrected first image frame.

[0138] Exemplarily, the electronic device may correct the motion vector corresponding to each pixel point in the area to be corrected by using a block matching search algorithm to obtain a corrected first image frame.

[0139] Optionally, in an embodiment of the present application, when the first image frame is a previous image frame, the electronic device may correct the forward area to be corrected corresponding to the previous image frame to obtain a corrected previous image frame; when the first image frame is a current image frame, the electronic device may correct the backward area to be corrected corresponding to the current image frame to obtain a corrected current image frame.

[0140] In the embodiment of the present application, since only the motion vector corresponding to the area to be corrected can be corrected, the motion vectors of other areas in the first image frame still use the motion vectors output by the game engine. Therefore, on the basis of reducing the ghosting phenomenon of the image inserted in the electronic device, the power consumption of the electronic device can be ensured to be low.

[0141] It should be noted that the specific correction process can be found in the following embodiments, and will not be described again here to avoid repetition.

[0142] For example, Figure 3 As shown, the motion vector correction method provided in the embodiment of the present application is explained through a motion vector correction framework diagram.

[0143] The electronic device can obtain the forward motion vector corresponding to the previous image frame, the previous frame color map of the previous image frame, the backward motion vector corresponding to the current image frame and the current frame color map of the current image frame; then, input the forward motion vector and the backward motion vector into a motion consistency calculation module to obtain a motion consistency map; and input the forward motion vector corresponding to the previous image frame, the previous frame color map of the previous image frame, the backward motion vector corresponding to the current image frame and the current frame color map of the current image frame into a color consistency calculation module to obtain a color consistency map; finally, input the color consistency map and the motion consistency map into a motion vector correction module to obtain a corrected forward motion vector and a corrected backward motion vector.

[0144] It should be noted that the specific implementation process of the motion vector correction method shown in the above motion vector correction framework diagram can be found in the relevant description in the above embodiments, and will not be repeated here to avoid repetition.

[0145] Optionally, in an embodiment of the present application, the electronic device can perform interpolation processing based on the corrected previous image frame and the corrected current image frame. Since the motion vectors of objects in the corrected previous image frame and the corrected current image frame are relatively accurate, when the interpolation processing is performed, each object in the image frame generated based on these motion vectors can correspond to its own motion vector, without the need to reuse the motion vectors of other objects.

[0146] In the motion vector correction method provided in the embodiment of the present application, the electronic device can obtain a motion vector consistency map and a color consistency map; then, based on the motion vector consistency map and the color consistency map, determine the area to be corrected in the target image frame; finally, correct the motion vector corresponding to the area to be corrected to obtain the corrected target image frame. In this scheme, since the motion vector consistency map of the image frame can characterize the degree of motion vector deviation of the image frame and the color consistency map of the image frame can characterize the degree of deviation of the color value of the image frame. Therefore, through the motion vector consistency map and the color consistency map of the image frame, the area where the motion vector of the object in the image frame is inaccurate, that is, the area to be corrected, can be determined; then, by correcting the motion vector corresponding to the area to be corrected, the motion vectors of the objects in the image frame can be made more accurate (such as each object has a corresponding motion vector), so that when performing interpolation processing, each object in the image frame generated based on these motion vectors (i.e., the interpolation image) can correspond to its own motion vector, without the need to reuse the motion vectors of other objects, thereby reducing the ghosting phenomenon of the image interpolated by the electronic device.

[0147] Optionally, in the embodiment of the present application, the motion vector consistency map and the color consistency map may be binary maps.

[0148] For example, in combination Figure 1 ,like Figure 4 As shown, the above step 202 can be specifically implemented through the following steps 202a and 202b.

[0149] Step 202a: The electronic device determines a first area based on areas with first values ​​in the motion vector consistency map and areas with second values ​​in the color consistency map.

[0150] Optionally, in the embodiment of the present application, the above-mentioned first area may be one or more.

[0151] Optionally, in the embodiment of the present application, the first value may be 0, and the second value may be 1. Alternatively, the first value may be 1, and the second value may be 0. The specific value may be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0152] Exemplarily, the first area may be a pixel or an area consisting of a plurality of pixels.

[0153] Optionally, in an embodiment of the present application, the first region may be a region composed of a plurality of continuous pixels; or, the first region may be a plurality of regions composed of a plurality of discrete pixels.

[0154] Optionally, in the embodiment of the present application, the first region may include a forward first region and a backward first region.

[0155] Exemplarily, when the first image frame is the previous image frame, the above-mentioned first area can be the forward first area corresponding to the forward motion vector consistency map and the forward color consistency map; when the first image frame is the current image frame, the above-mentioned first area can be the backward first area corresponding to the backward motion vector consistency map and the backward color consistency map.

[0156] Optionally, in an embodiment of the present application, the electronic device may perform pixel alignment processing on the motion vector consistency map and the color consistency map, and then determine an area where a value of 0 in the motion vector consistency map and a value of 1 in the color consistency map overlap as the first area.

[0157] Exemplarily, the electronic device may perform pixel alignment processing on the forward motion vector consistency map and the forward color consistency map, and then determine a region where a value of 0 in the forward motion vector consistency map overlaps with a value of 1 in the forward color consistency map as the first region.

[0158] It should be noted that the electronic device determines the forward first area and the backward first area in the same manner, and will not be described again here to avoid repetition.

[0159] In an embodiment of the present application, the electronic device can determine the first area in combination with the motion vector consistency map and the color consistency map. That is, the electronic device can determine the first area from multiple dimensions, thereby improving the accuracy of the electronic device in determining the first area.

[0160] Optionally, in the embodiment of the present application, combined with Figure 4 ,like Figure 5 As shown, the above step 202a can be specifically implemented through the following step 202a1.

[0161] Step 202a1: The electronic device determines the intersection of the region with the first value in the motion vector consistency map and the region with the second value in the color consistency map (ie, the region where the two overlap) as the first region.

[0162] Optionally, in an embodiment of the present application, taking the first area as the forward first area as an example, the electronic device can use the intersection of the area with a value of 0 in the forward motion vector consistency map and the area with a value of 1 in the forward color consistency map as the forward first area, which can be specifically implemented by formula (9).

[0163]

[0164] in, is the first forward region, is the forward color consistency map, is the forward motion vector consistency map.

[0165] Optionally, in an embodiment of the present application, taking the first area as the backward first area as an example, the electronic device can use the intersection of the area with a value of 0 in the backward motion vector consistency map and the area with a value of 1 in the backward color consistency map as the backward first area, which can be specifically implemented by formula (10).

[0166]

[0167] in, For the first backward area, is the backward color consistency map, Back-forward motion vector consistency map.

[0168] Step 202b: determine the area in the target image frame corresponding to the position of the first area as the area to be corrected.

[0169] Optionally, in an embodiment of the present application, the electronic device may obtain first position information of the third area, and then determine an area in the target image frame whose position information is the same as the first position information as the area to be corrected.

[0170] In one example, when the target image frame is a previous image frame and the first area is a forward first area, the electronic device can determine, based on the first position information of the forward first area, an area in the previous image frame whose position information is the same as the first position information as a forward area to be corrected.

[0171] In another example, when the target image frame is the current image frame and the first area is the backward first area, the electronic device can determine the area in the current image frame whose position information is the same as the first position information as the backward area to be corrected based on the first position information of the backward first area.

[0172] Optionally, in an embodiment of the present application, the electronic device may obtain first position information of the third area, then perform pixel alignment processing on the first area and the target image frame, and then determine the area where the target image frame overlaps with the first area as the area to be corrected.

[0173] In one example, when the target image frame is a previous image frame and the first area is a forward first area, the electronic device can perform pixel alignment processing with the previous image frame based on the first position information of the forward first area, and then determine the area in the previous image frame that overlaps with the forward first area as the forward area to be corrected.

[0174] In another example, when the target image frame is the current image frame and the first area is the backward first area, the electronic device can perform pixel alignment processing with the current image frame based on the first position information of the backward first area, and then determine the area in the current image frame that overlaps with the backward first area as the backward area to be corrected.

[0175] In an embodiment of the present application, the electronic device can determine the area in the target image frame that corresponds to the first area as the area to be corrected. After the electronic device corrects the area to be corrected, the motion vectors of the objects in the image frame can be relatively accurate, so that each object in the image frame (i.e., the interpolated image) generated based on these motion vectors can correspond to its own motion vector without reusing the motion vectors of other objects, thereby reducing the ghosting phenomenon in the image interpolated by the electronic device.

[0176] Optionally, in the embodiment of the present application, combined with Figure 1 ,like Figure 6 As shown, the target image frame is the first image frame; the step 203 can be implemented through the following steps 203a to 203c.

[0177] Step 203a: The electronic device determines, in the second image frame, a second pixel whose color value matches the color value of the first pixel based on the color value of the first pixel in the area to be corrected.

[0178] In the embodiment of the present application, the second image frame may be an image frame adjacent to the first image frame.

[0179] Optionally, in an embodiment of the present application, the second image frame may be an image frame adjacent to the front of the first image frame; or, the second image frame may be an image frame adjacent to the back of the first image frame.

[0180] In the embodiment of the present application, the electronic device may determine, based on the criterion of minimizing color difference, a second pixel point whose color value matches the color value of the first pixel point in the second image frame.

[0181] For example, it is assumed that the maximum displacement of the first pixel in the X direction and the Y direction is X max and Y max ; Then at (M+2*X max , N+2*Y max ), find the pixel with the smallest color difference with the first pixel in the range, and use the pixel as the second pixel, where M and N are the sizes of the first pixel.

[0182] In one example, taking the first image frame as the previous image frame, the electronic device can determine the second pixel point whose color value has the smallest difference with the color value of the first pixel point in the current image frame by the above method.

[0183] In another example, with the first image frame as the current image frame, the electronic device may determine, in the previous image frame, a second pixel point whose color value has the smallest difference with the color value of the first pixel point in the above manner.

[0184] It should be noted that, for each pixel in the area to be corrected, the electronic device can determine the pixel whose color value matches the color value of each pixel in the second image frame in the above manner, which will not be described again here to avoid repetition.

[0185] Step 203b: The electronic device calculates a position offset value between the first pixel point and the second pixel point.

[0186] In the embodiment of the present application, the electronic device can obtain the position offset value between the first pixel point and the second pixel point by calculating the Euclidean distance between the first pixel point and the second pixel point. Specifically, this can be achieved by the following formula (10).

[0187] F=||ab||2 (10)

[0188] Wherein, F is the position offset value between the first pixel point and the second pixel point, a is the position information of the first pixel point, and b is the position information of the second pixel point.

[0189] It should be noted that, for each pixel point in the area to be corrected, the electronic device can calculate the motion vector offset value corresponding to each pixel point in the area to be corrected in the above manner, which will not be described again here to avoid repetition.

[0190] Step 203c: The electronic device corrects the motion vector of the first pixel based on the position offset value to obtain a corrected first image frame.

[0191] In the embodiment of the present application, the electronic device may add the motion vector of the first pixel point to the position offset value to obtain a corrected motion vector of the first pixel point.

[0192] In one example, when the first image frame is the previous image frame, the electronic device may correct each pixel in the area to be corrected in the previous image frame based on the position offset value corresponding to the pixel to obtain a corrected previous image frame.

[0193] In another example, when the target image frame is the current image frame, the electronic device may correct each pixel in the area to be corrected in the current image frame based on the position offset value corresponding to the pixel to obtain a corrected current image frame.

[0194] It should be noted that, for each pixel point in the area to be corrected, the electronic device can correct each pixel point in the area to be corrected in the above manner to obtain a corrected first image frame, which will not be described again here to avoid repetition.

[0195] For example, Figure 7 As shown, Figure 7 The visual comparison of the interpolated frames before and after motion vector correction is shown. Figure 7 It can be found that the interpolated frame before the motion vector correction has a ghosting effect at the character's shadow. After the correction by the algorithm of the present application, the ghosting effect of the final interpolated frame disappears at the character's shadow and does not affect other areas. Figure 7 It is found that the interpolated frame before the motion vector correction has a ghosting effect at the halo (special effect area) under the feet of the character. After the correction by the algorithm of this application, the ghosting effect of the halo in the final interpolated frame disappears. In summary, using this application to correct the inaccurate motion vector area can effectively alleviate the occurrence of ghosting in the character shadow and special effect area in the interpolated frame, and the interpolated frame quality is better.

[0196] In an embodiment of the present application, the electronic device can perform vector correction on each pixel point in the correction area through a block matching search algorithm, so that the motion vectors of objects in the image frame can be relatively accurate. Therefore, when performing interpolation processing, each object in the image frame generated based on these motion vectors can correspond to its own motion vector without reusing the motion vectors of other objects, thereby reducing the ghosting phenomenon in the image interpolated by the electronic device.

[0197] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0198] It should be noted that the motion vector correction method provided in the embodiment of the present application can be executed by a motion vector correction device. In the embodiment of the present application, the motion vector correction device executing the motion vector correction method is taken as an example to illustrate the motion vector correction device provided in the embodiment of the present application.

[0199] Figure 8 A possible structural diagram of a motion vector correction device involved in an embodiment of the present application is shown. Figure 8 As shown, the motion vector correction device 70 may include: an acquisition module 71 , a determination module 72 and a correction module 73 .

[0200] The acquisition module 71 is used to acquire a motion vector consistency map and a color consistency map, which are obtained based on the first image frame and the second image frame. The determination module 72 is used to determine the area to be corrected in the target image frame based on the motion vector consistency map and the color consistency map obtained by the acquisition module 71, and the target image frame is at least one of the first image frame or the second image. The correction module 73 is used to correct the motion vector corresponding to the area to be corrected determined by the determination module 72 to obtain a corrected target image frame.

[0201] In a possible implementation, the motion vector consistency map and the color consistency map are binarized maps. The determination module 72 is specifically configured to determine the first region based on the region with the first value in the motion vector consistency map and the region with the second value in the color consistency map; and determine the region in the target image frame corresponding to the position of the first region as the region to be corrected.

[0202] In a possible implementation, the determination module 72 is specifically configured to determine the first region by taking the intersection of the region whose value in the motion vector consistency map is the first value and the region whose value in the color consistency map is the second value.

[0203] In a possible implementation manner, the area to be corrected is an area in the first image frame where the motion vector is inaccurate.

[0204] In one possible implementation, combining Figure 7 ,like Fig. 9 As shown, the target image frame is the first image frame; the motion vector correction device 70 provided in the embodiment of the present application also includes a processing module 74. The determination module 72 is also used to determine a second pixel whose color value matches the color value of the first pixel in the area to be corrected in the second image frame, based on the color value of the first pixel in the area to be corrected, and the second image frame is an image frame adjacent to the first image frame. The processing module 74 is used to calculate the position offset value between the first pixel and the second pixel. The correction module 73 is specifically used to correct the motion vector of the first pixel based on the position offset value to obtain a corrected first image frame.

[0205] The embodiment of the present application provides a motion vector correction device, because the motion vector consistency map of the image frame can represent the degree of motion vector deviation of the image frame and the color consistency map of the image frame can represent the degree of deviation of the color value of the image frame. Therefore, through the motion vector consistency map and the color consistency map of the image frame, the area where the motion vector of the object in the image frame is inaccurate, that is, the area to be corrected, can be determined; then, by correcting the motion vector corresponding to the area to be corrected, the motion vectors of the objects in the image frame can be made more accurate (such as each object has a corresponding motion vector), so that when performing interpolation processing, each object in the image frame (i.e., the interpolated image) generated based on these motion vectors can correspond to its own motion vector, without the need to reuse the motion vectors of other objects, thereby reducing the ghosting phenomenon in the interpolated image of the motion vector correction device.

[0206] The motion vector correction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0207] The motion vector correction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0208] The motion vector correction device provided in the embodiment of the present application can implement each process implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0209] Alternatively, if Fig.10 As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores programs or instructions that can be executed on the processor 91, and when the program or instructions are executed by the processor 91, the various steps of the above-mentioned motion vector correction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0210] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0211] Fig.11 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0212] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0213] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0214] Among them, the processor 110 is used to obtain a motion vector consistency map and a color consistency map, which are obtained based on the first image frame and the second image frame; based on the motion vector consistency map and the color consistency map, determine the area to be corrected in the target image frame, the target image frame is at least one of the first image frame or the second image; correct the motion vector corresponding to the area to be corrected to obtain a corrected target image frame.

[0215] Optionally, in an embodiment of the present application, the motion vector consistency map and the color consistency map are binary maps; the processor 110 is specifically used to determine the first region based on the region with the first value in the motion vector consistency map and the region with the second value in the color consistency map; and determine the region in the target image frame corresponding to the position of the first region as the region to be corrected.

[0216] Optionally, in an embodiment of the present application, the processor 110 is specifically configured to determine the first region by taking the intersection of a region whose value in the motion vector consistency map is the first value and a region whose value in the color consistency map is the second value.

[0217] Optionally, in an embodiment of the present application, the target image frame is a first image frame; the processor 110 is specifically used to determine, based on the color value of the first pixel in the area to be corrected, a second pixel whose color value matches the color value of the first pixel in the second image frame, and the second image frame is an image frame adjacent to the first image frame; calculate a position offset value between the first pixel and the second pixel; and based on the position offset value, correct a motion vector of the first pixel to obtain a corrected first image frame.

[0218] The embodiment of the present application provides an electronic device, since the motion vector consistency map of the image frame can characterize the degree of motion vector deviation of the image frame and the color consistency map of the image frame can characterize the degree of deviation of the color value of the image frame. Therefore, through the motion vector consistency map and the color consistency map of the image frame, the area where the motion vector of the object in the image frame is inaccurate, that is, the area to be corrected, can be determined; then, by correcting the motion vector corresponding to the area to be corrected, the motion vectors of the objects in the image frame can be made more accurate (such as each object has a corresponding motion vector), so that when performing interpolation processing, each object in the image frame (i.e., the interpolated image) generated based on these motion vectors can correspond to its own motion vector, without the need to reuse the motion vectors of other objects, thereby reducing the ghosting phenomenon in the interpolated image of the electronic device.

[0219] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0220] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0221] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0222] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0223] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0224] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0225] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0226] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0227] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0228] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned motion vector correction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0229] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0230] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0231] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A motion vector correction method, characterized in that: The method comprises: Acquire a motion vector consistency map and a color consistency map, wherein the motion vector consistency map and the color consistency map are obtained according to the first image frame and the second image frame; Determine, based on the motion vector consistency map and the color consistency map, a region to be corrected in a target image frame, the target image frame comprising at least one of the first image frame and the second image frame; The motion vector corresponding to the area to be corrected is corrected to obtain a corrected target image frame.

2. The method according to claim 1, characterized in that The motion vector consistency map and the color consistency map are binarized maps; The step of determining the area to be corrected in the target image frame based on the motion vector consistency map and the color consistency map includes: Determine a first region based on a region whose value in the motion vector consistency map is a first value and a region whose value in the color consistency map is a second value; An area in the target image frame corresponding to the position of the first area is determined as the area to be corrected.

3. The method according to claim 2, characterized in that The determining of the first area based on an area whose value is the first value in the motion vector consistency map and an area whose value is the second value in the color consistency map comprises: An intersection of an area with a first value in the motion vector consistency map and an area with a second value in the color consistency map is determined as the first area.

4. The method according to any one of claims 1 to 3, characterized in that The area to be corrected is an area in the target image frame where the motion vector is inaccurate.

5. The method according to claim 1, characterized in that The target image frame is the first image frame; The step of correcting the motion vector corresponding to the area to be corrected to obtain a corrected target image frame includes: Based on the color value of the first pixel in the area to be corrected, determining in a second image frame a second pixel whose color value matches the color value of the first pixel, the second image frame being an image frame adjacent to the first image frame; Calculating a position offset value between the first pixel point and the second pixel point; Based on the position offset value, the motion vector of the first pixel is corrected to obtain the corrected first image frame.

6. A motion vector correction device, characterized in that: The motion vector correction device comprises: an acquisition module, a determination module and a correction module; The acquisition module is used to acquire a motion vector consistency map and a color consistency map, wherein the motion vector consistency map and the color consistency map are obtained according to the first image frame and the second image frame; The determining module is used to determine the area to be corrected in the target image frame based on the motion vector consistency map and the color consistency map acquired by the acquiring module, wherein the target image frame is at least one of the first image frame or the second image frame; The correction module is used to correct the motion vector corresponding to the area to be corrected determined by the determination module to obtain a corrected target image frame.

7. The device according to claim 6, characterized in that The motion vector consistency map and the color consistency map are binarized maps; The determination module is specifically used to determine the first area based on the area with the first value in the motion vector consistency map and the area with the second value in the color consistency map; and determine the area in the target image frame corresponding to the position of the first area as the area to be corrected.

8. The device according to claim 7, characterized in that The determination module is specifically configured to determine an intersection of an area in the motion vector consistency map where the value is the first value and an area in the color consistency map where the value is the second value as the first area.

9. The device according to any one of claims 6 to 8, characterized in that The area to be corrected is an area in the first image frame where the motion vector is inaccurate.

10. The device according to claim 6, characterized in that The target image frame is the first image frame; the motion vector correction device further includes a processing module; The determination module is further used to determine, based on the color value of the first pixel in the area to be corrected, a second pixel with a color value matching the color value of the first pixel in a second image frame, the second image frame being an image frame adjacent to the first image frame; The processing module is used to calculate the position offset value between the first pixel point and the second pixel point; The correction module is specifically used to correct the motion vector of the first pixel point based on the position offset value to obtain the corrected first image frame.

11. An electronic device, characterized in that: The method comprises a processor, a memory and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the motion vector correction method as described in any one of claims 1 to 5.

Images