Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of communication. The image processing method comprises the following steps: performing re-projection operation on each first pixel point of a first image frame; determining second pixel points respectively matched with the first pixel points in the second image frame; sampling the first image frame to obtain a first depth value and a first color value of each first sampling position of the first image frame; sampling the second image frame to obtain a second depth value and a second color value of each second sampling position of the second image frame; performing depth filtering according to the first depth value and the second depth value, and performing color accumulation mixing according to a depth filtering result, the first color value and the second color value to obtain a first color map; performing neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map; and synthesizing the first color map and the first depth map to obtain a third image frame.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to an image processing method and device thereof. Background Art

[0002] Anti-aliasing technology has received extensive attention and development in recent years. Anti-aliasing technology aims to reduce the jagged edges and graphic distortion in digital images to improve the quality and realism of graphics.

[0003] Traditional anti-aliasing techniques include Multi Sample Anti-Aliasing (MSAA), Fast Approximate Anti-Aliasing (FXAA), Morphological Anti-Aliasing (SMAA), Temporal Anti-Aliasing (TAA), etc.

[0004] Although these technologies have been widely used on personal computers (PCs) and game consoles, anti-aliasing technology on electronic devices with relatively limited hardware resources still faces some challenges. For example, the hardware anti-aliasing used by mobile platforms is mainly the MSAA algorithm, which has poor temporal anti-aliasing and anti-flicker effects; while the TAA algorithm has a large power consumption overhead on the hardware platform and requires the graphics engine to run in a more complex way, such as supporting motion vector generation; therefore, traditional anti-aliasing methods are not suitable for electronic devices with limited hardware resources. The application of traditional anti-aliasing technology on electronic devices with relatively limited hardware resources may increase the burden on the graphics card, resulting in performance degradation and increased energy consumption. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image processing method and device thereof, which can solve the problem in the prior art that traditional anti-aliasing technology increases the burden on the graphics card, resulting in reduced performance of electronic equipment and increased energy consumption.

[0006] In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0007] Performing a reprojection operation on each first pixel point of the first image frame; and determining to obtain second pixel points in a second image frame that respectively match each first pixel point, wherein the second image frame is a previous frame of the first image frame;

[0008] Taking the position of each of the first pixel points as a first sampling position, sampling the first image frame to obtain a first depth value and a first color value of each of the first sampling positions of the first image frame;

[0009] Taking the position of each of the second pixel points as a second sampling position, sampling the second image frame to obtain a second depth value and a second color value of each of the second sampling positions of the second image frame;

[0010] Performing depth filtering according to the first depth value and the second depth value, and performing color accumulation and mixing according to the depth filtering result, the first color value, and the second color value to obtain a first color map;

[0011] Performing neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map;

[0012] The first color map and the first depth map are synthesized to obtain a third image frame.

[0013] In a second aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0014] a reprojection module, configured to perform a reprojection operation on each first pixel point of a first image frame; and determine to obtain second pixel points in a second image frame that respectively match each first pixel point, wherein the second image frame is a previous frame of the first image frame;

[0015] A first sampling module, configured to sample the first image frame by taking the position of each of the first pixel points as a first sampling position, and obtain a first depth value and a first color value of each of the first sampling positions of the first image frame;

[0016] A second sampling module, configured to sample the second image frame by taking the position of each of the second pixel points as a second sampling position, to obtain a second depth value and a second color value of each of the second sampling positions of the second image frame;

[0017] A first processing module, configured to perform depth filtering according to the first depth value and the second depth value, and perform color accumulation and mixing according to the depth filtering result, the first color value, and the second color value to obtain a first color map;

[0018] A second processing module, configured to perform neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map;

[0019] A synthesis module is used to synthesize the first color map and the first depth map to obtain a third image frame.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0022] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0024] In an embodiment of the present application, the first image frame and the second image frame are sampled respectively to obtain corresponding depth values ​​and color values. During the image processing process, only the depth values ​​and color values ​​are relied upon to implement anti-aliasing processing of the first image frame. The image processing method requires less information and can be implemented on electronic devices with relatively efficient hardware resources. Moreover, the method does not need to calculate the motion vectors of all objects, thereby reducing the required amount of calculation, memory usage and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 One of the flowcharts showing the steps of the image processing method provided in the embodiment of the present application;

[0026] Figure 2 An example diagram showing an application module of the image processing method provided in an embodiment of the present application;

[0027] Figure 3 One of the application example diagrams of the image processing method provided in the embodiment of the present application is shown;

[0028] Figure 4 A second diagram showing an application example of the image processing method provided in an embodiment of the present application;

[0029] Figure 5 An example diagram showing calculation of a flickering edge area in an image processing method provided in an embodiment of the present application;

[0030] Figure 6 An example diagram showing the direction of historical boundaries in the image processing method provided in an embodiment of the present application;

[0031] Figure 7 A schematic diagram showing the structure of an image processing device provided in an embodiment of the present application;

[0032] Figure 8 One of the structural schematic diagrams of the electronic device provided in the embodiment of the present application is shown;

[0033] Fig. 9 The second structural schematic diagram shows the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, the embodiment of the present application also provides an image processing method, which includes:

[0038] Step 101, performing a reprojection operation on each first pixel point of a first image frame; and determining second pixel points in a second image frame that match each first pixel point respectively; wherein the second image frame is a previous frame of the first image frame;

[0039] Optionally, the first image frame refers to an image frame that is currently being drawn and is about to be displayed during computer graphics rendering processing.

[0040] Optionally, the second image frame refers to an image frame that has been displayed in a computer graphics rendering process, and each frame represents a historical moment of the picture.

[0041] The second image frame in the embodiment of the present application is specifically a frame before the first image frame. For example, if the first image frame is the Nth frame, then the second image frame is the N-1th frame.

[0042] Step 102, taking the position of each first pixel point as a first sampling position, sampling the first image frame to obtain a first depth value and a first color value of each first sampling position of the first image frame;

[0043] For example, the color map of the first image frame is sampled to obtain the color value at the first sampling position (ie, the first color value); the depth map of the first image frame is sampled to obtain the depth value at the first sampling position (ie, the first depth value).

[0044] Step 103, taking the position of each second pixel point as a second sampling position, sampling the second image frame to obtain a second depth value and a second color value of each second sampling position of the second image frame;

[0045] For example, the color map of the second image frame is sampled to obtain the color value at the second sampling position (ie, the second color value); the depth map of the second image frame is sampled to obtain the depth value at the second sampling position (ie, the second depth value).

[0046] Step 104, performing depth filtering according to the first depth value and the second depth value, and performing color accumulation and mixing according to the depth filtering result, the first color value, and the second color value to obtain a first color map;

[0047] Optionally, the first color map is a color map after anti-aliasing processing. A color map refers to a commonly used image type in computer graphics and is used to record color information of each pixel.

[0048] Step 105, performing neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map;

[0049] Optionally, the first depth map is a depth map after anti-aliasing processing. The depth map refers to a commonly used image type in computer graphics, which is used to record the distance or depth information from each pixel to an observer (usually a camera).

[0050] Optionally, the neighborhood extreme value filtering includes: for the depth value of the current frame (ie, the first depth value), the depth value M closest to the camera position in the 3x3 area may be used. d (Some graphics device drivers use the minimum depth value, while others use the maximum depth value.) d Output to a new depth map (i.e., the first depth map) for calculation in the next frame (i.e., the N+1th frame);

[0051] Step 106: synthesize the first color map and the first depth map to obtain a third image frame.

[0052] Optionally, the third image frame is an image frame obtained by performing anti-aliasing processing on the first image frame.

[0053] The embodiment of the present application only relies on depth values ​​and color values ​​to implement anti-aliasing processing on the first image frame during the image processing process. The image processing method requires less information; and there is no need to calculate the motion vectors of all objects, thereby reducing the necessary amount of calculation, memory usage and power consumption. Furthermore, the image processing method has no special requirements for the game engine, and a dedicated image processor can be used for anti-aliasing processing to achieve better anti-aliasing effects and lower power consumption.

[0054] Alternatively, if Figure 2 As shown, the image processing method provided in the embodiment of the present application is executed by an anti-aliasing module, which can be an algorithm module built into the GPU (Graphics Processing Unit) of the electronic device, or a separate hardware processor, which further processes the scene image output by the game. A method of combining the second image frame and the first image frame provided by the anti-aliasing module can better solve the flickering and jagged phenomena of continuous frames in graphics rendering.

[0055] Among them, GPU is a processor specially designed for processing graphics and images. GPU was originally designed for graphics rendering to accelerate the generation and display of computer graphics. With the development of technology, GPU has gradually been used for general computing, especially in the fields of deep learning, scientific computing, cryptography, etc.

[0056] In at least one embodiment of the present application, step 104 includes:

[0057] depth filtering according to the first depth value and the second depth value;

[0058] Optionally, performing depth filtering according to the first depth value and the second depth value includes:

[0059] The first depth value and the second depth value are compared to determine whether a difference between the first depth value and the second depth value is less than or equal to a first threshold.

[0060] Processing the first color value and the second color value according to the depth filtering result to obtain a credible color value of the first image frame and a credible color value of the second image frame;

[0061] Optionally, according to the depth filtering result, the first color value and the second color value are processed to obtain a credible color value of the first image frame and a credible color value of the second image frame, including:

[0062] If the difference between the first depth value and the second depth value is less than or equal to a first threshold, and the position deviation between the first sampling position and the second sampling position is less than or equal to a distance of 1 pixel, determine that the first color value is a credible color value of the first image frame, and determine that the second color value is a credible color value of the second image frame;

[0063] or,

[0064] If the difference between the first depth value and the second depth value is greater than a first threshold, or the position deviation between the first sampling position and the second sampling position is greater than 1 pixel distance, determine that the first color value is a credible color value of the first image frame, and adjust the color value of the second image frame to obtain a credible color value of the second image frame; the credible color value of the second image frame is between the maximum color value and the minimum color value in a first preset area of ​​the first pixel.

[0065] Performing color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain a target color value of the first image frame;

[0066] For example, the credible color value of the first image frame and the credible color value of the second image frame are mixed to set different weights. The following formula describes s t =αx t +(1-α)s t-1 , where α takes a small value, such as about 0.05, s t-1 Represents the color value of the second image frame; x t Indicates the color of the first image frame; s t Represents the color value obtained after color accumulation and mixing; doing so is equivalent to the weighted average result obtained after accumulating multiple second image frame data.

[0067] The target color value of the first image frame is transformed from the YCoCr color space to the RGB color space to obtain a first color map after anti-aliasing.

[0068] Color space refers to a mathematical model used to describe and represent colors in computer graphics and digital image processing. A color space is usually composed of a set of coordinate systems used to represent the values ​​of different colors. Common color spaces include RGB (red, green, and blue), CMYK (cyan, magenta, yellow, and black), YUV, and YCoCr.

[0069] For example, Figure 3The image processing method shown includes:

[0070] Reprojection operation: After rendering the N-1th frame (i.e., the second image frame), it is stored in an off-screen buffer image. When rendering the Nth frame (i.e., the first image frame), a certain pixel point (i.e., the first pixel point) is first reversely projected to determine whether a matching pixel point can be found in the frame buffer of the N-1th frame. If a matching pixel point (i.e., the second pixel point) is found in the second image frame, it is considered that the value of the pixel point can be used; otherwise, it is considered that the pixel point is blocked in the previous frame and its value cannot be used. For example, at the beginning of each frame rendering, the CPU calculates the mapping transformation matrix between the projection transformation matrix used to draw the current image frame and the previous image frame. The calculation method can be to combine the coordinates (U, V) of the current pixel point in the projection space with the pixel depth value at the current position in the previous frame depth map to form a 3D space coordinate p t , and then with Multiply to get position p t-1 , such as the formula: Describes the reprojection process.

[0071] Historical depth value: The pixel coordinate position (i.e., the position of the second pixel) projected to the previous frame obtained by reprojection is sampled on the historical depth map and the historical color map to obtain the historical depth value D1 and the historical color value HistoryColor. In this embodiment, it is mentioned that the HistoryColor value needs to be transformed from the RGB color space to the YCoCr color space; optionally, the transformation scheme can flexibly sample other color spaces, not limited to the YCoCr space;

[0072] Depth filtering and color minimum filtering: First, the color map and depth map of the first image frame are sampled, and the sampling position p t Then, combined with the historical depth value, it is compared to determine whether the current pixel has not produced a large depth change, such as when it is less than a relative threshold Y, and p t With p t-1 When the position deviation is less than 1 pixel, the color value of the current pixel is completely reliable; otherwise, HistoryColor is limited to the first image frame position p t Between the maximum color value and the minimum color value in the 3x3 area;

[0073] Color accumulation mixing: The credible color value of the first image frame is mixed with the credible color value of the second image frame to set different weights. The following formula describes s t =αx t +(1-α)s t-1, where α takes a small value, such as about 0.05, s t-1 Represents the color value of the second image frame; x t Indicates the color of the first image frame; s t Represents the color value obtained after color accumulation and mixing; this is equivalent to the weighted average result obtained after accumulating multiple second image frame data;

[0074] Neighboring extreme value filtering: For the current depth map, the depth M closest to the camera position in the 3x3 area can be used d (Some graphics device drivers use the minimum value, while others use the maximum value). d Output to a new depth map for calculation in the next frame (i.e., frame N+1);

[0075] Output color frame (also called color map): The result of color accumulation and mixing is first transformed into RGB space, and then the transformed value is output to a new color map for calculation in the next frame (i.e., the N+1th frame).

[0076] In order to further expand the calculation method, the embodiment of the present application incorporates an edge anti-flickering method into the color accumulation and mixing stage, thereby being able to better suppress the aliasing flicker problem.

[0077] In at least one embodiment of the present application, performing color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain the target color value of the first image frame includes:

[0078] Acquire a flickering edge region of the first image frame according to a credible color value of the first image frame;

[0079] It should be noted that the "flickering edge area of ​​the first image frame" mentioned in the embodiment of the present application refers to the area corresponding to the connection between the target object and the environment in the first image frame; for example, the connection is judged based on the comparison between the color value of the target object and the color value of the environment, thereby further determining the flickering edge area.

[0080] Performing a reprojection operation on the boundary pixel points of the flickering edge area to obtain historical boundary pixel points matching the boundary pixel points in the second image frame;

[0081] According to the historical boundary pixel points, query the direction of the historical boundary and determine the fuzzy weight value of the Gaussian blur area;

[0082] The pixel points in the Gaussian blur area are blurred using the blur weight value, and the brightness value of the blurred pixel point color is adjusted by brightness fusion to obtain the target color value of the first image frame after anti-aliasing and anti-flickering.

[0083] Wherein, the method further comprises:

[0084] Accumulating the brightness values ​​of the pixels matching the historical boundary pixels to form historical brightness values; correspondingly, performing brightness fusion adjustment on the brightness values ​​of the blurred pixel colors, including:

[0085] The brightness value of the blurred pixel color is averaged with the historical brightness value, and the brightness value of the blurred pixel color is adjusted to the average;

[0086] or,

[0087] Different weights are set for the brightness value of the blurred pixel color and the historical brightness value, the brightness value of the blurred pixel color and the historical brightness value are fused through the weights, and the brightness value of the blurred pixel color is adjusted to the value obtained by weight fusion.

[0088] Optionally, obtaining the flickering edge area of ​​the first image frame according to the credible color value of the first image frame includes:

[0089] The flickering edge area of ​​the first image frame is calculated according to the brightness value of the credible color value of the first pixel of the first image frame and the brightness values ​​of each pixel within the second preset area of ​​the first pixel.

[0090] For example, Figure 4 As shown, the process of incorporating the edge anti-flickering method into the color accumulation stage in the embodiment of the present application is as follows:

[0091] Step 4001, obtaining the current color, that is, the color value of the position Pt in the current image frame;

[0092] Step 4002, calculate the flicker edge area, the flicker area is calculated as follows Figure 5 As shown, four pixels in the center area of ​​the current pixel are taken for calculation. The method for extracting the boundary can be to analyze the difference between the brightness value (Luma value) of the four adjacent points and the Luma value of the center point, and determine it by setting a threshold or other methods;

[0093] Step 4003, reproject brightness and accumulate historical information; the flickering boundary pixels calculated in the previous step can be projected into the boundary information recorded in the Alpha channel of the second image frame through the reprojection method; the brightness value of the pixel at the location of the previous frame is obtained through mapping, and then the brightness is accumulated using the color accumulation method, where α takes a smaller value, such as about 0.05. When the mapped pixel point is not at the boundary position, a query judgment can be made within the 2x2 area. If the boundary information still does not exist, it can be determined that the boundary flickering area has changed, and the historical information needs to be discarded. The fused color value is completely determined by the current pixel;

[0094] Step 4004, Gaussian Blur 3×3 depends on and performs brightness fusion adjustment; when the mapping is successful, the direction of the query history boundary can be used, such as Figure 6 As shown, by judging whether there are boundary pixels directly above or directly below, the possible flicker direction can be determined as the adjacent points PtL and PtR in the horizontal direction. If there are boundary pixels directly to the left or directly to the right, the possible flicker direction is the adjacent points PtT and PtB in the vertical direction. Similarly, the boundary directions of 45 degrees and 135 degrees can be judged. By judging all the directions, the blur weight value of Gaussian Blur can be taken in the corresponding situation. The jagged edges at the boundary can be effectively suppressed by weight blurring. The anti-flicker adjustment can be obtained by averaging the Luma value of the blurred pixel color with the historical Luma value and adjusting its Luma value, or by using weight fusion for adjustment. This method effectively utilizes the time domain information to improve the time domain anti-flicker effect.

[0095] Step S4005, outputting the anti-aliasing and anti-flickering color; that is, outputting the calculated color value to the RGB channel of the new second image frame buffer, and at the same time, writing the boundary grayscale information of the current image frame into the alpha channel of the new second image frame.

[0096] In one implementation, the image processing method can be implemented alone, without being used in conjunction with MSAA, and can also achieve good anti-aliasing and anti-flickering effects, and can be implemented as a low-power temporal anti-aliasing solution.

[0097] In another implementation, the image processing method can be superimposed on the MSAA anti-aliasing provided by the electronic device hardware to further improve the anti-aliasing effect and anti-flickering capability. Electronic device hardware developers can place the process of the image processing method in the image processing chip as an image post-processing step to further improve the image quality of applications such as games, with little impact on the image drawing process of the game itself.

[0098] Post-processing refers to further processing of the generated images or videos in computer graphics processing or video editing. It aims to improve image quality, enhance visual effects or achieve a specific artistic style. Post-processing can include adjusting hue, contrast and brightness, adding filter effects, blurring or sharpening, and adding special effects.

[0099] MSAA: Multi Sample Anti-Aliasing, is an anti-aliasing technology used in computer graphics. Each pixel not only stores color information, but also stores additional sample information. When multiple samples are used to draw a pixel, the graphics hardware calculates the color differences between these samples and then uses this information to achieve anti-aliasing effects.

[0100] The embodiment of the present application can effectively combine the anti-aliasing algorithm effect of the electronic device GPU with the time sampling of the algorithm device of the present invention, which will not incur excessive time cost while increasing the sampling rate, significantly improves the anti-aliasing effect and further improves the anti-flicker effect.

[0101] In summary, in the embodiments of the present application, the image processing method can be implemented on electronic devices with relatively efficient hardware resources, and the anti-aliasing calculation process only relies on depth information and requires less information; the method does not need to calculate the motion vectors of all objects, reducing the required amount of calculations, memory usage and power consumption.

[0102] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, the image processing device provided in the embodiment of the present application is described by taking the method for executing anti-aliasing by the image processing device as an example.

[0103] like Figure 7 As shown, the embodiment of the present application also provides an image processing device, the device comprising:

[0104] A reprojection module 701 is used to perform a reprojection operation on each first pixel point of a first image frame; and determine second pixel points in a second image frame that match each first pixel point respectively; wherein the second image frame is a frame previous to the first image frame;

[0105] A first sampling module 702 is used to sample the first image frame by taking the position of each first pixel point as a first sampling position to obtain a first depth value and a first color value of each first sampling position of the first image frame;

[0106] A second sampling module 703 is used to sample the second image frame by taking the position of each second pixel point as a second sampling position to obtain a second depth value and a second color value of each second sampling position of the second image frame;

[0107] A first processing module 704 is used to perform depth filtering according to the first depth value and the second depth value, and perform color accumulation and mixing according to the depth filtering result, the first color value and the second color value to obtain a first color map;

[0108] The second processing module 705 is used to perform neighborhood extremum filtering on the depth map of the first image frame to obtain a first depth map;

[0109] The synthesis module 706 is used to synthesize the first color map and the first depth map to obtain a third image frame.

[0110] As an optional embodiment, the first processing module includes:

[0111] A first submodule, configured to perform depth filtering according to the first depth value and the second depth value;

[0112] A second submodule is used to process the first color value and the second color value according to the depth filtering result to obtain a credible color value of the first image frame and a credible color value of the second image frame;

[0113] A third submodule, configured to perform color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain a target color value of the first image frame;

[0114] The fourth submodule is used to transform the target color value of the first image frame from the YCoCr color space to the RGB color space to obtain a first color map.

[0115] As an optional embodiment, the second submodule includes:

[0116] a first determining unit, configured to determine, if a difference between the first depth value and the second depth value is less than or equal to a first threshold, and a position deviation between the first sampling position and the second sampling position is less than or equal to a distance of 1 pixel, the first color value is determined to be a credible color value of a first image frame, and the second color value is determined to be a credible color value of a second image frame;

[0117] and / or,

[0118] A second determining unit is configured to determine that the first color value is a credible color value of a first image frame, and adjust the color value of the second image frame to obtain a credible color value of the second image frame if a difference between the first depth value and the second depth value is greater than a first threshold, or a position deviation between the first sampling position and the second sampling position is greater than a distance of 1 pixel; and the credible color value of the second image frame is between a maximum color value and a minimum color value within a first preset area of ​​the first pixel.

[0119] As an optional embodiment, the second submodule includes:

[0120] A first acquisition unit, configured to acquire a flickering edge region of the first image frame according to a credible color value of the first image frame;

[0121] A second acquisition unit is used to perform a reprojection operation on the boundary pixel points of the flickering edge area to obtain historical boundary pixel points matching the boundary pixel points in the second image frame;

[0122] A third determining unit is used to query the direction of the historical boundary according to the historical boundary pixel point, and determine the fuzzy weight value of the Gaussian blur area;

[0123] The third acquisition unit is used to use the blur weight value to blur the pixels in the Gaussian blur area, and perform brightness fusion adjustment on the brightness value of the blurred pixel color to obtain the target color value of the first image frame after anti-aliasing and anti-flickering.

[0124] As an optional embodiment, the device further includes:

[0125] A brightness accumulation module, used to accumulate brightness values ​​of pixel points matched by the historical boundary pixel points to form historical brightness values;

[0126] The third acquisition unit is further used for:

[0127] The brightness value of the blurred pixel color is averaged with the historical brightness value, and the brightness value of the blurred pixel color is adjusted to the average;

[0128] or,

[0129] Different weights are set for the brightness value of the blurred pixel color and the historical brightness value, the brightness value of the blurred pixel color and the historical brightness value are fused through the weights, and the brightness value of the blurred pixel color is adjusted to the value obtained by weight fusion.

[0130] As an optional embodiment, the first acquiring unit includes:

[0131] The calculation subunit is used to calculate the flickering edge area of ​​the first image frame according to the brightness value of the credible color value of the first pixel point of the first image frame and the brightness values ​​of each pixel point within the second preset area of ​​the first pixel point.

[0132] In an embodiment of the present application, the image processing method can be implemented on an electronic device with relatively efficient hardware resources, and the anti-aliasing calculation process only relies on depth information and requires less information; the method does not need to calculate the motion vectors of all objects, thereby reducing the required amount of calculations, memory usage and power consumption.

[0133] It should be noted that the image processing device provided in the embodiment of the present application is a device capable of executing the above-mentioned image processing method. All embodiments of the above-mentioned image processing method are applicable to the device and can achieve the same or similar beneficial effects, which will not be repeated here.

[0134] The anti-aliasing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0135] The anti-aliasing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0136] The anti-aliasing device provided in the embodiment of the present application can achieve Figures 1 to 6 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0137] Alternatively, if Figure 8 As shown, an embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0138] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0139] Fig. 9 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0140] The electronic device 900 includes but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910 and other components.

[0141] Those skilled in the art will appreciate that the electronic device 900 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 910 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 9 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0142] Among them, the processor 910 is used to perform a reprojection operation on each first pixel point of the first image frame; and determine the second pixel points in the second image frame that match each first pixel point respectively; wherein the second image frame is the previous frame of the first image frame; taking the position of each first pixel point as the first sampling position, sampling the first image frame to obtain the first depth value and the first color value of each first sampling position of the first image frame; taking the position of each second pixel point as the second sampling position, sampling the second image frame to obtain the second depth value and the second color value of each second sampling position of the second image frame; performing depth filtering according to the first depth value and the second depth value, and performing color accumulation and mixing according to the depth filtering result, the first color value and the second color value to obtain a first color map; performing neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map; synthesizing the first color map and the first depth map to obtain a third image frame.

[0143] Optionally, the processor 910 is further used to perform depth filtering according to the first depth value and the second depth value; according to the depth filtering result, process the first color value and the second color value to obtain a credible color value of the first image frame and a credible color value of the second image frame; perform color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain a target color value of the first image frame; transform the target color value of the first image frame from the YCoCr color space to the RGB color space to obtain a first color map.

[0144] Optionally, the processor 910 is further configured to determine, if a difference between the first depth value and the second depth value is less than or equal to a first threshold, and a position deviation between a first sampling position and a second sampling position is less than or equal to a distance of 1 pixel, determine that the first color value is a credible color value of a first image frame, and determine that the second color value is a credible color value of a second image frame;

[0145] Alternatively, if the difference between the first depth value and the second depth value is greater than a first threshold, or the position deviation between the first sampling position and the second sampling position is greater than 1 pixel distance, the first color value is determined to be a credible color value of the first image frame, and the color value of the second image frame is adjusted to obtain a credible color value of the second image frame; the credible color value of the second image frame is between the maximum color value and the minimum color value within the first preset area of ​​the first pixel.

[0146] Optionally, the processor 910 is also used to obtain a flickering edge area of ​​the first image frame based on a trusted color value of the first image frame; perform a reprojection operation on the boundary pixels of the flickering edge area to obtain historical boundary pixels in the second image frame that match the boundary pixels; query the direction of the historical boundary based on the historical boundary pixels, and determine the blur weight value of the Gaussian blur area; use the blur weight value to blur the pixels of the Gaussian blur area, and perform brightness fusion adjustment on the brightness values ​​of the blurred pixel colors to obtain the target color value of the first image frame after anti-aliasing and anti-flickering.

[0147] Optionally, the processor 910 is further configured to accumulate brightness values ​​of the pixel points matched by the historical boundary pixel points to form historical brightness values;

[0148] The processor 910 is further configured to average the brightness value of the blurred pixel color and the historical brightness value, and adjust the brightness value of the blurred pixel color to the average value;

[0149] Alternatively, different weights are set for the brightness value of the blurred pixel color and the historical brightness value, the brightness value of the blurred pixel color and the historical brightness value are fused through the weights, and the brightness value of the blurred pixel color is adjusted to the value obtained by weight fusion.

[0150] In an embodiment of the present application, the image processing method can be implemented on an electronic device with relatively efficient hardware resources, and the anti-aliasing calculation process only relies on depth information and requires less information; the method does not need to calculate the motion vectors of all objects, thereby reducing the required amount of calculations, memory usage and power consumption.

[0151] It should be understood that in the embodiment of the present application, the input unit 904 may include a graphics processor (Graphics Processing Unit, GPU) 9041 and a microphone 9042, and the graphics processor 9041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 907 includes a touch panel 9071 and at least one of other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 may include two parts: a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0152] The memory 909 can be used to store software programs and various data. The memory 909 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 909 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 909 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0153] The processor 910 may include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 910.

[0154] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0155] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0156] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0157] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0158] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0159] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0161] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Performing a reprojection operation on each first pixel point of a first image frame, and determining second pixel points in a second image frame that respectively match each first pixel point, wherein the second image frame is a previous frame of the first image frame; Taking the position of each of the first pixel points as a first sampling position, sampling the first image frame to obtain a first depth value and a first color value of each of the first sampling positions of the first image frame; Taking the position of each of the second pixel points as a second sampling position, sampling the second image frame to obtain a second depth value and a second color value of each of the second sampling positions of the second image frame; Performing depth filtering according to the first depth value and the second depth value, and performing color accumulation and mixing according to the depth filtering result, the first color value, and the second color value to obtain a first color map; Performing neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map; The first color map and the first depth map are synthesized to obtain a third image frame.

2. The method according to claim 1, characterized in that Performing depth filtering according to the first depth value and the second depth value, and performing color accumulation and mixing according to the depth filtering result, the first color value, and the second color value to obtain a first color map, including: Performing depth filtering according to the first depth value and the second depth value; Processing the first color value and the second color value according to the depth filtering result to obtain a credible color value of the first image frame and a credible color value of the second image frame; Performing color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain a target color value of the first image frame; The target color value of the first image frame is transformed from the YCoCr color space to the RGB color space to obtain a first color map after anti-aliasing.

3. The method according to claim 2, characterized in that Processing the first color value and the second color value according to the depth filtering result to obtain a credible color value of the first image frame and a credible color value of the second image frame includes: If the difference between the first depth value and the second depth value is less than or equal to a first threshold, and the position deviation between the first sampling position and the second sampling position is less than or equal to a distance of 1 pixel, determine that the first color value is a credible color value of the first image frame, and determine that the second color value is a credible color value of the second image frame; or, If the difference between the first depth value and the second depth value is greater than a first threshold, or the position deviation between the first sampling position and the second sampling position is greater than 1 pixel distance, determine that the first color value is a credible color value of the first image frame, and adjust the color value of the second image frame to obtain a credible color value of the second image frame; the credible color value of the second image frame is between the maximum color value and the minimum color value in a first preset area of ​​the first pixel.

4. The method according to claim 2, characterized in that: Performing color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain a target color value of the first image frame includes: Acquire a flickering edge region of the first image frame according to a credible color value of the first image frame; Performing a reprojection operation on the boundary pixel points of the flickering edge area to obtain historical boundary pixel points matching the boundary pixel points in the second image frame; According to the historical boundary pixel points, query the direction of the historical boundary and determine the fuzzy weight value of the Gaussian blur area; The pixel points in the Gaussian blur area are blurred using the blur weight value, and the brightness value of the blurred pixel point color is adjusted by brightness fusion to obtain the target color value of the first image frame after anti-aliasing and anti-flickering.

5. The method according to claim 4, characterized in that The method further comprises: Accumulating the brightness values ​​of the pixel points that match the historical boundary pixel points to form a historical brightness value; The step of performing brightness fusion adjustment on the brightness value of the blurred pixel color includes: The brightness value of the blurred pixel color is averaged with the historical brightness value, and the brightness value of the blurred pixel color is adjusted to the average; or, Different weights are set for the brightness value of the blurred pixel color and the historical brightness value, the brightness value of the blurred pixel color and the historical brightness value are fused through the weights, and the brightness value of the blurred pixel color is adjusted to the value obtained by weight fusion.

6. An image processing device, characterized in that: The device comprises: a reprojection module, configured to perform a reprojection operation on each first pixel point of a first image frame; and determine second pixel points in a second image frame that respectively match each first pixel point, wherein the second image frame is a previous frame of the first image frame; A first sampling module, configured to sample the first image frame by taking the position of each of the first pixel points as a first sampling position, and obtain a first depth value and a first color value of each of the first sampling positions of the first image frame; A second sampling module, configured to sample the second image frame by taking the position of each of the second pixel points as a second sampling position, to obtain a second depth value and a second color value of each of the second sampling positions of the second image frame; A first processing module, configured to perform depth filtering according to the first depth value and the second depth value, and perform color accumulation and mixing according to the depth filtering result, the first color value, and the second color value to obtain a first color map; A second processing module, configured to perform neighborhood extremum filtering on the first depth value of the first image frame to obtain a first depth map; A synthesis module is used to synthesize the first color map and the first depth map to obtain a third image frame.

7. The device according to claim 6, characterized in that The first processing module comprises: A first submodule, configured to perform depth filtering according to the first depth value and the second depth value; A second submodule is used to process the first color value and the second color value according to the depth filtering result to obtain a credible color value of the first image frame and a credible color value of the second image frame; A third submodule, configured to perform color accumulation mixing on the credible color value of the first image frame and the credible color value of the second image frame to obtain a target color value of the first image frame; The fourth submodule is used to transform the target color value of the first image frame from the YCoCr color space to the RGB color space to obtain a first color map after anti-aliasing.

8. The device according to claim 7, characterized in that The second submodule includes: a first determining unit, configured to determine, if a difference between the first depth value and the second depth value is less than or equal to a first threshold, and a position deviation between the first sampling position and the second sampling position is less than or equal to a distance of 1 pixel, the first color value is determined to be a credible color value of a first image frame, and the second color value is determined to be a credible color value of a second image frame; and / or, A second determining unit is configured to determine that the first color value is a credible color value of a first image frame, and adjust the color value of the second image frame to obtain a credible color value of the second image frame if a difference between the first depth value and the second depth value is greater than a first threshold, or a position deviation between the first sampling position and the second sampling position is greater than a distance of 1 pixel; and the credible color value of the second image frame is between a maximum color value and a minimum color value within a first preset area of ​​the first pixel.

9. The device according to claim 7, characterized in that The third submodule includes: A first acquisition unit, configured to acquire a flickering edge region of the first image frame according to a credible color value of the first image frame; A second acquisition unit is used to perform a reprojection operation on the boundary pixel points of the flickering edge area to obtain historical boundary pixel points matching the boundary pixel points in the second image frame; A third determining unit is used to query the direction of the historical boundary according to the historical boundary pixel point, and determine the fuzzy weight value of the Gaussian blur area; The third acquisition unit is used to use the blur weight value to blur the pixels in the Gaussian blur area, and perform brightness fusion adjustment on the brightness value of the blurred pixel color to obtain the target color value of the first image frame after anti-aliasing and anti-flickering.

10. The device according to claim 9, characterized in that The device also includes: A brightness accumulation module, used to accumulate brightness values ​​of pixel points matched by the historical boundary pixel points to form historical brightness values; The third acquisition unit is further used for: The brightness value of the blurred pixel color is averaged with the historical brightness value, and the brightness value of the blurred pixel color is adjusted to the average; or, Different weights are set for the brightness value of the blurred pixel color and the historical brightness value, the brightness value of the blurred pixel color and the historical brightness value are fused through the weights, and the brightness value of the blurred pixel color is adjusted to the value obtained by weight fusion.

Images