Image filtering method and related apparatus

Abstract

The application relates to the technical field of image processing, and provides an image filtering method and related devices, which are used for solving the problem of how to filter an image in the related art. In the method, any point in a target image is taken as a target point, a spatial domain weight coefficient of the target point is determined according to the similarity of the target point and adjacent points, the coefficient has a positive correlation with the similarity, a value domain weight coefficient of the target point is determined according to the difference between the two side pixel points of the target point, the coefficient has a positive correlation with the difference, and the result of filtering of the target point is determined based on the spatial domain weight coefficient and the value domain weight coefficient. Therefore, the spatial domain weight coefficient is determined by using color similarity, for an intermediate point, if the intermediate point is similar to surrounding points, the spatial domain weight coefficient changes greatly and the purpose of smooth filtering can also be achieved, for an edge point, the difference between the two side pixel points of the edge point is large, the value domain weight coefficient changes greatly and the purpose of keeping the edge can be achieved, and thus the application provides an effective image filtering method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing technology, and in particular, to an image filtering method and related device. BACKGROUND

[0002] Motion Estimation and Motion Compensation (MEMC) can be understood as estimating the motion trajectory of multiple frames of images, predicting the motion vector of an object, and finally compensating for the frames that are not in the video source to achieve a smoother frame.

[0003] An important application scenario of motion estimation and motion compensation is video interpolation technology. Video interpolation technology (i.e., frame rate up-conversion technology) converts low frame rate video into high frame rate video by interpolating low frame rate video, thereby achieving video frame rate enhancement. Video interpolation technology can achieve dynamic frame compensation of the picture, ensure the coherence and clarity of the picture, and improve the visual effect and viewing experience.

[0004] Due to the large amount of motion estimation calculation and long time consumption, how to effectively filter the image before motion estimation to improve the accuracy of motion estimation is a problem to be solved. SUMMARY

[0005] Embodiments of the present application provide a filtering method and related device to solve the problem of how to effectively filter the image before motion estimation in related technologies.

[0006] In a first aspect, the present application provides an image filtering method, any point in a target image is taken as a target point, and the method comprises:

[0007] determining a spatial domain weight coefficient of the target point based on the similarity between the target point and the neighboring points, the spatial domain weight coefficient having a positive correlation with the similarity; and

[0008] determining a value domain weight coefficient of the target point based on the difference between the two pixel points on both sides of the target point, the value domain weight coefficient having a positive correlation with the difference;

[0009] determining the filtered result of the target point based on the spatial domain weight coefficient and the value domain weight coefficient.

[0010] Optionally, the determination of the spatial domain weight coefficient of the target point based on the similarity between the target point and the neighboring points specifically comprises:

[0011] selecting the pixel points in the horizontal neighborhood of the target point as the neighboring points;

[0012] The difference between the color value of the target point and the color value of each of the adjacent points is taken as an input parameter of the spatial domain model, to obtain a spatial domain correlation coefficient of the target point;

[0013] The spatial domain correlation coefficient of each pixel point in the vertical neighborhood of the target point is obtained;

[0014] The difference between the spatial domain correlation coefficient of the target point and the spatial domain correlation coefficient of each of the adjacent points is taken as an input parameter of the spatial domain model, to obtain the spatial domain weight coefficient of the target point.

[0015] Optionally, the spatial domain model comprises a first adaptive adjustment function, the first adaptive adjustment function has a negative correlation with the input difference, and the output of the spatial domain model has a positive correlation with the output of the first adaptive adjustment function.

[0016] Optionally, the spatial domain model comprises:

[0017] For n parameter components in the input parameter, a component coefficient corresponding to each parameter component is determined based on the following formula:

[0018] wherein

[0019] wherein, Yi O represents the component coefficient, CurveL() represents the first adaptive adjustment function, I O represents a parameter component of the target point, I q represents a parameter component of the adjacent point q, (u, v) represents a neighborhood range, and n is a positive integer greater than or equal to 1.

[0020] The sum of the component coefficients of each of the parameter components is taken as an output result of the spatial domain model.

[0021] Optionally, the value domain weight coefficient of the target point is determined based on the difference between the two side pixel points of the target point, and specifically comprises:

[0022] A horizontal value domain component is determined based on the color difference between the left neighborhood and the right neighborhood of the target point, and a vertical value domain component is determined based on the color difference between the upper neighborhood and the lower neighborhood of the target point;

[0023] The value domain weight coefficient is obtained by adopting a positive correlation between the horizontal value domain component, the vertical value domain component and the value domain weight coefficient.

[0024] Optionally, the horizontal value domain component is determined based on the color difference between the left neighborhood and the right neighborhood of the target point, and specifically comprises:

[0025] dividing a horizontal neighborhood of the target point into a left neighborhood and a right neighborhood which are left-right symmetrical based on the target point;

[0026] determining a first color value accumulation value of each pixel point in the left neighborhood and a second color value accumulation value of each pixel point in the right neighborhood;

[0027] taking a first difference value between the first color value accumulation value and the second color value accumulation value as an input of a second adaptive control function to obtain a horizontal value domain component;

[0028] The vertical value domain component is determined based on color difference of the upper neighborhood and the lower neighborhood of the target point, and specifically includes:

[0029] dividing a vertical neighborhood of the target point into an upper neighborhood and a lower neighborhood which are up-down symmetrical based on the target point;

[0030] determining a third color value accumulation value of each pixel point in the upper neighborhood and a fourth color value accumulation value of each pixel point in the lower neighborhood;

[0031] taking a second difference value between the third color value accumulation value and the fourth color value accumulation value as an input of the second adaptive control function to obtain the vertical value domain component.

[0032] Optionally, the first adaptive adjustment function is a function with an operation complexity lower than an exponential operation complexity.

[0033] Optionally, the second adaptive adjustment function is a function with an operation complexity lower than an exponential operation complexity.

[0034] In a second aspect, the present application also provides an electronic device, comprising:

[0035] a processor;

[0036] a memory for storing instructions executable by the processor;

[0037] The processor is configured to execute the instructions to implement any method provided in the first aspect of the present application.

[0038] In a third aspect, an embodiment of the present application also provides a computer readable storage medium, when instructions in the computer readable storage medium are executed by a processor of an electronic device, the electronic device can execute any method provided in the first aspect of the present application.

[0039] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, when the computer program is executed by a processor, any method provided in the first aspect of the present application is implemented.

[0040] The embodiments of the present application provide at least the following beneficial effects:

[0041] In the embodiments of the present application, the spatial domain weight coefficient is determined by color similarity. For the middle point, the color similarity with the surrounding points is similar, and the large change of the spatial domain weight coefficient can also achieve the purpose of smoothing filtering. For the edge point, the difference between the two sides of the pixel point is large, and the large change of the spatial domain weight coefficient can achieve the purpose of keeping the edge. Thus, the present application provides an effective image filtering method.

[0042] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Obviously, the drawings introduced below are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0044] Figure 1a The use scenario of the display device provided by the embodiments of the present application is provided.

[0045] Figure 1b The hardware configuration block diagram of the control device 100 provided by the embodiments of the present application is provided.

[0046] Figure 1c The hardware configuration block diagram of the display device 200 provided by the embodiments of the present application is provided.

[0047] Figure 1d The software configuration diagram in the display device 200 provided by the embodiments of the present application is provided.

[0048] Figure 2 One of the flowcharts of the image filtering method provided by an embodiment of the present application is provided.

[0049] Figure 3 The second flowchart of the image filtering method provided by an embodiment of the present application is provided.

[0050] Figure 4 The schematic diagram of the horizontal neighborhood and the vertical neighborhood provided by an embodiment of the present application is provided.

[0051] Figure 5 The third flowchart of the image filtering method provided by an embodiment of the present application is provided.

[0052] Figure 6 The effect diagram before and after the image filtering method provided by an embodiment of the present application is provided. DETAILED DESCRIPTION

[0053] In order to make the people in the neighborhood better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings.

[0054] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. As such, the data used in the description should be interchangeable under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0055] In the following, some terms in the embodiments of the present application are explained and described in order to facilitate the understanding of the skilled person in the art.

[0056] (1) The term "a plurality of" in the embodiments of the present application means two or more, and other quantifiers are similar.

[0057] (2) "And / or", which describes the association between the associated objects, means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. The character " / " generally represents that the associated objects before and after are in an "or" relationship.

[0058] (3) Bilateral filtering method, the traditional bilateral filtering algorithm is a kind of nonlinear bilateral filter, it can achieve the effect of keeping edge, noise reduction and smoothing. The weight of bilateral filtering not only considers the spatial domain (such as Gaussian low-pass filtering only considers the influence of position on the center pixel), but also considers the radiation difference of pixel range domain (such as the degree of similarity between pixels and center pixel in convolution kernel, color intensity, depth distance, etc.).

[0059] The spatial domain weight coefficient calculation formula of the traditional bilateral filtering is shown in formula (1):

[0060]

[0061] The value domain weight coefficient calculation formula is shown in formula (2):

[0062]

[0063] In formula (1) and formula (2), (i, j) is the center point coordinate, and (k, l) is the position coordinate of any point in the center neighborhood. σ d is the spatial domain smoothing coefficient, and σr is a value range smoothing coefficient, I() represents a pixel value at a corresponding coordinate position. d(i, j, k, l) represents a spatial domain weight coefficient at the center point (i, j), and r(i, j, k, l) represents a value range weight coefficient at the center point (i, j).

[0064] The kernel function of the bilateral filter is a comprehensive result of the spatial domain and the value range, and the bilateral filter weight calculation formula is obtained by multiplying the spatial domain weight coefficient and the value range weight coefficient.

[0065]

[0066] It can be known from formula (2) that in a flat area of an image, a value range change is very small, and a corresponding value range weight coefficient is close to 1. At this time, the spatial domain plays a main role, and is equivalent to Gaussian blur. In an edge area of the image, the value range change is very large, and the value range weight coefficient is large, so that edge information is maintained.

[0067] A weighted average value is calculated as a filtered value of the center point coordinate, as shown in formula (4):

[0068]

[0069] wherein, I lpf (i, j) represents a filtered value of the center point. Other parameters are described in the above formula, and will not be described here.

[0070] In the traditional bilateral filter method, an exponential operation exists, which leads to high computational complexity, and processing of two-dimensional data leads to large amount of computation. These determinations lead to the fact that the traditional bilateral filter method is difficult to implement in hardware. In view of these determinations of the traditional bilateral filter method, some embodiments of the present application also provide some improvement schemes, which will be pointed out one by one in the following, and will not be described here.

[0071] First, the use scenario of the display device provided by the present application will be described. The filtering method provided by the embodiments of the present application is applicable to an electronic device, for example, a smart terminal, a display device (such as a smart television) and the like, which needs to use image filtering.

[0072] Figure 1a A schematic diagram of the use scenario of the display device provided by the embodiments of the present application is shown in FIG. 2. As shown in FIG. 2, the display device 200 also communicates data with the server 400, and a user can operate the display device 200 through the smart device 300 or the control device 100. Figure 1a

[0073] ​In some embodiments, the control device 100 can be a remote controller, and the communication between the remote controller and the display device can include at least one of infrared protocol communication, Bluetooth protocol communication, and other short-distance communication modes, to control the display device 200 in a wireless or wired manner. The user can input user instructions through at least one of a button on the remote controller, voice input, control panel input, and the like, to control the display device 200.

[0074] In some embodiments, the smart device 300 can include any one of a mobile terminal, a tablet computer, a computer, a notebook computer, an AR / VR device, and the like.

[0075] In some embodiments, the smart device 300 can also be used to control the display device 200. For example, the display device 200 can be controlled by using an application program running on the smart device.

[0076] In some embodiments, the smart device 300 and the display device can also be used for data communication.

[0077] In some embodiments, the display device 200 can also be controlled in a manner other than the control device 100 and the smart device 300. For example, the display device 200 can directly receive voice instructions from a user through a voice instruction acquisition module configured inside the display device 200, or the display device 200 can receive voice instructions from a user through a voice control device configured outside the display device 200.

[0078] In some embodiments, the display device 200 can also communicate data with a server 400. The display device 200 can be communicatively connected through a local area network (LAN), a wireless local area network (WLAN), and other networks. The server 400 can provide various content and interactions to the display device 200. The server 400 can be a cluster or multiple clusters, and can include one or more types of servers.

[0079] In some embodiments, a software step performed by one execution subject can be migrated to another step execution subject in data communication therewith, as needed. For example, a software step performed by a server can be migrated to a display device in data communication therewith, as needed, and vice versa.

[0080] Figure 1b A configuration block diagram of the control device 100 is provided for embodiments of the present application. As shown in Figure 1b The control device 100 includes a controller 110, a communication interface 130, a user input / output interface 140, a memory, and a power supply. The control device 100 can receive input operation instructions from a user, and convert the operation instructions into instructions recognizable and responsive by the display device 200, to serve as an intermediary for interaction between the user and the display device 200.

[0081] In some embodiments, the communication interface 130 is configured to communicate with the outside world, and includes at least one of a WIFI chip, a Bluetooth module, an NFC module, or the like.

[0082] In some embodiments, the user input / output interface 140 includes at least one of a microphone, a touchpad, a sensor, a button, or the like.

[0083] The following embodiments are described with reference to the display device 200. It should be understood that, Figure 1c The display device 200 shown is merely an example, and the display device 200 can have more or fewer components than those shown, can combine two or more components, or can have a different configuration of components. The various components shown in the figure can be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits. Figure 1c

[0084] The following embodiments are described with reference to the display device 200. It should be understood that, Figure 1c The display device 200 shown is merely an example, and the display device 200 can have more or fewer components than those shown, can combine two or more components, or can have a different configuration of components. The various components shown in the figure can be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0085] In some embodiments, the display device 200 includes at least one of a tuner and demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, a user interface, and the like.

[0086] In some embodiments, the controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, a first interface to an n-th interface for input / output, and the like.

[0087] In some embodiments, the display 260 includes a display screen component for presenting a picture, a driving component for driving the image display, a component for receiving an image signal output from the controller, and a component for displaying video content, image content, and a menu control interface, and the like.

[0088] In some embodiments, the display 260 can be at least one of a liquid crystal display, an OLED display, and a projection display, and can also be a projection device and a projection screen.

[0089] In some embodiments, the tuner and demodulator 210 receives a broadcast television signal through a wired or wireless reception method, and demodulates audio and video signals, such as an EPG data signal, from a plurality of wireless or wired broadcast television signals.

[0090] In some embodiments, the communicator 220 is a component for communicating with external devices or servers according to various communication protocol types. For example, the communicator can include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near field communication protocol chips, and an infrared receiver. The display device 200 can establish transmission and reception of control signals and data signals with the control apparatus 100 or the server 400 through the communicator 220.

[0091] In some embodiments, the detector 230 is used to collect signals of external environment or external interaction. For example, the detector 230 includes a light receiver for collecting ambient light intensity, or the detector 230 includes an image collector such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures, or the detector 230 includes a sound collector such as a microphone, etc., for receiving external sound.

[0092] In some embodiments, the external device interface 240 can include, but is not limited to, any one or more of the following: a high-definition multimedia interface (HDMI), an analog or data high-definition component input interface (component), a composite video input interface (CVBS), a USB input interface (USB), an RGB port, etc. It can also be a composite input / output interface formed by the above multiple interfaces.

[0093] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, i.e., the tuner demodulator 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box, etc.

[0094] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations by storing various software control programs on the memory. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

[0095] In some embodiments, the object can be any one of selectable objects, such as a hyperlink, an icon or other operable control. The operation related to the selected object includes an operation of displaying a page connected to a hyperlink, a document, an image, etc., or an operation of executing a program corresponding to the icon.

[0096] In some embodiments, the controller includes at least one of a Central Processing Unit (CPU), a video processor, an audio processor, a Graphics Processing Unit (GPU), a RAM (Random Access Memory), a ROM (Read-Only Memory), a first interface to an n-th interface for input / output, a communication bus, and the like.

[0097] The CPU processor is configured to execute operating system and application program instructions stored in the memory, and execute various application programs, data and content according to various interactive instructions received from external input, so as to finally display and play various audio and video content. The CPU processor can include multiple processors. For example, it can include a main processor and one or more sub-processors.

[0098] In some embodiments, the GPU is configured to generate various graphical objects, such as at least one of an icon, an operation menu, and a user input instruction display graph, etc. The GPU includes an operator configured to perform operations by receiving various interactive instructions input by a user, and display various objects according to display attributes; and a renderer configured to render various objects obtained based on the operator, and the rendered objects are used for display on the display.

[0099] In some embodiments, the video processor is configured to receive external video signals, and perform at least one of video processing such as decompression, decoding, scaling, noise reduction, frame rate conversion, resolution conversion, and image synthesis according to a standard codec protocol of the input signals, so as to obtain signals directly displayable on the device 200.

[0100] In some embodiments, the video processor includes at least one of a demultiplexing module, a video decoding module, an image synthesizer, a frame rate conversion module, and a display formatting module. The demultiplexing module is configured to perform demultiplexing processing on input audio and video data streams. The video decoding module is configured to process the demultiplexed video signals, including decoding and scaling processing, etc. The image synthesizer is configured to perform superimposition and mixing processing on the video image after scaling processing and the GUI signal generated by the graphical generator according to user input or self-generation, so as to generate an image signal for display. The frame rate conversion module is configured to convert the input video frame rate. The display formatting module is configured to change the output signal of the video after frame rate conversion to a signal conforming to a display format, such as an output RGB data signal.

[0101] In some embodiments, the audio processor is configured to receive an external audio signal, decompress and decode the audio signal according to a standard codec protocol of the input signal, and perform at least one of noise reduction, digital-to-analog conversion, and amplification to obtain a sound signal that can be played on a speaker.

[0102] In some embodiments, the user can input a user command through a graphical user interface (GUI) displayed on the display 260, and the user input interface receives the user input command through the graphical user interface (GUI). Alternatively, the user can input a user command by inputting a specific sound or gesture, and the user input interface receives the user input command by recognizing the sound or gesture through a sensor.

[0103] In some embodiments, the "user interface" is a medium interface for interaction and information exchange between an application or an operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The commonly used form of the user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be an icon, a window, a control, etc. displayed on the display screen of an electronic device, wherein the control can include at least one of an icon, a button, a menu, a tab, a text box, a dialog box, a status bar, a navigation bar, a widget, etc.

[0104] In some embodiments, the user interface 280 is an interface (e.g., a physical button on the body of the display device, or the like) that can be used to receive a control input.

[0105] In some embodiments, the system of the display device can include a kernel, a shell, a file system, and an application. The kernel, the shell, and the file system together form a basic operating system structure, which allows a user to manage files, run programs, and use the system. After power-on, the kernel is started, the kernel space is activated, the hardware is abstracted, the hardware parameters are initialized, the virtual memory is run and maintained, the scheduler, the signal, and the inter-process communication (IPC) are maintained. After the kernel is started, the shell and the user application are loaded. The application is compiled into machine code after being started, forming a process.

[0106] Referring to Figure 1d In some embodiments, the system is divided into four layers from top to bottom, namely, an application layer (referred to as "application layer" for short), an application framework layer (referred to as "framework layer" for short), an Android runtime and system library layer (referred to as "system runtime library layer" for short), and a kernel layer.

[0107] In some embodiments, at least one application runs in the application layer. These applications may be Windows programs, system settings programs, or clock programs that come with the operating system; they may also be applications developed by third-party developers. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0108] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications in the application layer. Applications can access system resources and obtain system services during execution through the API interface.

[0109] like Figure 1d As shown, the application framework layer in this embodiment includes managers, content providers, etc., wherein the managers include at least one of the following modules: ActivityManager, which interacts with all activities running in the system; LocationManager, which provides access to system location services for system services or applications; PackageManager, which retrieves various information related to application packages currently installed on the device; NotificationManager, which controls the display and clearing of notification messages; and WindowManager, which manages icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0110] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling display window changes (e.g., shrinking the display window, shaking the display, distorting the display, etc.).

[0111] In some embodiments, the system runtime library layer provides support for the upper layer, namely the framework layer. When the framework layer is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer to implement the functions that the framework layer needs to perform.

[0112] In some embodiments, the kernel layer is a layer between hardware and software. For example... Figure 1dAs shown, the kernel layer at least includes at least one of the following drivers: an audio driver, a display driver, a Bluetooth driver, a camera driver, a WIFI driver, a USB driver, an HDMI driver, a sensor driver (such as a fingerprint sensor, a temperature sensor, a pressure sensor, etc.), and a power supply driver, etc.

[0113] To further illustrate the technical solutions provided by the embodiments of the present application, the following will combine the drawings and the specific embodiments to make a detailed description.

[0114] To further illustrate the technical solutions provided by the embodiments of the present application, the following will combine the drawings and the specific embodiments to make a detailed description. Although the embodiments of the present application provide the method operation steps as shown in the following embodiments or drawings, more or less operation steps can be included in the method based on conventional or non-creative labor. The execution order of the steps is not limited to the execution order provided by the embodiments of the present application in the logical sense.

[0115] Due to the imperfections of imaging systems, transmission media and recording devices, digital images are often contaminated by various noises during their formation, transmission and recording. In addition, when the input image object is not as expected in some links of image processing, noise will also be introduced into the resulting image. These noises often appear as isolated pixels or pixel blocks on the image, which cause strong visual effects. Generally, noise signals are not related to the object to be studied, and they appear in the form of useless information, disturbing the observable information of the image. For digital image signals, noise appears as large or small extreme values, which affect the image quality by adding or subtracting the true gray value of the image pixel, greatly reducing the image quality and affecting the subsequent work of image restoration, segmentation, feature extraction, image recognition, etc. To construct an effective noise suppression filter, two basic problems must be considered: effectively removing noise in the target and background; at the same time, the shape, size and specific geometric and topological structure features of the image target should be well protected.

[0116] Image filtering technology can remove noise as much as possible through smoothing processing, and protect the outline of the image target by preserving the edge. It is still worth studying to provide an effective and reliable image filtering method.

[0117] In view of this, the embodiments of the present application provide a feasible image filtering method. In the method, the color difference of the pixel point and the neighborhood point is used to determine the spatial weight coefficient to realize the smoothing processing of the image, wherein the color difference can be understood as the brightness difference, the gray difference, the color value difference. The color value is R value, G value and B value. When other color spaces are used, the color value can be taken as the basis of the color space used, for example, the color value in Yuv color space is Y value, u value and v value.

[0118] The smoothing processing can effectively remove noise, and in order to retain the edge features of the image, the color difference between the neighborhood points on both sides of the pixel point is used to determine the value range weight coefficient. In the application, when the pixel point is a middle point (i.e. a non-edge point), the spatial domain weight coefficient is relatively large, and the value range weight coefficient is relatively small, so as to realize the smoothing processing; when the pixel point is an edge point, the spatial domain weight coefficient is relatively small, and the value range weight coefficient is relatively large, so as to realize the edge retention.

[0119] Based on the inventive concept, an image filtering method is provided in the embodiments of the application. The filtering manner of each point is the same, and for the convenience of understanding, the following will be described from the perspective of a point, and the processing manner of other points is the same and will not be described herein.

[0120] Reference Figure 2 A flowchart of a filtering method provided by the embodiments of the application is shown in FIG. 1, and it is assumed that any point in the target image is taken as a target point, and the following operations are performed.

[0121] In step 201, a spatial domain weight coefficient of the target point is determined based on the similarity of the target point and the neighboring points, and the spatial domain weight coefficient has a positive correlation with the similarity.

[0122] In the embodiments of the application, when the difference value between the target point and the surrounding points is smaller, it indicates that the similarity is higher, and the spatial domain weight coefficient of the target point is larger, that is, the more similar the weight coefficient is, and vice versa, the less similar the spatial domain weight coefficient is, so as to realize the smoothing processing. The value range of the neighboring points can be configured according to actual requirements, and the application does not limit the value range.

[0123] In some possible implementation manners, in order to reduce the operation amount, a one-dimensional operation can be used instead of the traditional two-dimensional calculation of the bilateral filtering in the embodiments of the application, so as to make the image filtering method be realized by using hardware. In some possible implementation manners, the spatial domain weight coefficient of the target point can be determined based on the method shown in FIG. 2: Figure 3

[0124] In step 301, the pixel points in the horizontal neighborhood of the target point are selected as the neighboring points.

[0125] In step 302, the difference between the color value of the target point and the color value of each neighboring point is taken as an input parameter of the spatial domain model, so as to obtain the spatial domain correlation coefficient of the target point.

[0126] In step 303, the spatial domain correlation coefficients of the pixel points in the vertical neighborhood of the target point are obtained.

[0127] In step 304, the difference between the spatial domain correlation coefficient of the target point and the spatial domain correlation coefficient of each neighboring point is taken as an input parameter of the spatial domain model, so as to obtain the spatial domain weight coefficient of the target point. ​

[0128] For example, such as Figure 4 As shown, assuming each horizontal and vertical neighborhood comprises 5 pixels, the specific neighborhood range can be configured according to requirements; this application does not limit this. First, for each point, the spatial correlation coefficient Abi of the target point is obtained by using the color difference between its 5 horizontal neighbors and the target point Ai within that neighborhood. This yields an image composed of the spatial correlation coefficients of each point. Then, the spatial correlation coefficients of the 5 pixels within the vertical neighborhood of the target point are processed in this image to obtain the spatial weight coefficient Y0 of the target point. The processing method for other points is the same, thus obtaining the spatial weight coefficients of each target point.

[0129] In this embodiment, the two-dimensional calculation of traditional bilateral filtering is simplified to a one-dimensional calculation method, that is, the horizontal neighborhood calculation is performed first and then the vertical neighborhood calculation is performed, which can effectively simplify the amount of calculation and make the image filtering method provided in this application embodiment easy to implement in hardware.

[0130] In some embodiments, to improve the accuracy of image filtering, the spatial domain model includes a first adaptive adjustment function. The first adaptive adjustment function has a negative correlation with the difference between the input and the output, and the output of the spatial domain model has a positive correlation with the output of the first adaptive adjustment function. In this embodiment, the first adaptive adjustment function can be experimentally determined as needed to make the spatial weight coefficients more reasonable. During implementation, to reduce computational complexity and facilitate hardware implementation, the image filtering method provided in this embodiment has a first adaptive adjustment function whose computational complexity is less than that of a traditional bilateral filtering exponential function. For example, the input to the first adaptive adjustment function is a color difference, which can be achieved by calculating the difference. This difference can be, for example, the color difference between adjacent points and the target point. This color difference can be a brightness difference, a grayscale difference, or a color value difference. Therefore, by using the difference instead of exponential operations, the hardware implements integer operations instead of exponential operations, thereby reducing computational complexity and facilitating hardware implementation of the filtering method.

[0131] In one possible implementation, the spatial domain model can be implemented as follows:

[0132] For the n parameter components in the input parameters, the component coefficients corresponding to each parameter component are determined based on the following formula (5). Then, the sum of the component coefficients of each parameter component is used as the output result of the spatial domain model. Wherein, when using gray difference, the parameter components only include gray values; when using brightness difference, the parameter components only include brightness values; and when using color values ​​such as RGB values, R, G, and B are each a parameter component:

[0133] Among them W P =∑ q∈(u，v)CurveL(I O -I q |) (5)

[0134] wherein Yi O represents a component coefficient, CurveL() represents the first adaptive adjustment function, I O represents a parameter component of a target point, I q represents a parameter component of a neighboring point q, (u, v) represents a neighborhood range, and n is a positive integer greater than or equal to 1.

[0135] Taking RGB color values as an example, the component coefficients corresponding to the three color components of RGB are shown in formula (6):

[0136]

[0137] wherein YR O represents a component coefficient of an R component; YG O represents a component coefficient of a G component; YB O represents a component coefficient of a B component; Y O represents a spatial domain weight coefficient of a target point; IR O represents a value of an R component of the target point, IR q represents a value of an R component of the neighboring point q, IG O represents a value of a G component of the target point, IG q represents a value of a G component of the neighboring point q, IB O represents a value of a B component of the target point, IB q represents a value of a B component of the neighboring point q, and other parameters are the same as those in the foregoing formula and are not described herein again.

[0138] In some embodiments, in order to reduce the calculation complexity, the first adaptive function in the embodiments of the present application can be a linear function, or a linear segmented function, or other functions with low calculation complexity. Such functions can facilitate the hardware implementation of the filtering method.

[0139] In step 202, a value domain weight coefficient of the target point is determined based on a difference degree between the two side pixel points of the target point; wherein the value domain weight coefficient has a positive correlation with the difference degree.

[0140] If the difference between the two side pixel points of the target point is large, it indicates that the target point is an edge point, and therefore the value domain weight coefficient of the target point can be increased to retain the edge feature. On the contrary, if the difference between the two side pixel points of the target point is small, it indicates that the target point is a middle point, i.e., a non-edge point, and therefore the value domain weight coefficient of the target point is smaller to facilitate the smoothing processing.

[0141] Similarly, in order to facilitate the simplification of the amount of calculation, the range weight coefficient in the present application can also use one-dimensional data processing instead of two-dimensional data processing in the traditional bilateral filtering method, so that the image filtering method provided by the embodiments of the present application is more suitable for hardware implementation. For example, the method for determining the range weight coefficient of the target point can include the following steps as shown in Figure 5

[0142] In step 501, the color difference of the left and right neighborhoods of the target point, and the color difference of the upper and lower neighborhoods of the target point are determined.

[0143] In one possible implementation, the color difference of the left and right neighborhoods can be determined as shown in Figure 5

[0144] Step A1: dividing the horizontal neighborhood of the target point into left and right symmetrical left and right neighborhoods based on the target point;

[0145] As shown in Figure 4 , the target point Ai has two points in the left and right neighborhoods respectively, then the two points on the left of the target point Ai are the points in the left neighborhood, and the two points on the right of the target point Ai are the points in the right neighborhood.

[0146] Step A2: determining the first color value cumulative value of each pixel point in the left neighborhood, and determining the second color value cumulative value of each pixel point in the right neighborhood;

[0147] Continuing to take the 5-neighborhood as an example in Figure 4 , the color values of the two points in the left neighborhood are added to obtain the first color value cumulative value, and the color values of the two points in the right neighborhood are added to obtain the second color value cumulative value. The color value, for example, luminance, gray value, color value, etc., is applicable to the embodiments of the present application.

[0148] Step A3: taking the first difference value between the first color value cumulative value and the second color value cumulative value as the color difference of the left and right neighborhoods.

[0149] Similarly, as shown in Figure 5 , the color difference of the upper and lower neighborhoods of the target point can be determined as follows:

[0150] Step B1: dividing the vertical neighborhood of the target point into upper and lower symmetrical upper and lower neighborhoods based on the target point;

[0151] As shown in Figure 4 , the target point Abi has two points in the upper and lower neighborhoods respectively, then the two points on the upper of the target point Abi are the points in the upper neighborhood, and the two points on the lower of the target point Abi are the points in the lower neighborhood.

[0152] ​​Step B2: determine a third color value cumulative value of each pixel in the upper neighborhood, and determine a fourth color value cumulative value of each pixel in the lower neighborhood;

[0153] Step B3: take a second difference value between the third color value cumulative value and the fourth color value cumulative value as the color difference of the upper neighborhood and the lower neighborhood.

[0154] In step 502, take a first difference value between the first color value cumulative value and the second color value cumulative value as an input of a second adaptive control function, to obtain a horizontal value range component, and take a second difference value between the third color value cumulative value and the fourth color value cumulative value as an input of the second adaptive control function, to obtain the vertical value range component.

[0155] In step 503, take the horizontal value range component and the vertical value range component and the value range weight coefficient to be positively correlated, to obtain the value range weight coefficient.

[0156] Here, similar to the first adaptive control function, the second adaptive control function can also be measured by experiment. The second adaptive control function has a positive correlation with the input, so the greater the color difference of the left and right neighborhoods, the greater the output of the second adaptive control function, and the greater the value range weight coefficient. Similarly, the greater the color difference of the upper and lower neighborhoods of the edge point, the greater the corresponding value range weight coefficient, so that the edge feature can be preserved.

[0157] Taking a 7-neighborhood as an example, the horizontal value range component and the vertical value range component can be determined based on the following formula (7)

[0158]

[0159] wherein, G O represents the value range weight coefficient of the target point, G h represents the horizontal value range component, G v represents the vertical value range component, I q represents the color value of the pixel in the neighborhood of the target point, (x-i, y) represents the coordinate position of the point in the left neighborhood, (x, y) represents the coordinate position of the target point, (x+i, y) represents the coordinate position of the point in the right neighborhood, (x, y-i) represents the coordinate position of the point in the lower neighborhood, and (x, y+i) represents the coordinate position of the point in the upper neighborhood.

[0160] The color value used in formula (7) can be a gray value, a brightness value, or a color value. When it is a color value, since the color value includes multiple color components, each color component is calculated by formula (7) to obtain the G h of the color component, and then the G hThe sum of these values ​​serves as the horizontal range component of the target point. Similarly, the G value of each color component is calculated using formula (7). v Then G for all color components v The sum is taken as the vertical range component of the target point.

[0161] After obtaining the spatial domain weight coefficient and the value domain weight coefficient of the target point, in step 203, the filtered result of the target point is determined based on the spatial domain weight coefficient and the value domain weight coefficient.

[0162] For example, the sum of the spatial domain weight coefficients and the value domain weight coefficients can be used as the filtering result for each point. For a single frame of image, the filtered result is calculated as shown in formula (8):

[0163] I O =Y O +G O (8)

[0164] The filtered spatial domain weighting coefficients Y O Sum of range weight coefficients G O Add them together to obtain the filtered result I of the target point. O Spatial domain weighting coefficients apply smoothing filtering to the image, while range weighting coefficients achieve the effects of edge preservation and noise reduction.

[0165] Since image filtering can effectively remove noise and preserve image edges, inputting the filtered image into the motion estimation module for motion estimation can effectively improve the accuracy of motion estimation.

[0166] In summary, in this embodiment, the spatial domain weight coefficients are determined using color similarity. For intermediate points, which are similar to surrounding points, a large change in the spatial domain weight coefficients can still achieve the purpose of smoothing filtering. For edge points, where the differences between pixels on both sides are large, a large change in the spatial domain weight coefficients can achieve the purpose of preserving the edges. Figure 6 The image shown is a comparison before and after image filtering using the embodiments of this application. Figure 6 As can be seen, after value domain filtering, the image edges are preserved (as shown by the face edges in the image), which facilitates better extraction of edge information in subsequent motion estimation. After spatial domain filtering, noise points in the same area are smoothed out, and the overall image becomes blurrier than the original image. Therefore, the image filtering method of this application can more effectively remove noise while still effectively preserving image edge information. Thus, this application provides an effective image filtering method.

[0167] In addition, by defining the neighborhood as two one-dimensional neighborhoods (including a horizontal neighborhood and a vertical neighborhood), a one-dimensional data operation can be used instead of a traditional two-dimensional data operation of bilateral filtering, thereby reducing the amount of computation relative to the traditional bilateral filtering. By using an adaptive control function to process the color value difference instead of an exponential operation, the complexity of the operation can be effectively reduced.

[0168] Those skilled in the art of the technical field to which the present application pertains will appreciate that various aspects of the present application can be implemented as a system, method or program product. Accordingly, various aspects of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system."

[0169] In some possible embodiments, the electronic device according to the present application can include at least one processor and at least one memory. The memory stores program code which, when executed by the processor, causes the processor to perform the motion estimation method according to various exemplary embodiments of the present application described above in the specification. For example, the processor can perform the steps in the image filtering method.

[0170] In an exemplary embodiment, a computer program product is also provided, which includes a computer program that, when executed by the processor 131, implements any of the motion estimation methods provided by the present application.

[0171] In an exemplary embodiment, various aspects of the image filtering method provided by the present application can also be implemented in the form of a program product, which includes program code for causing a computer device to perform the steps in the image filtering method according to various exemplary embodiments of the present application described above in the specification when the program product is run on the computer device.

[0172] The program product can take any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium, for example, can be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0173] The program product for the image filtering method of the embodiments of the present application can employ a portable compact disc read only memory (CD-ROM) and include a program code, and can be executed on an electronic device. However, the program product of the present application is not limited thereto, and in the present document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0174] The readable signal medium can include a data signal propagated in a baseband or as a carrier wave in a propagation medium, in which the readable program code is embodied. Such a propagated signal can take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. The readable signal medium can also be any readable medium that is not a storage medium or is not a transmission medium, which can be used to store or transmit the program for use by or in connection with the instruction execution system, apparatus, or device.

[0175] The program code embodied on the readable medium can be transmitted using any suitable medium, including, but not limited to, wireless, wired, optical fiber cable, RF, etc., or any suitable combination thereof.

[0176] The program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's electronic device, partly on the user's electronic device, as a stand-alone software package, partly on the user's electronic device and partly on a remote electronic device or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user's electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device (for example, through the Internet using an Internet Service Provider).

[0177] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by multiple units.

[0178] Moreover, although the operations of the method(s) herein can be described in a particular, sequential order, this order is not meant to be a limitation and

[0179] Those of skill in the art would understand that embodiments of the present application can be provided as a method, system, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0180] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0181] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart illustrations and / or block diagrams block or blocks.

[0182] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0183] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0184] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method of image filtering, characterized by, The method comprises: Selecting a pixel point in a horizontal neighborhood of the target point as a neighboring point; taking a difference between a color value of the target point and a color value of each neighboring point as an input parameter of the spatial domain model to obtain a spatial domain correlation coefficient of the target point; Obtaining a spatial domain correlation coefficient of each pixel point in a vertical neighborhood of the target point; Taking a difference between the spatial domain correlation coefficient of the target point and a spatial domain correlation coefficient of each neighboring point as an input parameter of the spatial domain model to obtain a spatial domain weight coefficient of the target point; Determining a horizontal value domain component based on a color difference between a left neighborhood and a right neighborhood of the target point, and determining a vertical value domain component based on a color difference between an upper neighborhood and a lower neighborhood of the target point; Obtaining the value domain weight coefficient by using a positive correlation relationship between the horizontal value domain component and the vertical value domain component and the value domain weight coefficient; Determining a filtered result of the target point based on the spatial domain weight coefficient and the value domain weight coefficient.

2. The method of claim 1, wherein, The spatial domain model comprises a first adaptive adjustment function, the first adaptive adjustment function has a negative correlation relationship with the input difference, and an output of the spatial domain model has a positive correlation relationship with an output of the first adaptive adjustment function.

3. The method according to claim 1 or 2, characterized in that, The spatial domain model comprises: For n parameter components in the input parameter, a component coefficient corresponding to each parameter component is determined based on the following formula: wherein wherein Yi O represents a component coefficient, CurveL() represents the first adaptive adjustment function, I O represents a parameter component of a target point, I q represents a parameter component of a neighboring point q, (u, v) represents a neighborhood range, and n is a positive integer greater than or equal to 1. The sum of the component coefficients of each parameter component is taken as an output result of the spatial domain model.

4. The method of claim 1, wherein, The horizontal value domain component is determined based on a color difference between a left neighborhood and a right neighborhood of the target point, and specifically comprises: The horizontal neighborhood of the target point is divided into a left neighborhood and a right neighborhood which are left-right symmetrical based on the target point; A first color value cumulative value of each pixel point in the left neighborhood is determined, and a second color value cumulative value of each pixel point in the right neighborhood is determined; A first difference value between the first color value cumulative value and the second color value cumulative value is taken as an input of a second adaptive control function to obtain the horizontal value domain component. The vertical value domain component is determined based on a color difference between an upper neighborhood and a lower neighborhood of the target point, and specifically comprises: The vertical neighborhood of the target point is divided into an upper neighborhood and a lower neighborhood which are up-down symmetrical based on the target point; A third color value cumulative value of each pixel point in the upper neighborhood is determined, and a fourth color value cumulative value of each pixel point in the lower neighborhood is determined; A second difference value between the third color value cumulative value and the fourth color value cumulative value is taken as an input of the second adaptive control function to obtain the vertical value domain component.

5. The method of claim 2, wherein, The first adaptive adjustment function is a function with a lower operation complexity than an exponential operation complexity.

6. The method of claim 4, wherein, The second adaptive control function is a function with a lower operation complexity than an exponential operation complexity.

7. An electronic device, comprising: It comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method of any one of claims 1-6.

Images