Image processing method and device, electronic equipment and storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a storage medium. The image processing method comprises the following steps: acquiring M to-be-processed images; m hardware execution units respectively carry out median filtering operation on the M to-be-processed images to obtain a median filtering result of the input image, each hardware execution unit comprises L pieces of thread hardware, and the L pieces of thread hardware carry out median filtering operation on the to-be-processed images input into the hardware execution units in parallel to obtain the median filtering result of the input image. Each piece of thread hardware executes median filtering operation of n pixel points at a time, n is a positive integer, and the step of executing the median filtering operation of the n pixel points through the thread hardware comprises the substeps that filtering areas corresponding to the n pixel points are determined; ranking the pixel points in the filtering area in the row direction, the column direction and the diagonal direction, and removing the pixel points which are impossible to be median in the ranked filtering area; and determining a median filtering result of each pixel point in the n pixel points based on the remaining pixel points.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to an image processing method, a processor, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] Median filtering is a commonly used signal processing technique, mainly used to remove noise from data, especially salt-and-pepper noise. Salt-and-pepper noise is a type of randomly appearing black and white pixels that looks like salt and pepper sprinkled on an image.

[0003] The median filter works by replacing the value of each pixel with the median value of all the pixels in its neighborhood. The "neighborhood" here can be a 3x3, 5x5, or other sized matrix, depending on the size of the filter window. The median filter does not blur the image like some low-pass filters do, so it is very effective in removing noise while preserving the edges of the image. Summary of the invention

[0004] At least one embodiment of the present disclosure provides an image processing method, comprising: acquiring M images to be processed, wherein the M images to be processed are obtained by respectively expanding the edges of M block images, and the M block images are obtained by evenly dividing an input image into M blocks in a column direction, where M is a positive integer; performing median filtering operations on the M images to be processed by M hardware execution units to obtain median filtering results of the input images, wherein each hardware execution unit includes L thread hardware, and the L thread hardware performs the median filtering operation on the image to be processed input to the hardware execution unit in parallel, and each thread hardware performs the median filtering operation of n pixels at a time, where n is a positive integer, wherein the thread hardware performs the median filtering operation of the n pixels, comprising: determining a filtering area corresponding to the n pixels; sorting the pixels in the filtering area in a row direction, a column direction, and a diagonal direction, and removing the pixels in the sorted filtering area that cannot be the median; and determining the median filtering result of each of the n pixels based on the remaining pixels.

[0005] For example, in the image processing method provided by at least one embodiment of the present disclosure, the pixel points in the filter area are sorted in the row direction, column direction and diagonal direction, and the pixel points in the sorted filter area that cannot be the median are removed, including: sorting the pixel points in the filter area in the row direction and the column direction to obtain a first filter area, wherein, in the first filter area, the pixel values ​​of the pixel points in each row are in increasing order in the row direction, and the pixel values ​​of the pixel points in each column are in increasing order in the column direction; removing the pixel points in the first filter area whose coordinates do not meet the first median condition to obtain a second filter area; sorting the second filter area in the diagonal direction to obtain a third filter area, wherein, in the third filter area, the pixel values ​​of the pixel points on the diagonal are in increasing order in the diagonal direction; removing the pixel points in the third filter area that do not meet the second median condition.

[0006] For example, in the image processing method provided by at least one embodiment of the present disclosure, a pixel point whose coordinates satisfy the following relationship is determined not to satisfy the first median condition: x×y-1≥(c×d) / 2 (c-x+1)×(d-y+1)-1≥(c×d) / 2 Wherein, x and y represent the coordinates of the pixel point, c and d represent the size of the filtering area, and x, y, c, and d are positive integers.

[0007] For example, in the image processing method provided by at least one embodiment of the present disclosure, a pixel point that satisfies the following relationship is determined not to satisfy the second median condition: Plt>(R+1) / 2 or Prt>(R+1) / 2 Among them, R represents the number of pixel points in the second filtering area, Plt represents the number of pixel points in the third filtering area whose pixel values ​​are greater than or equal to the pixel value of the pixel point and are located on the first side of the pixel point, Prt represents the number of pixel points in the third filtering area whose pixel values ​​are less than or equal to the pixel value of the pixel point and are located on the second side of the pixel point, and the pixel values ​​of the pixel points increase along the direction from the first side to the second side.

[0008] For example, in the image processing method provided by at least one embodiment of the present disclosure, in response to n being greater than 1, the n pixel points include a rows and b columns of pixel points arranged in an array in the image to be processed, a and b are positive integers, and at least one of a and b is greater than 1, determining the filtering areas corresponding to the n pixel points includes: determining the filtering window areas corresponding to the n pixel points respectively, wherein the filtering window area corresponding to each pixel point includes an area centered on the pixel point and defined by the filtering window of the median filtering operation; according to the filtering window areas corresponding to the n pixel points respectively, determining the overlapping part of the filtering window areas corresponding to the n pixel points respectively as the filtering area corresponding to the n pixel points, wherein the size of the filtering area corresponding to the n pixel points is c×d, the size of the filtering window is h×w, c, d, h, and w are positive integers, c is less than h, and d is less than w.

[0009] For example, in the image processing method provided by at least one embodiment of the present disclosure, the median filtering result of each pixel among the n pixels is determined based on the remaining pixels, including: for any pixel among the n pixels: performing at least one first operation on the pixels that do not belong to the filtering area in the filtering window area corresponding to the any pixel and the remaining pixels, and obtaining the median point in the filtering window area corresponding to the any pixel as the median filtering result of the any pixel, wherein in each first operation, the first queue and the second queue are obtained, and the first queue and the second queue are merged and sorted at least once to obtain an output queue, wherein the number of elements in the output queue is r+1, r is the number of pixels in the filtering window area corresponding to the any pixel that have not undergone the first operation, and the output queue is r+1 elements whose values ​​are in the middle of the first queue and the second queue.

[0010] For example, in the image processing method provided by at least one embodiment of the present disclosure, the first queue is the remaining pixel points or the output queue obtained by the last first operation, and the second queue is c pixel points that have not undergone the first operation and are adjacent to the filter area in the row direction or d pixel points adjacent to the column direction, the c pixel points belong to the same column, and the d pixel points belong to the same row.

[0011] For example, in the image processing method provided by at least one embodiment of the present disclosure, when r=1, two elements in the output queue obtained by the current first operation and one element in the filtering window area corresponding to any pixel point that has not undergone the first operation are sorted, and the median point of the sorted three elements is used as the median filtering result of any pixel point.

[0012] For example, in the image processing method provided by at least one embodiment of the present disclosure, the merge sorting operation includes: obtaining two input pixel queues, wherein each input pixel queue is ordered; merging and sorting the two input pixel queues to remove the smallest t elements and the largest t elements in the two input pixel queues, wherein t is a positive integer greater than 1; and using the remaining elements in the two input pixel queues as a merged pixel queue, wherein the merged pixel queue is at least partially ordered.

[0013] For example, in the image processing method provided by at least one embodiment of the present disclosure, in response to n being equal to 1, the filtering area corresponding to the n pixel points is determined, including: based on the filtering window of the median filtering operation, limiting the area centered on the one pixel point in the image to be processed as the filtering area.

[0014] For example, in the image processing method provided by at least one embodiment of the present disclosure, the median filtering result of each pixel point among the n pixel points is determined based on the remaining pixel points, including: sorting the remaining pixel points, and determining the sorted median point as the median filtering result of the one pixel point.

[0015] For example, in the image processing method provided by at least one embodiment of the present disclosure, different thread hardware in the same hardware execution unit executes the median filtering operation on different pixels in the same execution cycle.

[0016] For example, in the image processing method provided by at least one embodiment of the present disclosure, each hardware execution unit includes a vector operation unit cache area, which is used for data caching of thread hardware in the hardware execution unit, and obtains M images to be processed, including: dividing the input image into the M block images in the column direction; pre-fetching the M block images into the vector operation unit cache areas of the M hardware execution units respectively; loading the pixel points of the edge areas obtained by performing the edge expansion on the M block images respectively into the vector operation unit cache areas of the corresponding hardware execution units, so as to directly obtain the corresponding image to be processed from the vector operation unit cache area of ​​the hardware execution unit when each hardware execution unit performs the median filtering operation.

[0017] For example, in the image processing method provided by at least one embodiment of the present disclosure, the pixel points of the edge area obtained by respectively performing the edge expansion on the M block images are loaded into the cache area of ​​the vector operation unit of the corresponding hardware execution unit, including: for each block image: expanding the upper side and the lower side of the block image in a direction away from the block image by p rows of pixel points, and expanding the left side and the right side of the block image in a direction away from the block image by q columns of pixel points, so as to obtain the image to be processed and the edge area of ​​the block image, wherein the edge area is the image to be processed. Processing an area of ​​the image that does not belong to the block image, wherein p and q are positive integers; determining the coordinates of a plurality of pixel points in the edge area that belong to an upper side adjacent block image or a lower side adjacent block image of the block image, and loading the plurality of pixel points into a vector operation unit buffer of the corresponding hardware execution unit; obtaining pixel values ​​of the remaining pixel points in the edge area that do not belong to the upper side adjacent block image and the lower side adjacent block image according to a preset filling rule, and loading the remaining pixel points into the vector operation unit buffer of the corresponding hardware execution unit.

[0018] For example, in the image processing method provided by at least one embodiment of the present disclosure, the median filtering results obtained by the L thread hardware performing the median filtering operation are spliced ​​according to the position relationship of the pixel points to obtain the filtering result of the median filtering operation performed by each hardware execution unit on the input image to be processed; the filtering results generated by each of the M hardware execution units are spliced ​​according to the relative position relationship of the M images to be processed in the row direction to obtain the median filtering result of the input image.

[0019] At least one embodiment of the present disclosure provides an image processing device, comprising: an acquisition module, configured to acquire M images to be processed, wherein the M images to be processed are obtained by respectively expanding the edges of M block images, and the M block images are obtained by evenly dividing an input image into M blocks in a column direction, where M is a positive integer; M hardware execution units, configured to respectively perform median filtering operations on the M images to be processed to obtain median filtering results of the input images, wherein each hardware execution unit comprises L thread hardware, and the L thread hardware is configured to perform the median filtering operations on the images to be processed input to the hardware execution unit in parallel, and each thread hardware is configured to perform median filtering operations on n pixels at a time, where n is a positive integer, wherein when the thread hardware performs the median filtering operations on the n pixels, the following operations are performed: determining a filtering area corresponding to the n pixels; sorting the pixels in the filtering area in a row direction, a column direction, and a diagonal direction, and removing pixels in the sorted filtering area that cannot be medians; and determining a median filtering result for each of the n pixels based on the remaining pixels.

[0020] For example, in the image processing device provided by at least one embodiment of the present disclosure, the hardware execution unit includes multiple computing units, each computing unit includes multiple execution units, and each execution unit includes multiple thread hardware.

[0021] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the image processing method according to any embodiment of the present disclosure.

[0022] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the image processing method according to any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0024] Figure 1 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU); Figure 2 A schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure; Figure 3 A schematic diagram of the relationship between an image to be processed and a block image provided in an embodiment of the present disclosure; Figure 4 A schematic diagram of a pixel processing sequence in an image to be processed provided by an embodiment of the present disclosure; Figure 5 A schematic flow chart of a median filtering operation provided for at least one embodiment of the present disclosure; Figure 6 A process diagram of step S202 provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of a filtering area provided for at least one embodiment of the present disclosure; Figure 8 A schematic block diagram of an image processing device provided by at least one embodiment of the present disclosure; Fig. 9 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure; Fig.10A schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0026] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0027] Figure 1 A schematic diagram of a general-purpose graphics processor.

[0028] like Figure 1 As shown, the general-purpose graphics processor is actually an array of programmable multiprocessors. For example, the programmable multiprocessor can be a streaming processor cluster (SPC), for example, including Figure 1 The stream processor clusters 1, ..., and stream processor cluster M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one stream processor cluster processes one computing task, or multiple stream processor clusters process one computing task. Multiple stream processor clusters share data through a global cache or a global memory.

[0029] like Figure 1 As shown, taking stream processor cluster 1 as an example, one stream processor cluster includes multiple computing units, such as Figure 1In the figure, there are computing units 1, 2, ..., N, where N is a positive integer. Each computing unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A computing unit includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic logic unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the computing unit also includes registers (e.g. Figure 1 The register file in the computing unit and the shared memory are used to hierarchically store the source data and destination data related to the computing task. The shared memory in a computing unit is used to share data between the cores of the computing unit. Figure 1 As shown, each stream processor cluster also includes a vector operation unit cache (GEMM-Main-Buffer, referred to as GMB), which is shared by the computing units in the same stream processor cluster and is used to share data between the computing units belonging to the same stream processor cluster.

[0030] For example, the register stack can be an extended thread local register (Thread-Local-Register, abbreviated as TLR), which is closest to the computing unit and has a smaller capacity; the shared memory (Global-Shared-Memory, abbreviated as GSM) is closer to the computing unit, but farther from the computing unit than the TLR, and has a smaller capacity; the vector operation unit cache (GEMM-Main-Buffer, abbreviated as GMB) is closer to the computing unit than the global cache (such as L2 cache), but farther from the computing unit than the shared memory, and has a larger capacity.

[0031] In parallel computing, computing tasks are generally performed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then distributed through the thread block distribution module ( Figure 1 The CPU (not shown) distributes multiple thread blocks to each computing unit. All threads in a thread block must be assigned to the same computing unit for execution. At the same time, the thread block is split into minimum execution thread bundles (or simply thread bundles, warps), each of which contains a fixed number of threads (or less than this fixed number), for example, 32 threads. Multiple thread blocks can be executed in the same computing unit or in different computing units.

[0032] In each compute unit, the warp scheduling / dispatching module ( Figure 1The thread warps are scheduled and allocated (not shown) so that multiple computing cores of the computing unit can run the thread warps. Depending on the number of computing cores in the computing unit, multiple thread warps in a thread block can be executed simultaneously or in time-sharing. Multiple threads in each thread warp will execute the same instruction. The memory execution instruction will be emitted to the shared memory in the computing unit or further emitted to the intermediate level cache (such as the vector operation unit cache) or the global cache or the global memory for read and write operations, etc.

[0033] There are currently two mainstream methods for median filtering of images. One is the conventional method of sorting and taking the median, and the other is the method of using a histogram to obtain the median for larger resolution images.

[0034] The principle of the method of sorting and taking the median is to first select a neighborhood centered on the current pixel, then sort all the pixel values ​​in the neighborhood, and finally select the sorted median as the filter value of the current pixel.

[0035] The method of taking the median of the histogram is to regard the entire image as a sliding window. During the movement process (for example, from left to right), there is a shared part in the middle of two adjacent filters, and the newly added part is the part on the right side of the neighborhood, so the histogram can be used to update the pixel value.

[0036] For the first filtering method of sorting and taking the median, the complexity of the median filter depends on the complexity of the sorting algorithm. In addition, when the filtering window is large, many repeated calculations are introduced, resulting in serious algorithm time consumption and poor performance.

[0037] For the second filtering method of taking the median of the histogram, in the horizontal direction, the filtering result of each pixel point needs to wait until the filtering calculation of the adjacent previous (for example, the left) pixel point is completed. That is, the filtering process has data dependence in the row direction, which is not suitable for parallel computing on multi-threaded hardware computing resources (such as graphics processors or general-purpose graphics processors).

[0038] At least one embodiment of the present disclosure provides an image processing method, a processor, an electronic device, and a non-transitory computer-readable storage medium. The image processing method includes: obtaining M images to be processed, wherein the M images to be processed are obtained by respectively expanding the edges of M block images, and the M block images are obtained by evenly dividing the input image into M blocks in the column direction, where M is a positive integer; performing median filtering operations on the M images to be processed by M hardware execution units to obtain the median filtering results of the input images, wherein each hardware execution unit includes L thread hardware, and the L thread hardware performs the median filtering operation of the image to be processed of the input hardware execution unit in parallel, and each thread hardware performs the median filtering operation of n pixels at a time, where n is a positive integer, wherein the thread hardware performs the median filtering operation of n pixels, including: determining the filtering area corresponding to the n pixels; sorting the pixels in the filtering area in the row direction, the column direction, and the diagonal direction, and removing the pixels in the sorted filtering area that cannot be the median; and determining the median filtering result of each pixel in the n pixels based on the remaining pixels.

[0039] In at least one embodiment, the image processing method can utilize multiple thread hardware to perform median filtering operations on pixels in parallel, thereby making full use of multi-thread hardware computing resources, accelerating the processing speed of the median filtering operator in the image algorithm, and improving the execution efficiency of the image algorithm; in addition, the image processing method combines the characteristics of multi-thread hardware computing resources to provide a median filtering operation (or median filtering operator) to accelerate the sorting process.

[0040] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0041] Figure 2 A schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure.

[0042] like Figure 2 As shown, the image processing method provided by at least one embodiment of the present disclosure includes steps S10-S20.

[0043] In step S10, M images to be processed are obtained.

[0044] For example, M images to be processed are obtained by respectively expanding the edges of M block images, and the M block images are obtained by dividing the input image into M blocks in a column direction (for example, a vertical direction), where M is a positive integer.

[0045] For example, the number of rows of the M block images may be the same or different.

[0046] For example, the input image can be divided into M blocks in the column direction to obtain M block images, for example, each block image can include H / M rows and W columns, H is the number of rows of the input image, W is the number of columns of the input image, and H and W are positive integers. For example, if H is not an integer multiple of M, the number of rows of some block images can be set to be greater than H / M, so that the M block images can cover all pixels in the input image. In this way, the hardware execution units can make full use of their respective computing power, balance the amount of median filtering calculations, and obtain the final median filtering result of the input image at the fastest speed.

[0047] For example, refer to Figure 2 In the general-purpose graphics processor shown, each hardware execution unit includes a vector operation unit cache area, which is used for data caching of thread hardware in the hardware execution unit. For example, N computing units belonging to a hardware execution unit can share the vector operation unit cache area, which is different from the cache mechanism of the conventional L1 cache or L2 cache and can provide data storage function.

[0048] For example, in this embodiment, step S10 may include: dividing the input image into M block images in the column direction; pre-fetching the M block images into the vector operation unit caches of M hardware execution units respectively; loading the pixel points of the edge areas obtained by expanding the M block images respectively into the vector operation unit caches of the corresponding hardware execution units, so as to directly obtain the corresponding image to be processed from the vector operation unit cache of the hardware execution unit when each hardware execution unit performs the median filtering operation.

[0049] In this embodiment, the edge expansion process does not need to open up memory space to generate a new image data, but uses the "original site mapping" method to expand the edge. Specifically, the coordinate positions of the pixel points belonging to the edge area are recalculated through logical judgment and directly read. This can save memory overhead and time consumption caused by opening up memory.

[0050] Figure 3 A schematic diagram of the relationship between an image to be processed and a block image provided in one embodiment of the present disclosure.

[0051] like Figure 3 As shown, the gray rectangular area is the block image. Figure 3 , three adjacent block images are shown: block image 1, block image 2, and block image 3. For example, the input image may also include more block images, which are not shown one by one here.

[0052] Taking the block image 2 as an example, the dotted frame around the block image 2 is the range of the image to be processed obtained by expanding the edge of the block image 2. For example, the area defined by the dotted frame is the image to be processed, and the image to be processed includes the block image 2 in the middle. The part around the block image 2 is called the edge area, which is the area in the image to be processed that does not belong to the block image 2, that is, Figure 3 The black stripe area around the middle block image 2.

[0053] For example, the upper side and the lower side of the block image 2 are extended by p rows of pixels in a direction away from the block image 2, for example, p is related to the size of the filter window of the median filter operation, for example, p=h / 2; the left side and the right side of the block image 2 are extended by q columns of pixels in a direction away from the block image, for example, q is related to the size of the filter window of the median filter operation, for example, q=w / 2, where q and p are positive integers, and h and w are the height and width of the filter window. Thus, the image to be processed and the edge area of ​​the block image 2 are obtained.

[0054] Afterwards, the coordinates of multiple pixel points in the edge area belonging to the upper adjacent block image (i.e., block image 1) or the lower adjacent block image (i.e., block image 3) of block image 2 are calculated, and these pixel points are loaded from block image 1 and block image 3 stored in the memory to the vector operation unit cache of the corresponding hardware execution unit.

[0055] Afterwards, the pixel values ​​of the remaining pixels in the edge area that do not belong to the block image 1 and the block image 3 are obtained according to the preset filling rule, and loaded into the vector operation unit buffer of the corresponding hardware execution unit. For example, the preset filling rule can adopt border replication, border constant filling, border reflection, border reflection 101, etc., and the present disclosure does not make specific restrictions on this.

[0056] In the above embodiment, what is stored in the memory are the various block images, and the image to be processed after the expansion is obtained by calculating the mapping coordinates and stored in the cache area of ​​the vector operation unit. Therefore, there is no need to store each complete image to be processed, saving memory overhead and the time consumed by allocating memory.

[0057] For example, in some other embodiments, traditional edge expansion processing may also be used, that is, the complete data of the image to be processed obtained according to the edge expansion is stored in the memory. The specific edge expansion process of the image to be processed can refer to the relevant parts of the above embodiments; thereafter, when needed, it is directly loaded from the memory to the vector operation unit buffer or register, and the present disclosure does not impose specific restrictions on this.

[0058] In the present disclosure, the in-situ mapping method as described above or the method of directly loading the image to be processed from the memory is adopted, and the required data (the image to be processed) is pre-loaded into the vector operation unit cache before the median filtering operation is performed, so that the data can be directly obtained from the vector operation unit cache when the median filtering operation is performed, without the need to load the data from the memory. The vector operation unit cache is closer to the computing core than the memory, thereby speeding up the data reading process and reducing the memory access cost.

[0059] For example, in other embodiments, if the hardware execution unit does not provide a vector operation unit cache, after obtaining the area range of the image to be processed, the block image and the above-mentioned multiple pixel points can be read into the register in sequence in the conventional way of reading from the memory, and the pixel points after the boundary is filled according to the filling rule can be read into the register, which will not be described in detail here.

[0060] In step S20, M hardware execution units respectively perform median filtering operations on the M images to be processed to obtain median filtering results of the input images.

[0061] For example, each hardware execution unit performs a median filtering operation on one image to be processed. Thus, M hardware execution units can be used to perform median filtering operations on M images to be processed in parallel, making full use of multi-threaded hardware computing resources, performing median filtering operations in parallel, and improving processing efficiency.

[0062] For example, the image processing method can be applied to multi-threaded hardware computing resources, such as a graphics processor, a general-purpose graphics processor, etc. For example, a multi-threaded hardware computing resource provides multiple thread hardware that can be used to execute multiple instructions or computing tasks in parallel. For example, taking a general-purpose image processor as an example, a thread is a basic unit for executing a computing task, and thread hardware refers to a hardware structure that supports the execution of these threads. Figure 1 For example, the hardware execution unit may be a programmable multiprocessor, such as a streaming processor cluster, the hardware execution unit includes multiple computing units, each computing unit includes multiple execution units, each execution unit includes multiple thread hardware, and one thread hardware runs one thread, for example, one thread hardware performs a median filtering operation of n pixels at a time, where n is a positive integer. Here, single refers to a computing task that the thread hardware performs at a time, that is, after completing the median filtering operation of n pixels, reload the data to perform the next computing task.

[0063] For example, different thread hardware in the same hardware execution unit executes median filtering operations on different pixels at the same time.

[0064] For example, in a specific example, a general-purpose image processor includes M hardware execution units, each hardware execution unit includes 4 computing units, each computing unit includes 4 execution units, and each execution unit includes 32 thread hardware, so that one hardware execution unit can execute 512 threads simultaneously, and can perform median filtering operations on 512×n pixels simultaneously, thereby greatly improving the processing efficiency of median filtering.

[0065] Figure 4 A schematic diagram of the sequence of processing pixels in an image to be processed provided in one embodiment of the present disclosure.

[0066] like Figure 4 As shown in FIG. 1 , it is assumed that the hardware execution unit for processing the image to be processed includes T thread hardware, and each thread hardware can perform a median filtering operation on n pixels at a time. Therefore, within one execution cycle (performing one computing task), the median filtering results of T×n pixels can be obtained. For example, first Figure 4 Perform median filtering on the T×n pixels in the first rectangular box of the first row in the image. Then, in the next execution cycle, Figure 4 The median filtering operation is performed on T×n pixels in the second rectangular box in the first row of the rectangular box, and so on.

[0067] exist Figure 4 In the figure, T1 is a positive integer. Assume that n pixels include a rows and b columns, that is, a×b pixels arranged in an array in the row and column directions. If the number of pixels in the first a rows of pixels in the image to be processed that have not undergone median filtering is less than T×b, then continue to select the first (T-T1)×b pixels from the a+1th row to the 2×ath row for median filtering.

[0068] For example, when obtaining the median filtering result of the input image, the median filtering results obtained by the median filtering operations performed by the L thread hardware are spliced ​​according to the positional relationship of the pixel points to obtain the filtering result of the median filtering operation performed by each hardware execution unit on the input image to be processed; the filtering results generated by each of the M hardware execution units are spliced ​​according to the relative positional relationship of the M images to be processed in the row direction to obtain the median filtering result of the input image.

[0069] The specific process of the thread hardware performing the median filtering operation on n pixels at a time is described in detail below with reference to the accompanying drawings.

[0070] Figure 5 A schematic flow chart of a median filtering operation provided in at least one embodiment of the present disclosure is provided. For example, the median filtering operation is used to obtain median filtering results of n pixels.

[0071] like Figure 5As shown, the median filtering operation includes steps S201-S203.

[0072] For example, in step S201, a filtering area corresponding to n pixels is determined.

[0073] For example, in step S202, the pixels in the filter area are sorted in the row direction, the column direction and the diagonal direction, and the pixels in the sorted filter area that cannot be the median are removed.

[0074] For example, in step S203, the median filtering result of each pixel point among the n pixels is determined based on the remaining pixels points.

[0075] For example, in some embodiments, if n=1, step S201 may include: based on the filtering window of the median filtering operation, defining a filtering area centered on the one pixel in the image to be processed. That is, when n=1, the filtering area is an area defined by the filtering window in the image to be processed by overlapping the center of the filtering window with the center of the pixel.

[0076] For example, step S202 may include: sorting the pixel points in the filtering area in the row direction and the column direction to obtain the first filtering area, wherein, in the first filtering area, the pixel values ​​of the pixel points in each row are in increasing order in the row direction, and the pixel values ​​of the pixel points in each column are in increasing order in the column direction; removing the pixel points in the first filtering area whose coordinates do not meet the first median condition to obtain the second filtering area; sorting the second filtering area in the diagonal direction to obtain the third filtering area, wherein, in the third filtering area, the pixel values ​​of the pixel points on the diagonal line are in increasing order in the diagonal direction; removing the pixel points in the third filtering area that do not meet the second median condition.

[0077] For example, the pixel points whose coordinates satisfy the following relationship are determined not to satisfy the first median condition: x×y-1≥(c×d) / 2 (c-x+1)×(d-y+1)-1≥(c×d) / 2 Wherein, x and y represent the coordinates of the pixel point, c and d represent the size of the filter area, and x, y, c, and d are positive integers.

[0078] For example, the pixel points that satisfy the following relationship are determined not to satisfy the second median condition: Plt>(R+1) / 2 or Prt>(R+1) / 2 Among them, R represents the number of pixel points in the second filtering area, Plt represents the number of pixel points in the third filtering area whose pixel values ​​are greater than the pixel value of the pixel point and are located on the first side of the pixel point, Prt represents the number of pixel points in the third filtering area whose pixel values ​​are less than the pixel value of the pixel point and are located on the second side of the pixel point, and the pixel values ​​of the pixel points increase along the direction from the first side to the second side.

[0079] Figure 6 A process diagram of step S202 provided in an embodiment of the present disclosure.

[0080] like Figure 6 As shown, the size of the filter area is 7×7, which is determined by a 7×7 filter window.

[0081] like Figure 6 As shown, the filter area is first sorted in the row direction so that the pixel values ​​of each row of pixels are in increasing order in the row direction. For example, Figure 6 As shown, in the filtering area after the row direction is sorted, the pixel points in each row increase in sequence from left to right.

[0082] Afterwards, the filter area is sorted in the column direction so that the pixel values ​​of each column are in increasing order in the column direction, for example, Figure 6 As shown, in the first filtering area after the column direction is sorted, the pixel points in each column increase in sequence from top to bottom.

[0083] certainly, Figure 6 The figure is an example, and the column direction can be sorted first, and then the row direction can be sorted. The sorting order is not specifically limited. Figure 6 The upper left pixel point (149) of the filter area is taken as the origin, starting from the upper left pixel point, from left to right as the row direction, and from top to bottom as the column direction. When the origin position is modified (for example, the lower left pixel point (75) is taken as the coordinate origin), the sorting process and the subsequent median condition judgment process can also be adjusted accordingly, which will not be repeated here.

[0084] Afterwards, the pixel points whose coordinates do not satisfy the first median condition in the first filtering area are removed, and the pixel points that are most likely to be median points are retained.

[0085] For example, taking the upper left corner pixel point (3) in the first filtering area as the coordinate origin, the pixel points whose coordinates satisfy the following relationship are determined not to satisfy the first median condition: x×y-1≥(c×d) / 2 (c-x+1)×(d-y+1)-1≥(c×d) / 2 by Figure 6Taking the first filtering area shown as an example, at this time c = 7, d = 7, (c×d) / 2 = 25, for example, when y = 1, it is necessary to simultaneously satisfy x-1<25 and (7-x+1)×(7-1+1)-1<25, that is, x>4, for example, when y = 3, it is necessary to satisfy 3x-1<25 and (7-x+1)×5-1<25, that is, x>=3, and so on.

[0086] Therefore, remove Figure 6 The pixels of the dark background in the second filtering area are the remaining 29 pixels in the second filtering area that may be the median pixels, that is, Figure 6 The pixels of the white background in the second filtering area.

[0087] Afterwards, the second filter area is sorted in the diagonal direction so that the pixel values ​​of the pixels in the second filter area are in increasing order in the diagonal direction, for example, Figure 6 As shown, in the third filtering area after the diagonal direction is sorted, the pixel points on the diagonal line increase in sequence from the upper left to the upper right.

[0088] Of course, it should be noted here that the first median condition judgment process can also be executed after sorting in the diagonal direction, that is, after sorting the filter area in the row direction, column direction and diagonal direction, the pixel points in the sorted filter area whose coordinates do not meet the first median condition are removed.

[0089] Afterwards, the pixel points in the third filtering area that do not meet the second median condition are removed, and the pixel points that are most likely to be median points are retained.

[0090] For example, the pixel points that satisfy the following relationship are determined not to satisfy the second median condition: Plt>(R+1) / 2 or Prt>(R+1) / 2 For example, for Figure 6 In the example shown, R=29. For any pixel point P(x,y) in the third filtering area to become a median point, it must not fall into the smallest 29 / 2 number, nor the largest 29 / 2 number. After the diagonal sorting, the rows, columns, and diagonals are kept in order at the same time, that is, for the current pixel point P, the number of pixels that must be greater than or equal to P is counted in three parts: 1. The pixel values ​​of the pixels on the diagonal line to the right of point P must be greater than or equal to the pixel value of P. The total number of pixels that meet this condition is recorded as rt0.

[0091] 2. For the pixel points on the diagonal line to the right of point P, the pixel values ​​of the pixels below it in the column direction must be greater than or equal to the pixel value of P. The total number of pixels that meet this condition is recorded as rt1.

[0092] 3. The pixel values ​​of the pixels below point P in the column direction must be greater than or equal to the pixel value of P. The total number of pixels that meet this condition is recorded as rt2.

[0093] For pixel point P, the number of pixel points whose pixel values ​​are greater than or equal to point P is counted among the pixel points retained in the third filtering area, and is recorded as Prt, where Prt = rt0 + rt1 + rt2; Similarly, the number of pixel points whose pixel values ​​are less than or equal to the pixel value of point P in the retained data is counted and recorded as Plt.

[0094] The principle is similar to counting the number of points greater than or equal to P, but the comparison direction is just the opposite. It is also divided into three parts: 1. The pixel values ​​of the pixels on the diagonal line to the left of point P must be less than or equal to the pixel value of P. The total number of pixels that meet this condition is recorded as lt0.

[0095] 2. For the pixel points on the diagonal line located to the left of point P, the pixel values ​​of the pixels above it in the column direction must also be less than or equal to the pixel value of P. The total number of pixels that meet this condition is recorded as lt1.

[0096] 3. The pixel values ​​of the pixels above point P in the column direction must be less than or equal to the pixel value of P. The total number of pixels that meet this condition is recorded as lt2.

[0097] For a pixel point P, the number of pixel points whose pixel value is less than or equal to the pixel value of the point is counted in the retained data, which is recorded as Plt = lt0 + lt1 + lt2.

[0098] Refer to the above process, for Figure 6 In the example shown, 11 points are retained in the third filtering area, that is, Figure 6 White background pixels in the third filtering area.

[0099] Of course, the above median condition is an example, and those skilled in the art may also adopt other algorithms or sorting methods to obtain the median condition to determine the pixel points that cannot be the median point, and the present disclosure does not impose any specific limitation on this.

[0100] Thereafter, in step S203, a median filtering result of each pixel point among the n pixels points is determined based on the remaining pixels points.

[0101] For example, in some embodiments, if n=1, step S203 may include: sorting the remaining pixel points, and determining the sorted median point as a median filtering result of a pixel point.

[0102] Since the number of remaining pixels is limited and has been greatly reduced compared to the number of pixels in the filtering area, the median point can be determined through a limited number of comparison operations, such as obtaining the median filtering result of the pixel point through bubble sort or merge sort.

[0103] For example, in some other embodiments, if n is greater than 1, for example, the n pixels include a rows and b columns of pixels arranged in an array, that is, a×b pixels arranged in an array in the row and column directions, and at least one of a and b is greater than 1.

[0104] At this time, in some embodiments, n pixel points may be calculated independently of each other, for example, referring to the median filtering process of n=1, each pixel point is regarded as an independent pixel point for median filtering operation.

[0105] In other embodiments, the overlapping parts of the filter window area of ​​n pixels may be reused, for example, the n pixels may reuse the sorting results of the overlapping area to reduce the amount of calculation in the process of determining the median point.

[0106] For example, if n is greater than 1, step S201 may include: determining the filter window areas corresponding to n pixel points, wherein the filter window area corresponding to each pixel point includes an area centered on the pixel point and defined by the filter window of the median filtering operation; according to the filter window areas corresponding to the n pixel points, determining the overlapping part of the n filter window areas as the filter area, wherein the size of the filter area is c×d, the size of the filter window is h×w, c, d, h, and w are positive integers, c is less than h, and d is less than w.

[0107] For example, the filter window area corresponding to the pixel point is an area defined by the filter window in the image to be processed where the center of the filter window overlaps with the center of the pixel point.

[0108] Figure 7 A schematic diagram of a filtering area provided in at least one embodiment of the present disclosure. For example, Figure 7 Each small box in represents a pixel. Figure 7 The schematic diagram of the filtering area when a thread hardware performs a median filtering operation on 4 pixels at a time is shown. The 4 pixels are Figure 7 Pixel p1, pixel p2, pixel p3 and pixel p4 in the black rectangular box.

[0109] For example, assuming the filter window size is 7×7, Figure 7 As shown, 8×8 pixels in the image to be processed are shown, which are used to perform median filtering operations on pixel p1, pixel p2, pixel p3 and pixel p4.

[0110] For example, the filter window area of ​​pixel p1 is Figure 7 As shown in the dotted box in FIG. , it is a 7×7 area centered at pixel point p1. The filter window areas of pixel point p2, pixel point p3, and pixel point p4 are similar to the filter window area of ​​pixel point p1, and are not shown again here.

[0111] For example, the filter area corresponding to 4 pixels is the overlapping part of the 4 filter window areas, that is, Figure 7 There are 6×6 white rectangular boxes in the center, with a total of 6×6 pixels.

[0112] Figure 7 The figure shows an example, when the position and number of n pixels are different, the filter area also changes accordingly. For example, if the n pixels include 1×2 pixels, for example, two adjacent pixels located in the same column, the filter area corresponding to the two pixels includes 7×6 pixels.

[0113] Afterwards, step S202 is executed to sort the pixels in the filter area corresponding to the n pixels in the row direction, column direction and diagonal direction, and remove the pixels in the sorted filter area that cannot be the median. For the specific process, please refer to the relevant description in the above step S202, and the repeated parts will not be repeated.

[0114] Thereafter, step S203 is executed to determine the median filtering result of each pixel in the n pixels based on the remaining pixels.

[0115] For example, in this embodiment, step S203 may include: for any pixel point among the n pixel points: performing at least one first operation on the pixel points that do not belong to the filter area in the filter window area corresponding to any pixel point and the remaining pixel points, and obtaining the median point in the filter window area corresponding to any pixel point as the median filtering result of any pixel point, wherein in each first operation, the first queue and the second queue are obtained, and at least one merge sort operation is performed on the first queue and the second queue to obtain an output queue, wherein the number of elements in the output queue is r+1, r is the number of pixels in the filter window area corresponding to any pixel point that have not undergone the first operation, and the output queue is r+1 elements whose values ​​are in the middle of the first queue and the second queue.

[0116] For example, the first queue is the remaining pixel points or the output queue obtained from the last first operation, and the second queue is the c pixel points that have not undergone the first operation and are adjacent to the filter area in the row direction or the d pixel points that are adjacent to the filter area in the column direction. The c pixel points belong to the same column, and the d pixel points belong to the same row.

[0117] In at least one embodiment of the present disclosure, the process of finding the median point is based on the following theorem: For a sequence of length n, take out r numbers (nr>r), and then keep the middle r+1 numbers from the remaining nr numbers. Then the median of the new sequence composed of these r+1 numbers and the r numbers taken out is equal to the median of the original sequence.

[0118] Assume that in step S202, after removing pixels that cannot be median points, the number of remaining pixels in the third filtering area is P, where P is a positive integer. Based on the theorem, P should be greater than h×wc×d.

[0119] Afterwards, in step S203, the median filtering result of each pixel in the n pixels is calculated in sequence.

[0120] For example, for pixel 1 among n pixels, the P pixels are used as the first queue, and the c pixels in the filter window area of ​​pixel 1 that are adjacent to the filter area in the row direction are used as the second queue. The first queue and the second queue are merged and sorted at least once to obtain an output queue. The output queue is the r+1 elements with pixel values ​​in the middle of the P+c pixels in the first queue and the second queue, where r=w.

[0121] For example, the merge sort operation includes: obtaining two input pixel queues, where each input pixel queue is ordered; merge sorting the two input pixel queues to remove the smallest t elements and the largest t elements in the two input pixel queues, where t is a positive integer greater than 1; and using the remaining elements in the two input pixel queues as a merged pixel queue, where the merged pixel queue remains locally ordered.

[0122] In the present disclosure, locally ordered means that the queue includes one or more sub-queues, and each sub-queue is ordered (for example, from small to large or from large to small). The input pixel queue should be ordered, and if it is locally ordered, it can be converted into an ordered queue through a limited number of comparisons. For example, the input pixel queue can be an ordered sub-queue in the merged pixel queue.

[0123] For example, the first queue is sorted to make it ordered, thereby obtaining a first input pixel queue, and the second queue is used as the second input pixel queue. The first input pixel queue and the second input pixel queue are merged and sorted, and the two input pixel queues are merged into one queue, and the largest t elements and the smallest t elements in the queue are removed to obtain an output queue. For example, t is half of the total number of elements in the shorter queue of the first input pixel queue and the second input pixel queue, and if the total number of elements is an odd number, it is rounded down.

[0124] If the number of elements in the output queue is greater than r+1, the above merge sorting process is continued. For example, a locally ordered sub-queue in the output queue is used as the first input pixel queue, and another sub-queue is used as the second input pixel queue for merge sorting. The specific process is not repeated here.

[0125] Through the above merge sorting process, we can obtain the r+1 pixel points whose pixel values ​​are located in the middle of the P+c pixel points in the first queue and the second queue, and the r+1 pixel points are used as the r+1 elements in the output queue.

[0126] Afterwards, the r+1 elements are used as the first queue, and the d pixel points in the filter window area of ​​pixel point 1 that are adjacent to the filter area in the column direction are used as the second queue. The first queue and the second queue are merged and sorted at least once to obtain an output queue. The output queue is the r+1 elements in the middle of the w+1+d pixel points in the first queue and the second queue, where r=1.

[0127] The specific merge sort process can refer to the above steps and will not be repeated here.

[0128] For example, when r=1, the two elements in the output queue obtained by the current first operation are sorted with one element in the filtering window area corresponding to pixel 1 that has not undergone the first operation. For example, the element is the element at the diagonal vertex, and the median point of the sorted three elements is used as the median filtering result of any pixel.

[0129] Of course, during the second first operation, the remaining pixel points that have not been subjected to the first operation may be used as the second queue to directly output the median filtering result, which will not be elaborated here.

[0130] Thus, the median filtering result of pixel 1 is obtained through the above process. Similarly, the median filtering results of the remaining pixels can also be obtained by a similar process. The median filtering process of each pixel reuses the result of step S202, and only the pixels in the non-filtered area are considered in the subsequent sorting, thereby reducing the amount of sorting calculations, reducing the complexity of sorting, fully reusing the existing results, and improving the efficiency of median filtering, especially for larger filter windows (such as 5×5 or 7×7, etc.), the performance is significantly improved.

[0131] Below Figure 7 Taking the filtering area shown as an example, the execution process of step S203 is described in detail.

[0132] As mentioned above, for Figure 7The 4 pixels shown, excluding the 6×6=36 pixels in the filtering area corresponding to the 4 pixels, there are 7×7 -36=13 points left, so it is only necessary to find 13+1=14 pixels with pixel values ​​in the middle in the 6×6 area.

[0133] Referring to the aforementioned step S202, pixel points that are unlikely to be median points are removed through step S202, and finally 14 pixel points that are likely to be median points are obtained.

[0134] Afterwards, referring to step S203 , a median filtering result of each of the four pixels is determined based on the fourteen pixels.

[0135] Take pixel p1 as an example, refer to Figure 7 First, the 14 pixels and the 6 pixels located in the filter window area of ​​pixel p1 that do not belong to the filter area but are located on the left side of the filter area, that is, Figure 7 The first operation is performed on the six pixels of the checkerboard background in the first column on the left.

[0136] Specifically, for the filtering window area of ​​pixel point p1, there are 7 pixel points in the first row that have not been subjected to the first operation. Therefore, according to the above theorem, the middle 8 pixel points of the 14+6=20 points need to be retained.

[0137] For example, the 14 pixel points are used as the first queue, and the 6 pixel points are used as the second queue, and at least one merge sorting operation is performed to obtain an output queue, and the output queue is the 8 pixel points in the middle of the 20 points.

[0138] Specifically, assuming that the 6 pixels are represented as v0, v1, ..., v5, and the 14 pixels are represented as u0, u1, ..., u13, in the first merge sort operation, v0, v1, ..., v5 are used as the first input pixel queue, v0, v1, ..., v5 are increased in sequence, u0, u1, ..., u13 are used as the second input pixel queue, and the 14 pixels are sorted so that they are in order, and u0, u1, ..., u13 are increased in sequence.

[0139] First, compare v0, v1 and v2 with u0, u1 and u2, keep the larger 3 values ​​in u0, u1 and u2, and keep the smaller 3 elements in v0, v1 and v2 and remove them.

[0140] Similarly, compare v3, v4 and v5 with u11, u12 and u13, keep the smaller 3 values ​​in u11, u12 and u13, keep the larger 3 elements in v3, v4 and v5 and remove them, thus obtaining the merged pixel queue u0, u1, ..., u13, in which the largest 3 elements and the smallest 3 elements in 20 elements are removed. So far, a merge sort is completed.

[0141] The current merged pixel queues u0, u1, ..., u13 are locally ordered, the size relationship between u2 and u3 is unknown, and the size relationship between u10 and u11 is unknown. Therefore, u0, u1, u2 can be used as the first input pixel queue, and u3, u4 and u5 can be used as the second input pixel queue, and merged and sorted as described above, so that the three larger elements of the six elements are retained in u3, u4 and u5.

[0142] Similarly, u8, u9, and u10 are used as the first input pixel queue, and u11, u12, and u3 are used as the second input pixel queue, and merge sorting is performed as described above, so that the three smaller elements among the six elements are retained in u8, u9, and u10.

[0143] Thus, the output queues u3, u4, u5, u6, u7, u8, u9, and u10 are obtained.

[0144] Afterwards, the 8 elements in the above output queue are used as the first queue, and the 6 pixels located in the filter window area of ​​the pixel point p1 that do not belong to the filter area but are adjacent to the filter area above, that is, Figure 7 The first operation is performed on the six pixels of the diamond pattern background in the first row.

[0145] Specifically, for the filtering window area of ​​pixel point p1, there is still one pixel point with a diagonal background in the upper left corner that has not undergone the first operation. Therefore, according to the above theorem, the two middle pixels of the 8+6=14 points need to be retained in the first operation.

[0146] The specific process is similar to the merge sort process in the first operation above, and the repeated parts will not be repeated here.

[0147] Afterwards, the two pixel points and the last remaining vertex pixel point at the upper left corner are sorted to obtain the median point, thereby obtaining the median filtering result of the pixel point p1.

[0148] For the 14 pixel points obtained in the pixel point multiplexing step S202, the pixel point p2, the pixel point p3 and the pixel point p4 are respectively obtained by referring to the above process, which will not be repeated here.

[0149] In the image processing method provided in at least one embodiment of the present disclosure, a low-complexity median filtering operation that can be calculated in parallel in multi-threaded hardware computing resources is provided, and the performance is significantly improved on the median operator of a larger filtering window (5×5 / 7×7, etc.); and, the hardware characteristics are fully utilized, and the high-speed buffer provided by each hardware execution unit is used to further increase the parallel computing capability and reduce the memory access overhead.

[0150] At least one embodiment of the present disclosure further provides an image processing device. Figure 8 A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure.

[0151] like Figure 8 As shown, the image processing apparatus 100 includes an acquisition module 101 and M hardware execution units 102. Of course, the image processing apparatus 100 may also include more hardware execution units, and the present disclosure does not impose any specific limitation on this.

[0152] For example, the acquisition module 101 is configured to acquire M images to be processed. The M images to be processed are obtained by expanding the edges of M block images respectively, and the M block images are obtained by dividing the input image into M blocks in the column direction, where M is a positive integer.

[0153] For example, the M hardware execution units 102 are configured to perform median filtering operations on the M images to be processed respectively to obtain median filtering results of the input images.

[0154] For example, each hardware execution unit includes L thread hardware, and the L thread hardware is configured to perform median filtering operations on the image to be processed input to the hardware execution unit in parallel. Each thread hardware is configured to perform median filtering operations on n pixels at a time, where n is a positive integer.

[0155] For example, the image processing device 100 is a graphics processor or a general-purpose graphics processor. Figure 1 As described above, the hardware execution unit may be a programmable multiprocessor, such as a streaming processor cluster.

[0156] For example, the hardware execution unit includes multiple computing units, each computing unit includes multiple execution units, and each execution unit includes multiple thread hardware. Figure 1 The relevant description will not be repeated here.

[0157] Of course, the present disclosure is not limited thereto, and the graphics processing device 100 may also be other processing devices capable of providing multi-thread hardware.

[0158] For example, when the thread hardware performs a median filtering operation on n pixels, it includes the following operations: determining a filtering area corresponding to the n pixels; sorting the pixels in the filtering area in the row direction, column direction, and diagonal direction, and removing pixels in the sorted filtering area that cannot be the median; and determining a median filtering result for each of the n pixels based on the remaining pixels.

[0159] For example, the image processing device 100 further includes a memory and a vector operation unit buffer ( Figure 8 For example, the input image is stored in the memory of the image processing device 100, and each hardware execution unit has a vector operation unit cache area, which is used to load the image to be processed by the hardware execution unit from the memory to the vector operation unit cache area before performing the median filtering operation, so that when each hardware execution unit performs the median filtering operation, the corresponding image to be processed is directly obtained from the vector operation unit cache area of ​​the hardware execution unit without reading from the memory, thereby reducing the memory access overhead.

[0160] It should be noted that the acquisition module 101 can be used to implement Figure 2 Step S10 shown; the hardware execution unit 102 can be used to implement Figure 2 Step S20 shown. Therefore, for the specific description of the functions that can be implemented by the acquisition module 101, reference can be made to the relevant description of step S10 in the embodiment of the above-mentioned image processing method, and for the specific description of the functions that can be implemented by the hardware execution unit 102, reference can be made to the relevant description of step S20 in the embodiment of the above-mentioned image processing method, and the repeated parts will not be repeated. In addition, the image processing device 100 can achieve technical effects similar to those of the above-mentioned image processing method, which will not be repeated here.

[0161] It should be noted that in at least one embodiment of the present disclosure, the image processing device 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or in other applicable ways.

[0162] For example, the image processing device 100 may be implemented in hardware, software, or a combination of hardware and software, and the present disclosure does not impose any specific limitation on this.

[0163] Fig. 9 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Fig. 9As shown, the storage medium 200 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 210 may be non-transitory stored on the storage medium 200. For example, when the computer-readable instructions 210 are executed by a processor, one or more steps in the image processing method described above may be executed.

[0164] For example, the storage medium 200 may be applied in an electronic device 300 . For example, the storage medium 200 may include a storage device 308 in the electronic device 300 .

[0165] For example, the storage device may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the processor may run the computer-readable instructions to implement various functions of the processor. Various applications and various data may also be stored in the storage medium.

[0166] For example, the storage medium may include a memory card of a smart phone, a cache component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0167] Fig.10 The following is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Fig.10 As shown, the electronic device 300 is suitable for implementing the image processing method provided by the embodiment of the present disclosure. It should be noted that Fig.10 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0168] like Fig.10 As shown, the electronic device 300 may include a processing device (eg, a graphics processor, a central processing unit, etc.) 301, which may perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to implement various functions.

[0169] For example, when the computer readable instructions are executed by the processing device 301, one or more steps in the image processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the image processing method, reference may be made to the relevant description in the above-mentioned embodiments of the image processing method.

[0170] For example, the central processing unit in an electronic device can be configured to act as a control end to schedule and control the graphics processor when performing image processing. For example, the central processing unit is responsible for processing complex logical operations and overall control, while the graphics processor focuses on graphics processing and data parallel computing.

[0171] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory (cache), etc. For example, computer-readable instructions may be loaded from storage device 308 into random access memory (RAM) 303 to run computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by applications, etc., may also be stored in the computer-readable storage medium.

[0172] For example, the processing device 301 , the ROM 302 , and the RAM 303 are connected to one another via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0173] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig.10An electronic device 300 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown, and the electronic device 300 may alternatively implement or have more or fewer devices. For example, the processing device 301 can control other components in the electronic device 300 to perform the desired functions. The processing device 301 can be a device with data processing capabilities and / or program execution capabilities such as a central processing unit (CPU), a tensor processing unit (TPU) or a graphics processing unit GPU. The central processing unit (CPU) can be an X86, ARM, RISC-V architecture, etc. The GPU can be directly integrated into the SOC, directly integrated into the motherboard, or built into the north bridge chip of the motherboard.

[0174] For example, the electronic device 300 may be arranged on a server side (or a cloud side).

[0175] For example, in some embodiments, the electronic device 300 may be a mobile phone, a tablet computer, an electronic paper, a television, a monitor, a laptop computer, a digital photo frame, a navigator, a wearable electronic device, a smart home device, etc.

[0176] For example, the electronic device 300 may include a display panel, and the display panel may be used to segment an image, etc. For example, the display panel may be a rectangular panel, a circular panel, an elliptical panel, or a polygonal panel, etc. In addition, the display panel may be not only a flat panel, but also a curved panel, or even a spherical panel.

[0177] For example, the electronic device 300 may have a touch function, that is, the electronic device 300 may be a touch device.

[0178] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0179] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0180] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0181] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0182] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0183] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0184] There are a few points to note about this disclosure: (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0185] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0186] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and the protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. An image processing method, comprising: Acquire M images to be processed, wherein the M images to be processed are obtained by respectively expanding the edges of M block images, and the M block images are obtained by dividing the input image into M blocks in a column direction, and M is a positive integer; M hardware execution units respectively perform median filtering operations on the M images to be processed to obtain median filtering results of the input images, wherein each hardware execution unit includes L thread hardware, and the L thread hardware performs the median filtering operation on the image to be processed input to the hardware execution unit in parallel, and each thread hardware performs the median filtering operation of n pixels at a time, where n and L are positive integers. The median filtering operation of the n pixels is performed by the thread hardware, including: Determine the filtering area corresponding to the n pixels; Sorting the pixels in the filtering area in the row direction, the column direction and the diagonal direction, and removing the pixels in the filtering area that cannot be the median after the sorting; A median filtering result of each pixel point in the n pixels is determined based on the remaining pixels points.

2. The image processing method according to claim 1, wherein: Sorting the pixels in the filtering area in the row direction, the column direction and the diagonal direction, and removing the pixels in the filtering area that cannot be the median after the sorting, including: Sorting the pixel points in the filtering area in the row direction and the column direction to obtain a first filtering area, wherein in the first filtering area, the pixel values ​​of the pixel points in each row are in increasing order in the row direction, and the pixel values ​​of the pixel points in each column are in increasing order in the column direction; Remove the pixel points whose coordinates do not satisfy the first median condition in the first filtering area to obtain a second filtering area; Sorting the second filtering area in the diagonal direction to obtain a third filtering area, wherein in the third filtering area, pixel values ​​of pixels on the diagonal are in increasing order in the diagonal direction; Pixels in the third filtering area that do not satisfy the second median condition are removed.

3. The image processing method according to claim 2, wherein: The pixel points whose coordinates satisfy the following relationship are determined not to satisfy the first median condition: x×y-1≥(c×d) / 2, (c-x+1)×(d-y+1)-1≥(c×d) / 2; Wherein, x and y represent the coordinates of the pixel point, c and d represent the size of the filtering area, and x, y, c, and d are positive integers.

4. The image processing method according to claim 2, wherein: Pixel points that satisfy the following relationship are determined not to satisfy the second median condition: Plt > (R+1) / 2 or Prt > (R+1) / 2; Among them, R represents the number of pixel points in the second filtering area, Plt represents the number of pixel points in the third filtering area whose pixel values ​​are greater than or equal to the pixel value of the pixel point and are located on the first side of the pixel point, Prt represents the number of pixel points in the third filtering area whose pixel values ​​are less than or equal to the pixel value of the pixel point and are located on the second side of the pixel point, and the pixel values ​​of each pixel point increase along the direction from the first side to the second side.

5. The image processing method according to claim 1, wherein: In response to n being greater than 1, the n pixels include a rows and b columns of pixels arranged in an array in the image to be processed, a and b are positive integers, and at least one of a and b is greater than 1, Determining the filtering area corresponding to the n pixel points includes: Determine the filter window areas corresponding to the n pixel points respectively, wherein the filter window area corresponding to each pixel point includes an area centered on the pixel point and defined by the filter window of the median filter operation; According to the filter window areas corresponding to the n pixel points, the overlapping parts of the filter window areas corresponding to the n pixel points are determined as the filter areas corresponding to the n pixel points, wherein the size of the filter area corresponding to the n pixel points is c×d, and the size of the filter window of the median filtering operation is h×w, c, d, h, and w are positive integers, c is less than h, and d is less than w.

6. The image processing method according to claim 5, wherein: Determining a median filtering result of each pixel in the n pixels based on the remaining pixels, comprising: For any pixel among the n pixels: Performing at least one first operation on the pixel points that do not belong to the filtering area in the filtering window area corresponding to any one pixel point and the remaining pixel points to obtain a median point in the filtering window area corresponding to any one pixel point as a median filtering result of the any one pixel point, In which, in each first operation, the first queue and the second queue are obtained, and at least one merge sort operation is performed on the first queue and the second queue to obtain an output queue, wherein the number of elements of the output queue is r+1, r is the number of pixels in the filter window area corresponding to any pixel point that has not undergone the first operation, and the output queue is the r+1 elements with values ​​in the middle between the first queue and the second queue.

7. The image processing method according to claim 6, wherein: The first queue is the remaining pixel points or the output queue obtained by the last first operation, and the second queue is the c pixel points that have not undergone the first operation and are adjacent to the filter area in the row direction or the d pixel points adjacent to the column direction, the c pixel points belong to the same column, and the d pixel points belong to the same row.

8. The image processing method according to claim 6, wherein: When r=1, the two elements in the output queue obtained by the current first operation and the one element in the filtering window area corresponding to any pixel point that has not undergone the first operation are sorted, and the median point of the sorted three elements is used as the median filtering result of any pixel point.

9. The image processing method according to claim 6, wherein: The merge sort operation includes: Obtain two input pixel queues, wherein each input pixel queue is ordered; Merge and sort the two input pixel queues to remove the smallest t elements and the largest t elements in the two input pixel queues, where t is a positive integer greater than 1; The remaining elements in the two input pixel queues are used as a merged pixel queue, wherein the merged pixel queue is at least partially ordered.

10. The image processing method according to claim 1, wherein: In response to n being equal to 1, determining the filtering area corresponding to the n pixels includes: Based on the filtering window of the median filtering operation, an area centered on the one pixel point is defined in the image to be processed as the filtering area.

11. The image processing method according to claim 10, wherein: Determining a median filtering result of each pixel in the n pixels based on the remaining pixels, comprising: The remaining pixel points are sorted, and the sorted median point is determined as the median filtering result of the one pixel point.

12. The image processing method according to any one of claims 1 to 11, wherein: Different thread hardware in the same hardware execution unit executes the median filtering operation on different pixels in the same execution cycle.

13. The image processing method according to any one of claims 1 to 11, wherein: Each hardware execution unit includes a vector operation unit cache area, and the vector operation unit cache area is used for data caching of each thread hardware in the hardware execution unit. Get M images to be processed, including: Dividing the input image into the M block images in the column direction; Pre-fetching the M block images stored in the memory into the vector operation unit buffers of the M hardware execution units respectively; The pixel points of the edge areas obtained by respectively performing the edge expansion on the M block images are loaded into the vector operation unit cache of the corresponding hardware execution unit, so that when each hardware execution unit performs the median filtering operation, the corresponding image to be processed is directly obtained from the vector operation unit cache of the hardware execution unit.

14. The image processing method according to claim 13, wherein: The step of loading pixel points of edge areas obtained by respectively performing the edge expansion on the M block images into a vector operation unit buffer of a corresponding hardware execution unit includes: For each tile image: The upper side and the lower side of the block image are respectively extended by p rows of pixels in a direction away from the block image, and the left side and the right side of the block image are respectively extended by q columns of pixels in a direction away from the block image, so as to obtain an image to be processed and an edge region of the block image, wherein the edge region is a region of the image to be processed that does not belong to the block image, wherein p and q are positive integers; Determine coordinates of a plurality of pixel points in the edge region belonging to an upper side adjacent block image or a lower side adjacent block image of the block image, and load the plurality of pixel points from the upper side adjacent block image or the lower side adjacent block image stored in the memory to a vector operation unit buffer of the corresponding hardware execution unit; The pixel values ​​of the remaining pixels in the edge area that do not belong to the upper side adjacent block image and the lower side adjacent block image are obtained according to a preset filling rule, and the remaining pixels are loaded into the vector operation unit buffer of the corresponding hardware execution unit.

15. The image processing method according to any one of claims 1 to 11, wherein: The median filtering results obtained by the median filtering operation performed by the L thread hardware are spliced ​​according to the position relationship between the pixel points to obtain the filtering result of each hardware execution unit performing the median filtering operation on the input image to be processed; The filtering results generated by the M hardware execution units are respectively spliced ​​according to the relative position relationship of the M images to be processed in the column direction to obtain a median filtering result of the input image.

16. An image processing device, comprising: An acquisition module is configured to acquire M images to be processed, wherein the M images to be processed are obtained by respectively expanding the edges of M block images, and the M block images are obtained by dividing the input image into M blocks in a column direction, and M is a positive integer; M hardware execution units are configured to perform median filtering operations on the M images to be processed respectively to obtain median filtering results of the input images, Each hardware execution unit includes L thread hardwares, and the L thread hardwares are configured to perform the median filtering operation on the image to be processed input to the hardware execution unit in parallel, and each thread hardware is configured to perform the median filtering operation of n pixels at a time, where n and L are positive integers. Wherein, when the thread hardware performs the median filtering operation of the n pixels, the following operations are performed: Determine the filtering area corresponding to the n pixels; Sorting the pixels in the filtering area in the row direction, the column direction and the diagonal direction, and removing the pixels in the filtering area that cannot be the median after the sorting; A median filtering result of each pixel point in the n pixels is determined based on the remaining pixels points.

17. The image processing apparatus according to claim 16, wherein: The hardware execution unit includes multiple computing units, each computing unit includes multiple execution units, and each execution unit includes multiple thread hardware.

18. An electronic device, comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the image processing method according to any one of claims 1-15 is implemented.

19. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by a processor, the image processing method according to any one of claims 1 to 15 is implemented.

Images