Image guided filtering method and system

Abstract

This application discloses an image guided filtering method and system, comprising: reading an input image and a guided image from host memory into the global memory of a GPU; dividing the input image and the guided image into multiple input sub-images and multiple guided sub-images in the global memory according to the parallel data width, wherein the input sub-images and the multiple guided sub-images correspond one-to-one; allocating the guided filtering processing of each input sub-image to multiple work items of the GPU, wherein the process of guided filtering processing of each input sub-image and the corresponding guided sub-image corresponds to one work item; the multiple work items sequentially and in parallel calling a first kernel function and a second kernel function to obtain the mean filtering results of the scaling matrix and offset matrix of the multiple input sub-images, calculating the corresponding output sub-images based on the multiple input sub-images and the mean filtering results of the corresponding scaling matrix and offset matrix, and merging and saving the multiple output sub-images into the global memory to reduce resource consumption.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image guided filtering method and system. Background Technology

[0002] Image filtering is an important technique in image processing, possessing significant meaning and research value. Due to imperfections in imaging systems, transmission media, and recording equipment, digital images are often contaminated by various types of noise during their formation, transmission, and recording. Image filtering, which suppresses noise in the target image while preserving as much detail as possible, is an indispensable operation in image preprocessing. The quality of its processing directly affects the effectiveness and reliability of subsequent image processing and analysis.

[0003] Image filtering methods can be divided into two types: linear shift-invariant filtering and guided filtering. Linear shift-invariant filtering uses kernel weights independent of the input image content, such as Gaussian filtering, mean filtering, and Laplacian filtering. Guided filtering, also known as linear shift-varying filtering, utilizes information from the original image, called guide image information. The guide image may be the same as or different from the input image. Guided filtering is widely used in image denoising, image enhancement, image matching, and dehazing. Although guided filtering algorithms are simple and effective, they require calculating complex matrices and solving large systems of linear equations, resulting in significant computational time and space consumption.

[0004] Generally, guided image filtering is performed on PC-based systems with the CPU as the processing core. Although it significantly improves the image filtering effect and speed compared to other filtering methods, the CPU must concentrate on processing the guided filtering function, resulting in slow processing efficiency and poor real-time performance, especially for high-resolution images. Summary of the Invention

[0005] In view of the above problems, the purpose of this invention is to provide an image guided filtering method and system that can reduce resource consumption and improve resource utilization.

[0006] According to a first aspect of the present invention, an image guided filtering method is provided, comprising: reading an input image and a guided image from host memory into the global memory of a GPU, wherein the guided image is a preset image or the input image itself; dividing the input image and the guided image into multiple input sub-images and multiple guided sub-images in the global memory according to the parallel data width, wherein the input sub-images and the multiple guided sub-images correspond one-to-one; allocating the guided filtering processing of each input sub-image to multiple work items of the GPU, wherein the process of guided filtering processing of each input sub-image and the corresponding guided sub-image corresponds to one work item; the multiple work items calling a first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image respectively, and merging and saving the scaling matrix and offset matrix of the multiple input sub-images into shared memory; the multiple work items calling a second kernel function in parallel to obtain the mean filtering result of the scaling matrix and offset matrix of the multiple input sub-images, and calculating the corresponding output sub-image based on the mean filtering result of the multiple input sub-images and the corresponding scaling matrix and offset matrix, and merging and saving the multiple output sub-images into global memory.

[0007] Preferably, the parallel invocation of the first kernel function by multiple work items to obtain the scaling matrix and offset matrix of the input sub-image includes: obtaining the corresponding neighborhood image of each input sub-image according to the filtering width; each work item reading the data of the input sub-image and its neighborhood image from global memory using a memory-sharing method; shifting and summing the input sub-image and its neighborhood image according to the parallel data width and the filtering width to obtain the mean filtering result of the input sub-image; replacing the input sub-image with the guiding sub-image and obtaining the mean filtering result of the corresponding guiding sub-image; calculating the covariance matrix and variance matrix of the input sub-image and the guiding sub-image according to the mean filtering result of the input sub-image and the mean filtering result of the guiding sub-image; and calculating the scaling matrix and offset matrix of the input sub-image according to the covariance matrix of the input sub-image and the guiding sub-image and the variance matrix of the guiding sub-image.

[0008] Preferably, obtaining the corresponding neighborhood image of each input sub-image based on the filter width includes: performing edge-padding processing on the input image and the guide image respectively according to the filter radius; the edge-padding processing includes adding R rows of pixels on the top and bottom sides of the input image, and adding R columns of pixels on the left and right sides of the input image and the guide image, where R is the filter radius, and the pixel values ​​of the added pixels are obtained by folding with the boundary of the original input image and the original guide image as the axis of symmetry; taking R rows of pixels on the top and bottom and R columns of pixels on the left and right sides of the input sub-image as the corresponding neighborhood image of the input sub-image.

[0009] Preferably, the process of shifting and summing the input sub-image and its corresponding neighboring images according to the parallel data width and the filtering width to obtain the mean filtering result of the input sub-image includes: constructing a first kernel function, which includes at least a first kernel function and a second kernel function, wherein the first kernel function is used to calculate the sum of pixels in each row from column j1 to column j2 in the input sub-image and its corresponding neighboring images, and the second kernel function is used to calculate the sum of pixels in each column from row i1 to row i2 in the input sub-image and its corresponding neighboring images. j1 = (m-1)*k+1-R, m = 1, 2, ..., W / k, j2 = j1+w-1, i1 = -R, ..., H+R; i2 = i1+w-1, where k is the parallel data width, w is the filter width, w = 2R+1, R is the filter radius, and M is the work item number; each work item completes the horizontal filtering of the input sub-image by repeatedly calling the first kernel function and the vertical filtering of the input sub-image by repeatedly calling the second kernel function. Shared memory is used to save intermediate data in the loop.

[0010] Preferably, when performing horizontal filtering on the input sub-image, each row is shifted starting from the first pixel to obtain w sets of vector values, each set of vector values ​​including k pixel values. The w sets of vector values ​​are summed and averaged to obtain the horizontal filtering result. When performing vertical filtering on the input sub-image, an intermediate storage variable sum is set. The initial sum is the sum of the first w pixels in each column starting from the first pixel. Then, the sum is incremented each time by taking the difference between the (w+1)th pixel value and the first pixel value, where w is the filtering width. This process is repeated, and the horizontal filtering result is summed and averaged along the column direction to obtain the vertical filtering result as the mean filtering result.

[0011] Preferably, when performing horizontal filtering on the input sub-image, each row is shifted starting from the first pixel to obtain w sets of vector values, each set including k pixel values. The w sets of vector values ​​are summed to obtain the horizontal filtering result. When performing vertical filtering on the input sub-image, an intermediate storage variable `sum` is set. The initial `sum` is the sum of the first w pixels in each column starting from the first pixel. Then, it is incremented by the difference between the (w+1)th pixel value and the first pixel value, where `w` is the filtering width. This process is repeated to sum the horizontal filtering result along the column direction and obtain the vertical filtering result. The mean square value of the vertical filtering result is then calculated as the mean filtering result.

[0012] Preferably, each work item retrieves data from global memory according to work item number m. The starting column corresponding to the m-th work item is (m-1)*k+1, where m = 1, 2, ..., W / k, and W is the width of the input image and k is the width of the parallel data.

[0013] Preferably, the parallel invocation of the second kernel function by multiple work items to obtain the mean filtering results of the scaling matrix and offset matrix of multiple input sub-images includes: obtaining the scaling neighborhood matrix corresponding to the scaling matrix and the offset neighborhood matrix corresponding to the offset matrix of each input sub-image according to the filtering width; each work item reads the scaling matrix and its corresponding scaling neighborhood matrix and offset matrix and their corresponding offset neighborhood matrix of the input sub-image from shared memory using a memory-sharing method; and performing shift and summation on the scaling matrix and its corresponding scaling neighborhood matrix and offset matrix and their corresponding offset neighborhood matrix of the input sub-image according to the parallel data width and the filtering width to obtain the mean filtering results of the scaling matrix and the offset matrix of the input sub-image.

[0014] Preferably, the process of shifting and summing the scaling matrix and its corresponding scaling neighborhood matrix and offset matrix and their corresponding offset neighborhood matrix of the input sub-image according to the parallel data width and the filtering width to obtain the mean filtering result of the scaling matrix and the mean filtering result of the offset matrix of the input sub-image includes: constructing a second kernel function, wherein the first kernel function includes a first kernel function, a second kernel function, a third kernel function and a fourth kernel function, wherein the first kernel function is used to calculate the sum of pixels in each row from column j1 to column j2 in the scaling matrix and its corresponding scaling neighborhood matrix of the input sub-image, the second kernel function is used to calculate the sum of pixels in each column from row i1 to row i2 in the scaling matrix and its corresponding scaling neighborhood matrix of the input sub-image, and the first kernel function is used to calculate the sum of pixels in each row from row j1 to row i2 in the offset matrix and its corresponding offset neighborhood matrix of the input sub-image. The first kernel function calculates the sum of pixels from column 1 to column j2. The second kernel function is used to calculate the sum of pixels from row i1 to row i2 in each column of the offset matrix of the input sub-image and its corresponding offset neighborhood matrix, where j1 = 1, 2, ..., k; j2 = j1 + w - 1, i1 = 1, 2, ..., k; i2 = i1 + w - 1, where k is the parallel data width, w is the filter width, w = 2R + 1, and R is the filter radius. Each work item performs horizontal filtering of the scaling matrix of the input sub-image by repeatedly calling the first kernel function, vertical filtering of the scaling matrix of the input sub-image by repeatedly calling the second kernel function, horizontal filtering of the offset matrix of the input sub-image by repeatedly calling the third kernel function, and vertical filtering of the offset matrix of the input sub-image by repeatedly calling the fourth kernel function. Intermediate data is stored in shared memory during the loop.

[0015] According to another aspect of the present invention, an image guided filtering system is provided, comprising: an image acquisition module for reading an input image and a guided image from host memory into the global memory of a GPU, wherein the guided image is a preset image or the input image itself; an image partitioning module for partitioning the input image and the guided image into multiple input sub-images and multiple guided sub-images in the global memory according to the parallel data width, wherein the input sub-images and the multiple guided sub-images correspond one-to-one; and a task allocation module for allocating the guided filtering processing of each input sub-image to multiple work items of the GPU, wherein each input sub-image and its corresponding guided sub-image are processed... Each row-guided filtering process corresponds to one work item; the filtering parameter determination module is used to control multiple work items to call the first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image respectively, and to merge and save the scaling matrix and offset matrix of multiple input sub-images into shared memory; the output image determination module is used to control multiple work items to call the second kernel function in parallel to obtain the mean filtering result of the scaling matrix and offset matrix of multiple input sub-images, and to calculate the corresponding output sub-image based on the mean filtering result of the multiple input sub-images and the corresponding scaling matrix and offset matrix, and to merge and save the multiple output sub-images into global memory.

[0016] The image guided filtering method and system provided by this invention divides the input image and the guided image into multiple sub-images, and distributes the guided filtering processing of multiple sub-images to multiple work items for parallel processing, thereby improving computational efficiency. Each work item is responsible for the guided filtering processing of the corresponding sub-image, which can reduce resource consumption and improve resource utilization.

[0017] Furthermore, each work item uses a memory-sharing approach to retrieve data from global memory. Each work item obtains the address of the corresponding sub-image in global memory based on the work item number, and retrieves the data of the corresponding sub-image through the address, which can reduce the data copying overhead.

[0018] Furthermore, each work item can directly obtain data through addressing to complete edge patching, which can reduce data storage.

[0019] Furthermore, image guided filtering is implemented by calling the GPU kernel function only twice, and data is processed in parallel during each kernel calculation, which improves computational efficiency. In addition, while reducing the number of GPU kernel function calls, global memory and shared memory are fully utilized, thereby reducing data transfer between the CPU and GPU and reducing algorithm execution time. Attached Figure Description

[0020] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0021] Figure 1 This diagram shows a simplified block diagram of the hardware system used in the image guided filtering method provided in an embodiment of the present invention.

[0022] Figure 2 A flowchart illustrating the image guided filtering method provided in an embodiment of the present invention is shown;

[0023] Figure 3 This diagram illustrates a flow chart of step S240 in the image guided filtering method provided in an embodiment of the present invention.

[0024] Figure 4 This diagram illustrates the edge-padded input image provided in an embodiment of the present invention.

[0025] Figure 5 This diagram illustrates the horizontal and vertical filtering of the input sub-image provided in an embodiment of the present invention.

[0026] Figure 6 This diagram illustrates a step S250 in the image guided filtering method provided in an embodiment of the present invention.

[0027] Figure 7 This diagram illustrates the structure of the graphic guided filtering system provided in an embodiment of the present invention.

[0028] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0029] Various embodiments of the invention will now be described in more detail with reference to the accompanying drawings. In the various drawings, the same elements are indicated by the same or similar reference numerals. For clarity, the various parts in the drawings are not drawn to scale.

[0030] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0031] Figure 1 This diagram illustrates the OpenCL (Open Computing Language) platform. OpenCL is one of the few programming languages ​​that can be implemented, allowing different types of computing devices to run simultaneously. Therefore, programs created with OpenCL can run on multiple platforms at the same time.

[0032] See Figure 1 The OpenCL platform 100 may include a host processor 110 and at least one computing device 120.

[0033] The host processor 110 is the processor on which host programs are executed, and a workspace for executing the kernel can be defined. Here, the workspace is also divided into multiple workgroups, which can include multiple work items. Here, a work item corresponds to each point in the workspace and becomes the smallest unit of work. For example, the host processor 110 can be a general-purpose CPU (Central Processing Unit), but is not limited to this.

[0034] The computing device 120 may include at least one computing unit (CU), and each CU may include at least one processing unit (PE). The CU is a unit for processing a work group, and the PE is a unit for processing a work item. For example, the CU may process a work group received from the host processor 110, and the PE may process each work item contained within that work group. Therefore, multiple work items contained in a work group are processed in parallel by the PE. Alternatively, the computing device 120 may also be a multi-core processor, but is not limited to this; it can be any processor with at least one core. For example, the computing device 120 may be an embedded GPU (Graphics Processing Unit).

[0035] On the other hand, an OpenCL program may include a host program and at least one kernel program. The host program is executed by the host processor 110, and the host processor 110 sends the kernel program to the computing device 120 via commands. The computing device 120 performs calculations on its processing unit. Here, the kernel program executes on the computing device 120, completing the specific work of the OpenCL application. The computing device 120 processes the input data according to the kernel program and outputs it to the host.

[0036] Figure 2 This diagram illustrates a flowchart of the image guided filtering method provided in an embodiment of the present invention. Figure 2 As shown, the image guided filtering method includes the following steps.

[0037] In step S210, the input image and the guide image are read from the host memory into the global memory of the computing device, wherein the guide image is a preset image or the input image itself.

[0038] In this embodiment, the guided filtering includes an input image and a guide image. The guide image can be preset or it can be the input image itself.

[0039] The OpenCL compiler's memory model includes global memory, constant memory, local memory, and private memory. These four types of memory represent different memory regions, and their memory spaces are all related to the OpenCL Kernel source code. Within a kernel, there may be keywords such as `global`, `constant`, `local`, or `private`. Different keywords are used to specify the memory space used to store data created by variables. In this embodiment, the memory model includes global memory and private memory.

[0040] Global memory is visible to all work items in the executing kernel. Data transfer between the host processor 110 and the computing device 120 needs to be stored in global memory, which corresponds to the shared memory of the computing device 120 in this application. A work item refers to the innermost operation in a loop; work items accessing the same processing resources form a workgroup.

[0041] Private memory can only be accessed by the work item itself. Data retrieved by the work item from global memory is stored in private memory, which corresponds to the shared memory of the computing device 120 in this application. In this embodiment, variables using the `private` or `_private` keyword in the Kernel source code are stored in the shared memory of the computing device 120.

[0042] In a preferred embodiment, a ping-pong buffering method is used to read data from host memory into global memory. Global memory includes a first buffer and a second buffer. The input image and guide image of the current frame are read into the first buffer, and then the work item directly accesses the image data in the first buffer for image filtering. Simultaneously, the input image and guide image of the next frame are read into the second buffer. After the input image and guide image of the next frame are completely read into the second buffer, the work item directly accesses the image data in the second buffer for image filtering. While performing image filtering on the image data in the second buffer, the input image and guide image of the new frame are read into the first buffer. Data transmission and processing alternate between the first and second buffers; simultaneous data transmission and processing improve computational efficiency.

[0043] In step S220, in global memory, the input image and the guide image are divided into multiple input sub-images and multiple guide sub-images according to the parallel data width, and the input sub-images and multiple guide sub-images correspond one-to-one.

[0044] In this embodiment, the size of the input image and the guide image is W*H, where W is the width of the input image and H is the height of the input image. The input image and the guide image are divided into multiple sub-input images and multiple sub-guide images according to the parallel data width k, and the number of sub-input images and sub-guide images is W / k.

[0045] This embodiment uses a parallel data width w = 8 as an example, but it is not limited to this. The size of each sub-image is 8*H.

[0046] In step S230, the guided filtering processing of each input sub-image is assigned to multiple work items of the GPU, and the process of guided filtering processing of each input sub-image and the corresponding guided sub-image corresponds to one work item.

[0047] In this embodiment, there are a maximum of 1024 work items on the computing device. The work items are executed independently of each other. Each work group contains several work items. Each work item independently processes the kernel function corresponding to its work group. The work item completes the operation of the kernel function.

[0048] This application assigns the guided filtering processing of each input sub-image to a corresponding work item, and each work item processes the guided filtering processing of one input sub-image. The entire input image is divided into multiple input sub-images for parallel processing, which greatly reduces resource consumption.

[0049] In step S240, multiple work items call the first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image respectively, and merge and save the scaling matrix and offset matrix of the multiple input sub-images into shared memory.

[0050] In this embodiment, see Figure 3 Step S240 specifically includes steps S241 to S245.

[0051] In step S241, the neighborhood image of each input sub-image is obtained according to the filter width.

[0052] In this embodiment, to ensure the output image matches the input image size, edge-padding processing of the input image is required based on the filtering radius. Specifically, the edge-padding processing includes adding R rows of pixels to the top and bottom of the input image, and adding R columns of pixels to the left and right sides of both the input and guide images, where R is the filtering radius. The pixel values ​​of the added pixels are obtained by folding them symmetrically along the boundary between the original input and guide images (see [link to documentation]). Figure 4 Take R rows of pixels from the top and bottom of the input sub-image and R columns of pixels from the left and right as the corresponding neighborhood images of the input sub-image.

[0053] Figure 4The example uses W=640, H=720, parallel data width k=8, and filtering radius R=5, but it is not limited to these. Based on the parallel data width, the input image is divided into 80 input sub-images. Taking input sub-image 1 as an example, 5 rows or 5 columns of pixels are selected around the input sub-image as the neighborhood image of input image 1.

[0054] In step S242, each work item reads the corresponding input sub-image and its neighborhood image from the global memory using a memory-sharing method.

[0055] In this embodiment, each work item reads the corresponding input sub-image and its neighboring images from global memory. When retrieving data for each input sub-image and its neighboring images from global memory, each work item retrieves data from global memory according to its work item number m. The starting column corresponding to the m-th work item is (m-1)*k+1, where m = 1, 2, ..., W / k, and W is the width of the input image, and k is the width of the parallel data. Specifically, the starting column of the first input sub-image corresponding to the first work item is Col1, the starting row of its neighboring images is Row(-R+1), the ending row is Row(H+R), the starting column is Col(-R+1), and the ending column is Col(k+R). The starting column of the second input sub-image corresponding to the second work item is Col(k+1), the starting row of its neighboring images is Row(-R+1), the ending row is Row(H+R), the starting column is Col(k+1-R), and the ending column is Col(2k+R). Similarly, the starting column of the mth input sub-image corresponding to the mth work item is Col((m-1)*k+1), the starting column of its neighboring image is Row(-R+1), the ending column is Row(H+R), the starting column is Col((m-1)*k+1-R), and the ending column is Col(mk+R).

[0056] Each work item reads data from global memory according to its work item number m. The starting address for reading data from global memory is [-R+1, (m-1)*k+1-R]. Starting from row -R+1, k pixel values ​​are read sequentially from column (m-1)*k+1-R, shifting and looping w times to obtain w vector groups, each containing k pixel values. Each row is read in the same manner to obtain the data of the input sub-image and its neighboring images corresponding to that work item. It should be noted that multiple input sub-images are still stored in global memory. When each work item reads data from global memory, if the boundary of the input sub-image is the boundary of the original input image, R rows or R columns of data symmetrically along the boundary of the input sub-image are read to perform edge padding. For horizontal filtering, edge padding is performed on the left and right sides of the original input image; for vertical filtering, edge padding is performed on the top and bottom sides of the original input image.

[0057] A workgroup consists of 256 work items. Each work item can read a maximum of 32*H data from global memory, reading 4, 8, or 16 pixel values ​​at a time.

[0058] For example, when reading the first row of data, the first input sub-image reads the pixel values ​​of Col_5, Col_4, Col_3, Col_2, Col_1, and Col_1-Col_11 sequentially from global memory as a vector group; then it reads the pixel values ​​of Col_3, Col_2, Col_1, and Col_1-Col_13 as a vector group; thus realizing the edge-padding process of the input sub-image.

[0059] In step S243, the input sub-image and its neighboring images are shifted and summed according to the parallel data width and the filter width to obtain the mean filtering result avg_p of the input sub-image.

[0060] In this embodiment, step S243 specifically includes: constructing a first kernel function, which includes at least a first kernel function and a second kernel function. The first kernel function is used to calculate the sum of pixels in each row from column j1 to column j2 in the input sub-image and the corresponding neighborhood image. The second kernel function is used to calculate the sum of pixels in each column from row i1 to row i2 in the input sub-image and the corresponding neighborhood image. j1 = (m-1)*k+1-R, m = 1, 2, ..., W / k, j2 = j1+w-1, i1 = -R, ..., H+R, i2 = i1+w-1, where k is the parallel data width, w is the filtering width, w = 2R+1, and R is the filtering radius. Each work item completes the horizontal filtering of the input sub-image by cyclically calling the first kernel function and the vertical filtering of the input sub-image by cyclically calling the second kernel function. Intermediate data is stored in shared memory during the loop.

[0061] Specifically, each input sub-image and its neighborhood image include (k+w-1)*(H+w-1) pixels. When taking data for each row, k pixel values ​​are taken starting from the first column of the input sub-image and its neighborhood image. Then, the data is shifted backward to take k pixel values ​​starting from the second column. This process is repeated until k pixel values ​​are taken starting from column w, forming w vector groups. The average value of the w vector groups is then calculated to obtain the horizontal filtering result for each row. The horizontal filtering result for each row is then cyclically shifted along the column direction and the average value of the w rows is calculated to obtain the vertical filtering result for each column.

[0062] When performing vertical filtering on the input sub-image, an intermediate storage variable sum is set. The initial sum is the sum of the first w pixels in each column starting from the first pixel. Then, each time the sum is incremented by the difference between the (w+1)th pixel value and the first pixel value, where w is the filtering width. This process is repeated to sum and average the horizontal mean filtering results along the column direction to obtain the vertical mean filtering results.

[0063] In a preferred embodiment, each input sub-image and its neighborhood image include (k+w-1)*(H+w-1) pixels. When retrieving data for each row, k pixel values ​​are taken starting from the first column of the input sub-image and its neighborhood image. Then, shifting backward, k pixel values ​​are taken starting from the second column, and so on, starting from column w, forming w vector groups. These w vector groups are then summed to obtain the horizontal summation for each row. The horizontal summation for each row is then cyclically shifted along the column direction and summed sequentially for each of the w rows to obtain the vertical summation for each column. Finally, the vertical summation for each column is divided by R. 2 To obtain the final mean filtering result, first perform shifting and summation in the row and column directions, and then calculate the mean. This reduces calculation errors and image noise. Alternatively, the summation result can be divided by R. 2 The operation is converted into a multiplication operation to improve computational efficiency.

[0064] In this embodiment, the initial sum is the value of the previous sum plus the difference between the last row of the current sum and the first row of the previous sum. The current sum is calculated by using the difference from the previous sum, which can reduce the amount of calculation.

[0065] See Figure 5Taking the first input sub-image and its neighborhood image as an example, where k=8, R=5, w=11, and each row has 18 pixel values. Since data is read from global memory in batches of 16, to read all 18 pixel values ​​from the global data, we can read the 16 pixel values ​​in two steps: first, read the pixel values ​​of Col_5, Col_4, Col_3, Col_2, Col_1, and Col_1-Col_11 as a vector; then, read the pixel values ​​of Col_3, Col_4, Col_5, Col_6, Col_7, Col_8, Col_9, Col_1, and Col_1-Col_11 as a vector. The pixel values ​​of ol_2, Col_1, and Col_1-Col_13 are taken as a vector. Then, eight pixel values ​​are read from each of these two vectors as another vector. This process is repeated, shifting the values ​​eight times sequentially to obtain 11 vectors. These 11 vectors are then summed to obtain s1-s8, and the average of s1-s8 (divided by 11) is used as the horizontal filtering result for the corresponding row. This process is repeated for each row to obtain the horizontal filtering result of the input sub-image. The horizontal filtering result is then cyclically shifted along the column direction, summed, and averaged to obtain the vertical filtering result of the input sub-image. During the vertical filtering process, an intermediate variable `sum` is set. Initially, `sum` is the sum of the first 11 pixels in each column, starting from the first pixel. When calculating the `sum` value for rows 2 to 12, the difference between the first and second rows can be added to the `sum` value of the first 11 rows, thus reducing the computational load.

[0066] In step S244, the input sub-image is replaced with the guide sub-image, and the mean filtering result of the corresponding guide sub-image is obtained by repeating step one above.

[0067] In step S245, the covariance matrix and variance matrix of the input sub-image are calculated based on the mean filtering result of the input sub-image and the mean filtering result of the guide sub-image.

[0068] In step S246, the scaling matrix and offset matrix of the input sub-image are calculated based on the covariance matrix and variance matrix of the input sub-image, and the mean filtering results, scaling matrix and offset matrix of multiple input sub-images are stored in shared memory in a merged storage manner.

[0069] In this embodiment, the scaling matrix is ​​obtained based on the covariance matrix (cov) and variance matrix (var) of the input sub-image, with the formula: a = cov / (var + c), where c is an adjustment parameter that can be adjusted according to the effect of the output sub-image. The offset matrix is ​​obtained based on the mean filtering result of the input sub-image and the offset matrix, with the formula: b = avg_p*(1-a).

[0070] In step S250, multiple work items call the second kernel function in parallel to obtain the mean filtering results of the scaling and offset matrices of multiple input sub-images, calculate the corresponding output sub-images based on the mean filtering results of the multiple input sub-images and the corresponding scaling and offset matrices, and merge and save the multiple output sub-images into global memory.

[0071] In this embodiment, see Figure 6 Step S250 specifically includes steps S251 to S253.

[0072] In step S251, the scaling neighborhood matrix corresponding to the scaling matrix and the offset neighborhood matrix corresponding to the offset matrix of each input sub-image are obtained according to the filtering width.

[0073] In this embodiment, the scaling and offset matrices of the input image are padded with edges. The scaling and offset matrices of the input image are divided into scaling and offset matrices of multiple input sub-images according to the parallel data width. Then, the scaling and offset matrices of the input image are padded with edges. The scaling neighborhood matrix corresponding to the scaling matrix and the offset neighborhood matrix corresponding to the offset matrix of each input sub-image are obtained according to the filtering width. The specific steps are the same as in step S242, and will not be repeated here.

[0074] In step S252, each work item reads the scaling matrix of the input sub-image and its corresponding scaling neighborhood matrix and offset matrix and its corresponding offset neighborhood matrix from the shared memory using a memory-sharing method.

[0075] In this embodiment, when retrieving the scaling matrix of each input sub-image and its neighboring image data from shared memory, each work item retrieves data from shared memory according to its work item number m. The starting column corresponding to the m-th work item is (m-1)*k+1, where m = 1, 2, ..., W / k, and W is the width of the input image, and k is the width of the parallel data. Specifically, the starting column of the scaling matrix of the first input sub-image corresponding to the first work item is Col1, the starting row of its neighboring images is Row(-R+1), the ending row is Row(H+R), the starting column is Col(-R+1), and the ending column is Col(k+R). The starting column of the scaling matrix of the second input sub-image corresponding to the second work item is Col(k+1), the starting row of its neighboring images is Row(-R+1), the ending row is Row(H+R), the starting column is Col(k+1-R), and the ending column is Col(2k+R). Similarly, the scaling matrix of the mth input sub-image corresponding to the mth work item starts with Col((m-1)*k+1), the starting row of its neighborhood image is Row(-R+1), the ending row is Row(H+R), the starting column is Col((m-1)*k+1-R), and the ending column is Col(mk+R).

[0076] Each work item reads data from shared memory according to its work item number m. That is, the starting address for reading data from shared memory is [-R+1, (m-1)*k+1-R]. Starting from the -R+1th row, k pixel values ​​are read sequentially from the (m-1)*k+1-Rth column. The data is shifted and looped w times to obtain w vector groups, each of which includes k pixel values. Each row is read in the same way to read the scaling matrix of the input sub-image corresponding to the work item and the data of its neighborhood image.

[0077] The method for obtaining the offset matrix of the input sub-image and its neighboring image data is the same as the method for obtaining the scaling matrix of the input sub-image and its neighboring image data, and will not be repeated here.

[0078] In step S253, the scaling matrix of the input sub-image and its corresponding scaling neighborhood matrix and the offset matrix and their corresponding offset neighborhood matrix are shifted and summed according to the parallel data width and the filtering width to obtain the mean filtering result of the scaling matrix and the mean filtering result of the offset matrix of the input sub-image.

[0079] In this embodiment, the mean filtering method for the scaling matrix and the offset matrix is ​​the same as the mean filtering method for the input sub-image described above, and will not be repeated here.

[0080] The image filtering method described in the above embodiment, operating at a frequency of 800 MHz, includes a data input / output transfer process with a speed of 50 fps. After porting, this method not only improves performance but also saves resources for the embedded CPU chip, allowing it to perform more business processing and providing effective support for productization solutions.

[0081] The image guided filtering method provided by this invention divides the input image and the guided image into multiple sub-images, and distributes the guided filtering processing of multiple sub-images to multiple work items for parallel processing, thereby improving computational efficiency. Each work item is responsible for the guided filtering processing of the corresponding sub-image, which can reduce resource consumption and improve resource utilization.

[0082] Furthermore, each work item uses a memory-sharing approach to retrieve data from global memory. Each work item obtains the address of the corresponding sub-image in global memory based on the work item number, and retrieves the data of the corresponding sub-image through the address, which can reduce the data copying overhead.

[0083] Furthermore, each work item can directly obtain data through addressing to complete edge patching, which can reduce data storage.

[0084] Furthermore, image guided filtering is implemented by calling the GPU kernel function only twice, and data is processed in parallel during each kernel calculation, which improves computational efficiency. In addition, while reducing the number of GPU kernel function calls, global memory and shared memory are fully utilized, thereby reducing data transfer between the CPU and GPU and reducing algorithm execution time.

[0085] Figure 7 A schematic diagram of the image guided filtering system provided in an embodiment of the present invention is shown. Figure 7 As shown, the image-guided filtering structure includes an image acquisition module 710, an image segmentation module 720, a task allocation module 730, a filter parameter determination module 740, and an output image determination module 750.

[0086] The image acquisition module 710 is used to read the input image and the guide image from the host memory into the global memory of the GPU. The guide image is a preset image or the input image itself.

[0087] In this embodiment, the guided filtering includes an input image and a guide image. The guide image can be preset or it can be the input image itself.

[0088] The OpenCL compiler's memory model includes global memory, constant memory, local memory, and private memory. These four types of memory represent different memory regions, and their memory spaces are all related to the OpenCL Kernel source code. Within a kernel, there may be keywords such as `global`, `constant`, `local`, or `private`. Different keywords are used to specify the memory space used to store data created by variables. In this embodiment, the memory model includes global memory and private memory.

[0089] Global memory is visible to all work items in the executing kernel. Data transfer between the host processor 110 and the computing device 120 needs to be stored in global memory, which corresponds to the shared memory of the computing device 120 in this application. A work item refers to the innermost operation in a loop; work items accessing the same processing resources form a workgroup.

[0090] Private memory can only be accessed by the work item itself. Data retrieved by the work item from global memory is stored in private memory, which corresponds to the shared memory of the computing device 120 in this application. In this embodiment, variables using the `private` or `_private` keyword in the Kernel source code are stored in the shared memory of the computing device 120.

[0091] In a preferred embodiment, a ping-pong buffering method is used to read data from host memory into global memory. Global memory includes a first buffer and a second buffer. The input image and guide image of the current frame are read into the first buffer, and then the work item directly accesses the image data in the first buffer for image filtering. Simultaneously, the input image and guide image of the next frame are read into the second buffer. After the input image and guide image of the next frame are completely read into the second buffer, the work item directly accesses the image data in the second buffer for image filtering. While performing image filtering on the image data in the second buffer, the input image and guide image of the new frame are read into the first buffer. Data transmission and processing alternate between the first and second buffers; simultaneous data transmission and processing improve computational efficiency.

[0092] The image segmentation module 720 is used to divide the input image and the guide image into multiple input sub-images and multiple guide sub-images in global memory according to the parallel data width, wherein the input sub-images and multiple guide sub-images correspond one-to-one.

[0093] In this embodiment, the size of the input image and the guide image is W*H, where W is the width of the input image and H is the height of the input image. The input image and the guide image are divided into multiple sub-input images and multiple sub-guide images according to the parallel data width k, and the number of sub-input images and sub-guide images is W / k.

[0094] This embodiment uses a parallel data width w = 8 as an example, but it is not limited to this. The size of each sub-image is 8*H.

[0095] The task allocation module 730 is used to allocate the guided filtering processing of each input sub-image to multiple work items of the GPU. Each input sub-image and its corresponding guided sub-image undergo guided filtering processing, which corresponds to one work item.

[0096] In this embodiment, there are a maximum of 1024 work items on the computing device. The work items are executed independently of each other. Each work group contains several work items. Each work item independently processes the kernel function corresponding to its work group. The work item completes the operation of the kernel function.

[0097] This application assigns the guided filtering processing of each input sub-image to a corresponding work item, and each work item processes the guided filtering processing of one input sub-image. The entire input image is divided into multiple input sub-images for parallel processing, which greatly reduces resource consumption.

[0098] The filter parameter determination module 740 is used to control multiple work items to call the first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image respectively, and to merge and save the scaling matrix and offset matrix of multiple input sub-images into shared memory.

[0099] In this embodiment, the filter parameter determination module 740 obtains the neighborhood image of each input sub-image according to the filter width; controls each working item to read the corresponding input sub-image and its neighborhood image from the global memory using a memory-sharing method; performs shift and summation on the input sub-image and its neighborhood image according to the parallel data width and the filter width to obtain the mean filtering result avg_p of the input sub-image; replaces the input sub-image with the guiding sub-image, and repeats the above step one to obtain the mean filtering result of the corresponding guiding sub-image; calculates the covariance matrix and variance matrix of the input sub-image according to the mean filtering result of the input sub-image and the mean filtering result of the guiding sub-image; and calculates the scaling matrix and offset matrix of the input sub-image according to the covariance matrix and variance matrix of the input sub-image.

[0100] In this embodiment, to ensure the output image matches the input image size, edge-padding processing of the input image is required based on the filtering radius. Specifically, the edge-padding processing includes adding R rows of pixels to the top and bottom of the input image, and adding R columns of pixels to the left and right sides of both the input and guide images, where R is the filtering radius. The pixel values ​​of the added pixels are obtained by folding them symmetrically along the boundary between the original input and guide images (see [link to documentation]). Figure 4 Take R rows of pixels from the top and bottom of the input sub-image and R columns of pixels from the left and right as the corresponding neighborhood images of the input sub-image.

[0101] Figure 4 The example uses W=640, H=720, parallel data width k=8, and filtering radius R=5, but it is not limited to these. Based on the parallel data width, the input image is divided into 80 input sub-images. Taking input sub-image 1 as an example, 5 rows or 5 columns of pixels are selected around the input sub-image as the neighborhood image of input image 1.

[0102] Each work item reads the corresponding input sub-image and its neighboring images from global memory. When retrieving data for each input sub-image and its neighboring images from global memory, each work item retrieves data according to its work item number m. The starting column for the m-th work item is (m-1)*k+1, where m = 1, 2, ..., W / k, and W is the width of the input image, and k is the width of the parallel data. Specifically, the starting column of the first input sub-image corresponding to the first work item is Col1, the starting row of its neighboring images is Row(-R+1), the ending row is Row(H+R), the starting column is Col(-R+1), and the ending column is Col(k+R). The starting column of the second input sub-image corresponding to the second work item is Col(k+1), the starting row of its neighboring images is Row(-R+1), the ending row is Row(H+R), the starting column is Col(k+1-R), and the ending column is Col(2k+R). Similarly, the starting column of the mth input sub-image corresponding to the mth work item is Col((m-1)*k+1), the starting column of its neighboring image is Row(-R+1), the ending column is Row(H+R), the starting column is Col((m-1)*k+1-R), and the ending column is Col(mk+R).

[0103] Each work item reads data from global memory according to its work item number m. The starting address for reading data from global memory is [-R+1, (m-1)*k+1-R]. Starting from row -R+1, k pixel values ​​are read sequentially from column (m-1)*k+1-R, shifting and looping w times to obtain w vector groups, each containing k pixel values. Each row is read in the same manner to obtain the data of the input sub-image and its neighboring images corresponding to that work item. It should be noted that multiple input sub-images are still stored in global memory. When each work item reads data from global memory, if the boundary of the input sub-image is the boundary of the original input image, R rows or R columns of data symmetrically along the boundary of the input sub-image are read to perform edge padding. For horizontal filtering, edge padding is performed on the left and right sides of the original input image; for vertical filtering, edge padding is performed on the top and bottom sides of the original input image.

[0104] A workgroup consists of 256 work items. Each work item can read a maximum of 32*H data from global memory, reading 4, 8, or 16 pixel values ​​at a time.

[0105] For example, when reading the first row of data, the first input sub-image reads the pixel values ​​of Col_5, Col_4, Col_3, Col_2, Col_1, and Col_1-Col_11 sequentially from global memory as a vector group; then it reads the pixel values ​​of Col_3, Col_2, Col_1, and Col_1-Col_13 as a vector group; thus realizing the edge-padding process of the input sub-image.

[0106] Specifically, each input sub-image and its neighborhood image include (k+w-1)*(H+w-1) pixels. When taking data for each row, k pixel values ​​are taken starting from the first column of the input sub-image and its neighborhood image. Then, the data is shifted backward to take k pixel values ​​starting from the second column. This process is repeated until k pixel values ​​are taken starting from column w, forming w vector groups. The average value of the w vector groups is then calculated to obtain the horizontal filtering result for each row. The horizontal filtering result for each row is then cyclically shifted along the column direction and the average value of the w rows is calculated to obtain the vertical filtering result for each column, which is used as the mean filtering result.

[0107] When performing vertical filtering on the input sub-image, an intermediate storage variable sum is set. The initial sum is the sum of the first w pixels in each column starting from the first pixel. Then, the sum is incremented by taking the difference between the (w+1)th pixel value and the first pixel value each time, where w is the filtering width. This process is repeated to sum and average the horizontal filtering results along the column direction to obtain the vertical filtering result.

[0108] In a preferred embodiment, each input sub-image and its neighborhood image include (k+w-1)*(H+w-1) pixels. When data is retrieved for each row, k pixel values ​​are taken starting from the first column of the input sub-image and its neighborhood image, then shifted backward to take k pixel values ​​starting from the second column, and so on, starting from the w columns, forming w vector groups. The w vector groups are then summed to obtain the horizontal filtering result for each row. The horizontal filtering result for each row is then cyclically shifted along the column direction and summed for each of the w rows to obtain the vertical filtering result for each column. The mean square value of the vertical filtering result for each column is calculated as the mean filtering result. Specifically, the vertical filtering result for each column is divided by the square of the filtering radius (R). 2 This process yields the final mean-squared filtering result. First, shifting and summing along the row and column directions, then calculating the mean square value, reduces calculation errors and image noise. Furthermore, the vertical filtering result can be compared with the square of the filtering radius R. 2 The division operation is converted into a multiplication operation to improve computational efficiency.

[0109] In this embodiment, the initial sum is the value of the previous sum plus the difference between the last row of the current sum and the first row of the previous sum. The current sum is calculated by using the difference from the previous sum, which can reduce the amount of calculation.

[0110] See Figure 5Taking the first input sub-image and its neighborhood image as an example, where k=8, R=5, w=11, and each row has 18 pixel values. Since data is read from global memory in batches of 16, to read all 18 pixel values ​​from the global data, we can read the 16 pixel values ​​in two steps: first, read the pixel values ​​of Col_5, Col_4, Col_3, Col_2, Col_1, and Col_1-Col_11 as a vector; then, read the pixel values ​​of Col_3, Col_4, Col_5, Col_6, Col_7, Col_8, Col_9, Col_1, and Col_1-Col_11 as a vector. The pixel values ​​of ol_2, Col_1, and Col_1-Col_13 are taken as a vector. Then, eight pixel values ​​are read from each of these two vectors as another vector. This process is repeated, shifting the values ​​eight times sequentially to obtain 11 vectors. These 11 vectors are then summed to obtain s1-s8, and the average of s1-s8 (divided by 11) is used as the horizontal filtering result for the corresponding row. This process is repeated for each row to obtain the horizontal filtering result of the input sub-image. The horizontal filtering result is then cyclically shifted along the column direction, summed, and averaged to obtain the vertical filtering result of the input sub-image. During the vertical filtering process, an intermediate variable `sum` is set. Initially, `sum` is the sum of the first 11 pixels in each column, starting from the first pixel. When calculating the `sum` value for rows 2 to 12, the difference between the first and second rows can be added to the `sum` value of the first 11 rows, thus reducing the computational load.

[0111] The output image determination module 750 is used to control multiple work items to call the second kernel function in parallel to obtain the mean filtering results of the scaling matrix and offset matrix of multiple input sub-images, and to calculate the corresponding output sub-image based on the mean filtering results of the multiple input sub-images and the corresponding scaling matrix and offset matrix, and to merge and save the multiple output sub-images into global memory.

[0112] The image guided filtering system provided by this invention divides the input image and the guided image into multiple sub-images, and distributes the guided filtering processing of multiple sub-images to multiple work items for parallel processing, thereby improving computational efficiency. Each work item is responsible for the guided filtering processing of the corresponding sub-image, which can reduce resource consumption and improve resource utilization.

[0113] Furthermore, each work item uses a memory-sharing approach to retrieve data from global memory. Each work item obtains the address of the corresponding sub-image in global memory based on the work item number, and retrieves the data of the corresponding sub-image through the address, which can reduce the data copying overhead.

[0114] Furthermore, each work item can directly obtain data through addressing to complete edge patching, which can reduce data storage.

[0115] Furthermore, image guided filtering is implemented by calling the GPU kernel function only twice, and data is processed in parallel during each kernel calculation, which improves computational efficiency. In addition, while reducing the number of GPU kernel function calls, global memory and shared memory are fully utilized, thereby reducing data transfer between the CPU and GPU and reducing algorithm execution time.

[0116] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. For example... Figure 8 As shown, the electronic device 100 of this embodiment includes: at least one processor 101 ( Figure 8 (Only one is shown) a processor, a memory 102, and a computer program 103 stored in the memory 102 and executable on the at least one processor 101, wherein the processor 101 executes the computer program 103 to implement the steps of the neural network model compression method described above.

[0117] The electronic device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 100 and does not constitute a limitation on electronic device 100. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0118] The processor 101 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0119] In some embodiments, the memory 102 may be an internal storage unit of the electronic device 100, such as a hard disk or memory of the electronic device 100. In other embodiments, the memory 102 may be an external storage device of the electronic device 100, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 100. Furthermore, the memory 102 may include both internal and external storage units of the electronic device 100. The memory 102 is used to store operating systems, applications, boot loaders, data, and other programs, such as the program code of computer programs. The memory 102 can also be used to temporarily store data that has been output or will be output.

[0120] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0121] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0122] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0123] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] As described above, these embodiments of the present invention do not exhaustively cover all details, nor do they limit the invention to the specific embodiments described. Clearly, many modifications and variations can be made based on the above description. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to effectively utilize the invention and its modifications. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. An image guided filtering method, characterized in that, include: The input image and the guide image are read from the host memory into the global memory of the GPU. The guide image is either a preset image or the input image itself. In global memory, the input image and the guide image are divided into multiple input sub-images and multiple guide sub-images according to the parallel data width, and the input sub-images and multiple guide sub-images correspond one-to-one. The guided filtering process for each input sub-image is assigned to multiple work items on the GPU, with each input sub-image and its corresponding guided sub-image corresponding to one work item. Multiple work items call the first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image respectively, and then merge and save the scaling matrix and offset matrix of the multiple input sub-images into shared memory; Multiple work items call the second kernel function in parallel to obtain the mean filtering results of the scaling and offset matrices of multiple input sub-images, and calculate the corresponding output sub-images based on the mean filtering results of the multiple input sub-images and the corresponding scaling and offset matrices, and merge and save the multiple output sub-images into global memory; The process of multiple work items calling the first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image includes: Obtain the corresponding neighborhood image for each input sub-image based on the filter width; Each work item uses a memory-sharing approach to read data from the input sub-image and its neighboring images from global memory; The input sub-image and its neighboring images are shifted and summed according to the parallel data width and the filter width to obtain the mean filter result of the input sub-image; Replace the input sub-image with the guiding sub-image and obtain the mean filtering result of the corresponding guiding sub-image; Calculate the covariance matrix and variance matrix of the input sub-image and the guide sub-image based on the mean filtering results of the input sub-image and the guide sub-image; Calculate the scaling matrix and offset matrix of the input sub-image based on the covariance matrix of the input sub-image and the guide sub-image, and the variance matrix of the guide sub-image.

2. The image guided filtering method according to claim 1, characterized in that, The neighborhood image corresponding to each input sub-image is obtained based on the filter width, including: The input image and the guide image are padded with edges according to the filtering radius. The edge padded process includes adding R rows of pixels on the top and bottom sides of the input image and R columns of pixels on the left and right sides of the input image and the guide image, where R is the filtering radius and the pixel values ​​of the padded pixels are obtained by folding them with the boundary of the original input image and the original guide image as the axis of symmetry. Take R rows of pixels from the top and bottom of the input sub-image and R columns of pixels from the left and right as the corresponding neighborhood images of the input sub-image.

3. The image guided filtering method according to claim 2, characterized in that, The mean filtering result of the input sub-image is obtained by shifting and summing the input sub-image and its corresponding neighboring images based on the parallel data width and the filter width, including: Construct a first kernel function, which includes at least a first kernel function and a second kernel function. The first kernel function is used to calculate the sum of pixels in each row from column j1 to column j2 in the input sub-image and the corresponding neighborhood image. The second kernel function is used to calculate the sum of pixels in each column from row i1 to row i2 in the input sub-image and the corresponding neighborhood image. j1=(m-1)*k+1-R, m=1,2,……,W / k, j2=j1+w-1, i1=-R,……,H+R; i2=i1+w-1, where k is the parallel data width, w is the filter width, w=2R+1, R is the filter radius, and M is the work item number. Each task uses a loop to call the first kernel function to perform horizontal filtering on the input sub-image, and a loop to call the second kernel function to perform vertical filtering on the input sub-image. Intermediate data is stored in shared memory during the loop.

4. The image guided filtering method according to claim 3, characterized in that, When performing horizontal filtering on the input sub-image, each row is shifted starting from the first pixel to obtain w sets of vector values. Each set of vector values ​​includes k pixel values. The w sets of vector values ​​are summed and averaged to obtain the horizontal filtering result. When performing vertical filtering on the input sub-image, an intermediate storage variable `sum` is set. The initial `sum` is the sum of the first w pixels in each column starting from the first pixel. Then, it is incremented by the difference between the (w+1)th pixel value and the first pixel value, where w is the filtering width. This process is repeated to sum and average the horizontal mean filtering results along the column direction to obtain the vertical filtering result as the mean filtering result.

5. The image guided filtering method according to claim 3, characterized in that, When performing horizontal filtering on the input sub-image, each row is shifted starting from the first pixel to obtain w sets of vector values. Each set of vector values ​​includes k pixel values. The w sets of vector values ​​are summed to obtain the horizontal filtering result. When performing vertical filtering on the input sub-image, an intermediate storage variable `sum` is set. The initial `sum` is the sum of the first w pixels in each column starting from the first pixel. Then, it is incremented by the difference between the (w+1)th pixel value and the first pixel value, where `w` is the filtering width. This process is repeated to sum the horizontal filtering results along the column direction to obtain the vertical filtering result. The mean square value of the vertical filtering result is then calculated as the mean filtering result.

6. The image guided filtering method according to claim 2, characterized in that, Each work item retrieves data from global memory based on its work item number m. The starting column corresponding to the m-th work item is (m-1)*k+1, where m=1,2,...,W / k, and W is the width of the input image and k is the width of the parallel data.

7. The image guided filtering method according to claim 2, characterized in that, Multiple work items call the second kernel function in parallel to obtain the mean filtering results of the scaling and offset matrices of multiple input sub-images, including: Based on the filter width, obtain the scaling neighborhood matrix corresponding to the scaling matrix and the offset neighborhood matrix corresponding to the offset matrix for each input sub-image; Each work item uses a memory-sharing approach to read the scaling matrix and its corresponding scaling neighborhood matrix and offset matrix and its corresponding offset neighborhood matrix of the input sub-image from shared memory; Based on the parallel data width and the filtering width, the scaling matrix of the input sub-image and its corresponding scaling neighborhood matrix and offset matrix and their corresponding offset neighborhood matrix are shifted and summed to obtain the mean filtering result of the scaling matrix and the mean filtering result of the offset matrix of the input sub-image.

8. The image guided filtering method according to claim 7, characterized in that, Based on the parallel data width and the filtering width, the scaling matrix of the input sub-image and its corresponding scaling neighborhood matrix and offset matrix and their corresponding offset neighborhood matrix are shifted and summed to obtain the mean filtering result of the scaling matrix and the mean filtering result of the offset matrix of the input sub-image, including: A second kernel function is constructed. The first kernel function includes a first kernel function, a second kernel function, a third kernel function, and a fourth kernel function. The first kernel function is used to calculate the sum of pixels in each row from column j1 to column j2 in the scaling matrix of the input sub-image and its corresponding scaling neighborhood matrix. The second kernel function is used to calculate the sum of pixels in each column from row i1 to row i2 in the scaling matrix of the input sub-image and its corresponding scaling neighborhood matrix. The first kernel function is used to calculate the sum of pixels in each row from column j1 to column j2 in the offset matrix of the input sub-image and its corresponding offset neighborhood matrix. The second kernel function is used to calculate the sum of pixels in each column from row i1 to row i2 in the offset matrix of the input sub-image and its corresponding offset neighborhood matrix. j1=1,2,...,k; j2=j1+w-1, i1=1,2,...,k; i2=i1+w-1, where k is the parallel data width, w is the filter width, w=2R+1, and R is the filter radius. Each task uses a loop to call the first kernel function to perform horizontal filtering on the scaling matrix of the input sub-image, the second kernel function to perform vertical filtering on the scaling matrix of the input sub-image, the third kernel function to perform horizontal filtering on the offset matrix of the input sub-image, and the fourth kernel function to perform vertical filtering on the offset matrix of the input sub-image. Intermediate data is stored in shared memory during the loop.

9. An image-guided filtering system, characterized in that, include: An image acquisition module is used to read the input image and the guide image from the host memory into the global memory of the GPU, wherein the guide image is a preset image or the input image itself; The image segmentation module is used to divide the input image and the guide image into multiple input sub-images and multiple guide sub-images in global memory according to the parallel data width, wherein the input sub-images and multiple guide sub-images correspond one-to-one. The task allocation module is used to allocate the guided filtering processing of each input sub-image to multiple work items of the GPU. Each input sub-image and its corresponding guided sub-image undergo guided filtering processing, which corresponds to one work item. The filter parameter determination module is used to control multiple work items to call the first kernel function in parallel to obtain the scaling matrix and offset matrix of the input sub-image respectively, and to merge and save the scaling matrix and offset matrix of multiple input sub-images into shared memory; The output image determination module is used to control multiple work items to call the second kernel function in parallel to obtain the mean filtering results of the scaling matrix and offset matrix of multiple input sub-images, and to calculate the corresponding output sub-image based on the mean filtering results of the multiple input sub-images and the corresponding scaling matrix and offset matrix, and to merge and save the multiple output sub-images into global memory; The filter parameter determination module is further used to obtain the corresponding neighborhood image of each input sub-image based on the filter width; each work item reads the data of the input sub-image and its neighborhood image from the global memory using a memory-sharing method; shifts and sums the input sub-image and its neighborhood image according to the parallel data width and the filter width to obtain the mean filtering result of the input sub-image; replaces the input sub-image with the guide sub-image and obtains the mean filtering result of the corresponding guide sub-image; calculates the covariance matrix and variance matrix of the input sub-image and the guide sub-image based on the mean filtering result of the input sub-image and the mean filtering result of the guide sub-image; and calculates the scaling matrix and offset matrix of the input sub-image based on the covariance matrix of the input sub-image and the guide sub-image and the variance matrix of the guide sub-image.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image guided filtering method as described in any one of claims 1-8.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image guided filtering method as described in any one of claims 1-8.

Images