A method for adaptive image segmentation based on super-resolution convolution kernel size

Abstract

This invention discloses an adaptive image segmentation method based on the size of the super-resolution convolutional kernel, relating to the field of image processing technology. The method includes the following steps: Step 1: Determine the number of image blocks to be segmented based on the system memory size; Step 2: Determine the size of the first convolutional kernel used for feature extraction in the neural network and the number of connected boundaries of the image blocks during segmentation; Step 3: Calculate the pixels of each image block using an adaptive formula; Step 4: Perform overlapping segmentation on the original image. This invention solves the problem of data loss due to segmentation of boundary data; it avoids significant distortion of the merged image boundaries after super-resolution reconstruction, thereby ensuring the quality of the super-resolution image after reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a method for adaptive image segmentation based on the size of the super-resolution convolution kernel. Background Technology

[0002] When deploying image super-resolution using in-memory computation or edge testing, the requirements for segmenting large input images are becoming increasingly stringent due to memory constraints such as SDRAM. Currently, two methods are commonly used in the deployment phase. The first method is to downsample the large input image, but this leads to a decrease in image accuracy, significant information loss during downsampling, and ultimately poor super-resolution results. The second method is to crop the image into smaller pieces for separate processing, but if the images are simply and crudely cut apart, data loss at the boundaries can easily occur, resulting in significant distortion at the boundaries when the images are re-merged. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and propose a method for adaptive image segmentation based on the size of the super-resolution convolution kernel.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] An adaptive image segmentation method based on super-resolution convolution kernel size includes the following steps:

[0006] Step 1: Determine the number of image blocks to be divided based on the system memory size;

[0007] Step 2: Determine the size of the first convolutional kernel used for feature extraction in the neural network and the number of connected boundaries when segmenting the image patch;

[0008] Step 3: Calculate the pixels of each image patch using an adaptive formula, which is:

[0009]

[0010] The value of i is in the range of (0, 1, 2);

[0011] The value of j is in the range of (0, 1, 2);

[0012] Where Pic represents the number of pixels in the image, and w n The width and h represent the length of the image patch obtained after average segmentation. n represents the width (height) of the image patch obtained after average segmentation; n represents the number of small image patches formed after segmentation; i represents the number of connected boundaries on the left and right sides of the image patch during segmentation; j represents the number of connected boundaries on the top and bottom sides of the image patch during segmentation; k represents the size of the first convolutional kernel of the neural network used for feature extraction.

[0013] When i = j = 0, it indicates non-overlapping segmentation.

[0014] Step 4: Overlap the original image and segment it.

[0015] Furthermore, the overlapping segmentation method for the original image in step 3 includes global segmentation, width-direction segmentation, and height-direction segmentation.

[0016] Furthermore, the global segmentation specifically involves: calculating the expanded pixels of each image block using an adaptive formula according to the order from left to right and from top to bottom; expanding the overlapping boundary of the globally segmented image blocks based on the original coordinates of the image blocks and the expanded image block pixels to obtain the expanded image block coordinates; and performing overlapping slicing based on the expanded image block coordinates.

[0017] Furthermore, the expansion of the overlapping boundaries of the globally segmented image blocks specifically involves: determining whether the overlapping boundaries of the image blocks simultaneously include left and right boundaries or simultaneously include top and bottom boundaries; if so, then adding half the expansion pixels to both left and right boundaries or adding half the expansion pixels to both top and bottom boundaries; if not, then all the expansion pixels are expanded to the left or right boundary or all to the top or bottom boundary.

[0018] Furthermore, the width-direction segmentation specifically involves: calculating the expanded pixels of each image block using an adaptive formula in a top-to-bottom order; expanding the overlapping boundary of the width-direction segmented image blocks based on the original coordinates of the image blocks and the expanded image block pixels to obtain the expanded image block coordinates; and performing overlapping slicing based on the expanded image block coordinates.

[0019] Furthermore, the expansion of the overlapping boundary of the image blocks segmented in the width direction specifically involves: determining whether the overlapping boundary of the image blocks includes both the upper and lower boundaries; if so, increasing the upper and lower boundaries by half the number of pixels; if not, expanding all the pixels to the upper or lower boundary.

[0020] Furthermore, the height-direction segmentation specifically involves: calculating the expanded pixels of each image block using an adaptive formula in a left-to-right order; expanding the overlapping boundary of the height-direction segmented image blocks based on the original coordinates of the image blocks and the expanded image block pixels to obtain the expanded image block coordinates; and performing overlapping slicing based on the expanded image block coordinates.

[0021] Furthermore, the expansion of the overlapping boundary of the image blocks segmented in the height direction specifically involves: determining whether the overlapping boundary of the image blocks includes both left and right boundaries; if so, increasing the left and right boundaries by half the number of pixels; if not, expanding all the pixels to either the left or right boundary.

[0022] The beneficial effects of this invention are: This invention solves the problem of data loss due to boundary data segmentation; it avoids the phenomenon of obvious distortion at the boundaries of the re-merged images after super-resolution reconstruction, thereby ensuring the quality of the super-resolution image after super-resolution reconstruction. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0024] Figure 1 This is a flowchart of the method of the present invention.

[0025] Figure 2 It is a global segmentation graph.

[0026] Figure 3 This is a diagram illustrating the loss of boundary data during convolution.

[0027] Figure 4 It is a width-segmented image.

[0028] Figure 5 It is a height segmentation image. Detailed Implementation

[0029] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] In this embodiment, as Figure 1 As shown, a method for adaptive image segmentation based on super-resolution convolution kernel size includes the following steps:

[0032] Step 1: Determine the number of image blocks to be divided based on the system memory size;

[0033] Step 2: Determine the size of the first convolutional kernel used for feature extraction in the neural network and the number of connected boundaries when segmenting the image patch;

[0034] Step 3: Calculate the pixels of each image patch using an adaptive formula;

[0035] Step 4: Overlap the original image and segment it.

[0036] When deploying image super-resolution, assuming the input is a 480P (720*480) image, this case will take the example of dividing it into n=9 blocks according to the size of the SDRAM in memory. Figure 2 The solid line portion shown indicates that i = j = 0, meaning there is no overlapping segmentation. The size of each image block is w = 240 and h = 160. The coordinates of the first image A in the upper left corner are (1, 3, 45, 15), the coordinates of the second image B are (3, 6, 46, 45), and so on.

[0037] However, this method of segmentation can lead to the loss of boundary data during convolution, such as... Figure 3 As shown, the kernel size k = 3, and the kernel size defines the receptive field of the convolution. For the segmented image, using... Figure 1 Taking the top-left image A as an example, assuming padding is 0, when the convolution kernel performs convolution at the boundary of the segmentation (right or bottom), the data originally corresponding to the boundary is lost due to the segmentation, which will lead to the following two situations.

[0038] A. The number of convolutions on the right or bottom side will be two fewer than the original, and the result will be inconsistent with the original.

[0039] B. Even if you fill the right or bottom, the result will still be inconsistent because the fill value is different from the original.

[0040] Therefore, after super-resolution reconstruction, the re-merged image boundaries have obvious distortion problems.

[0041] In this scheme, the size of the first convolutional kernel used for feature extraction in the neural network is substituted into the adaptive formula for image patch expansion calculation. This is because if the outward expansion pixels are a preset value m, assuming m = 1, then when encountering a convolutional kernel with k >= (m + 2), Figure 1 Taking the top-left image A as an example, when the convolution kernel performs convolution at the boundary (right or bottom) of the segmentation, the data originally corresponding to the boundary is lost due to the segmentation. However, since only m is expanded, the number of convolutions on the right or bottom side will be >= 1 fewer than the original, and the result will be inconsistent with the original.

[0042] Similarly, assuming a preset value m = 2 or 3, if a convolution kernel with k >= (m+2) is encountered, then... Figure 1 Taking the top-left image A as an example, when the convolution kernel performs convolution at the boundary (right or bottom) of the segmentation, the data originally corresponding to the boundary is lost due to the segmentation. However, since only m is expanded, the number of convolutions on the right or bottom side will be >= 1 fewer than the original, and the result will be inconsistent with the original.

[0043] Similarly, assuming the preset value m = 4 or 5, then when encountering a convolution kernel with k >= (m+2), Figure 1 Taking the top-left image A as an example, when the convolution kernel performs convolution at the boundary (right or bottom) of the segmentation, the data originally corresponding to the boundary is lost due to the segmentation. However, since only m is expanded, the number of convolutions on the right or bottom side will be >= 1 fewer than the original, and the result will be inconsistent with the original.

[0044] In summary, the larger the convolution kernel, the more pixels need to be expanded. If the preset value of padding is very small, the number of convolutions at the split boundary will be much less than the original. When merging, the boundary pixels will differ greatly from the original values. Even if median filtering is used to smooth out the distortion, there will still be a difference compared to the original values.

[0045] This proposal suggests first determining the number of image blocks to be segmented based on the system memory size, and then performing adaptive overlapping segmentation of the original image based on the size of the first convolutional kernel used for feature extraction in the deep neural network and the number of connected boundaries during segmentation. The adaptive formula is as follows:

[0046]

[0047] The value of i is in the range of (0, 1, 2);

[0048] The value of j is in the range of (0, 1, 2);

[0049] Among them, w n The width and h represent the length of the image patch obtained after average segmentation. n represents the width (height) of the image patch obtained after average segmentation; n represents the number of small image patches formed after segmentation; i represents the number of connected boundaries on the left and right sides of the image patch during segmentation; j represents the number of connected boundaries on the top and bottom sides of the image patch during segmentation; k represents the size of the first convolutional kernel of the neural network used for feature extraction.

[0050] Based on the memory usage and code implementation difficulty, there are three possible splitting methods:

[0051] 1. Global partitioning

[0052] This case uses a division into n=9 pieces as an example. See the detailed division diagram below. Figure 2 The dashed section is defined according to the adaptive segmentation formula:

[0053]

[0054] The first image A in the upper left corner,

[0055] Width = w1 + i*(k-1);

[0056] Height = h1 + j*(k-1);

[0057] Where i = 1, j = 1; assume the size of the first convolutional kernel used for feature extraction is k = 9;

[0058] The coordinates after overlapping and segmentation are (1, 4, 18, 16), and the corresponding actual Width and Height are (240+8)*(160+8).

[0059] The second image, B, is divided into overlapping sections on both the left and right sides.

[0060] Width = w2 + i*(k-1);

[0061] Height = h² + j*(k-1);

[0062] Where i = 2, j = 1;

[0063] The corresponding coordinates are (2, 17, 20, 7), and the corresponding actual Width and Height are (240+16)*(160+8).

[0064] Similarly, the coordinates of Figure C are (5, 19, 21, 8).

[0065] Figure D requires overlapping on three sides, with corresponding coordinates (9, 30, 32, 11).

[0066] Figure E is divided into overlapping sections on all sides, with corresponding coordinates (10, 13, 34, 31). ......

[0068] Assuming a super-resolution of twice the normal value is performed, the reconstructed images need to be merged. Image A, after reconstruction, has a size of 2*(240+8)*(160+8)*2, but only the image size before overlapping and segmentation (2*240*160*2) is used. Image B also only uses the image size before overlapping and segmentation, and so on. Finally, the nine images are merged back into one large image according to the order in which they were segmented.

[0069] 2. Width direction segmentation

[0070] See detailed segmentation diagram Figure 4 It is divided into 9 blocks along the width direction, according to the adaptive segmentation formula:

[0071]

[0072] Therefore, the average size of each image segment is Width = 720 and Height = 480 / 9, which we'll tentatively take as 54.

[0073] Width = w1 + i*(k-1);

[0074] Height = h1 + j*(k-1);

[0075] Where i = 0, j = 1; assume the size of the first convolutional kernel used for feature extraction is k = 9;

[0076] After overlapping and segmentation, the coordinates of the first graph A are (1, 8, 35, 30), and the corresponding actual Width and Height are 720*(54+8). The coordinates of the second graph B are (23, 28, 21, 16), and so on.

[0077] 3. Height direction segmentation

[0078] See detailed segmentation diagram Figure 5 It is divided into 9 blocks along the height direction, according to the adaptive segmentation formula:

[0079]

[0080] Therefore, the average size of each image segment is Width = 720 / 9 = 80, and Height = 480.

[0081] Width = w1 + i*(k-1);

[0082] Height = h1 + j*(k-1);

[0083] Where i = 1, j = 0; assume the size of the first convolutional kernel used for feature extraction is k = 9;

[0084] The coordinates of the first graph A are (1, 4, 40, 37), and the corresponding actual Width and Height are (80+8)*480. The coordinates of the second graph B are (2, 7, 43, 38), and so on.

[0085] For these three segmentation methods, the size of each image block is determined based on the size of the SDRAM. For a 480P (720*480) image, assuming the size of the first convolutional kernel used for feature extraction is k=9,

[0086] (a) If it is divided into n = 4 pieces:

[0087] 1) Global Segmentation

[0088] After expansion, the size of each block is (320+8)*(240+8)=81,344;

[0089] 2) Segmentation in the width direction

[0090] After expansion, the size of the first and fourth blocks is 720*(120+8)=92160;

[0091] The size of the second and third blocks is 720*(120+8*2)=97920;

[0092] 3) Height direction segmentation

[0093] After expansion, the size of the first and fourth blocks is (180+8)*480=90240;

[0094] The size of the second and third blocks is (180 + 8 * 2) * 480 = 94080;

[0095] The first global partitioning method occupies the least memory space and is the most efficient, so it should be the preferred partitioning method.

[0096] (ii) If it is divided into n = 9 pieces:

[0097] 1) Global Segmentation

[0098] The size of the first block after expansion is (240+8)*(160+8)=41664;

[0099] The size of the second block is (240 + 8 * 2) * (160 + 8) = 43008;

[0100] The size of the third block is (240+8)*(160+8)=41664;

[0101] The size of the fourth piece is (240+8)*(160+8*2)=43648;

[0102] The size of the fifth piece is (240 + 8 * 2) * (160 + 8 * 2) = 45056; ......

[0104] 2) Segmentation in the width direction

[0105] The size of the first expanded block is 720*(54+8)=44640;

[0106] The size of the last piece is 720*(48+8)=40320;

[0107] The remaining seven pieces are 720*(54+8*2)=50400 in size;

[0108] 3) Height direction segmentation

[0109] After expansion, the size of the first and ninth blocks is (80+8)*480=42240;

[0110] The remaining seven pieces are (80 + 8 * 2) * 480 = 46080 in size;

[0111] The second partitioning method occupies the most memory space, while the first partitioning method has large fluctuations in memory space, which is not conducive to code implementation. It is recommended to use the third method first.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, ROM, RAM, etc.

[0114] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for adaptive image segmentation based on super-resolution convolution kernel size, characterized in that, Includes the following steps: Step 1: Determine the number of image blocks to be divided based on the system memory size; Step 2: Determine the size of the first convolutional kernel used for feature extraction in the neural network and the number of connected boundaries when segmenting the image patch; Step 3: Calculate the pixels of each image patch using an adaptive formula, which is: ， Where Pic represents the number of pixels in the image, and w n The width and h represent the length of the image patch obtained after average segmentation. n The height represents the width of the image patch obtained after average segmentation; n represents the number of image patches formed after segmentation; i represents the number of connected boundaries on the left and right sides of the image patch during segmentation; j represents the number of connected boundaries on the top and bottom sides of the image patch during segmentation; k represents the size of the first convolutional kernel used for feature extraction in the neural network. Step 4: Overlap the original image and segment it.

2. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 1, characterized in that, The overlapping segmentation methods for the original image in step 3 include global segmentation, width-direction segmentation, and height-direction segmentation.

3. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 2, characterized in that, The global segmentation specifically involves: calculating the expanded pixels of each image block using an adaptive formula, following the order from left to right and from top to bottom; expanding the overlapping boundaries of the globally segmented image blocks based on the original coordinates of the image blocks and the expanded image block pixels to obtain the expanded image block coordinates; and performing overlapping slicing based on the expanded image block coordinates.

4. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 3, characterized in that, The specific steps for expanding the overlapping boundaries of globally segmented image blocks are as follows: determine whether the overlapping boundaries of the image blocks include both left and right boundaries or both top and bottom boundaries. If so, increase the expansion pixels by half for both left and right boundaries or increase the expansion pixels by half for both top and bottom boundaries. If not, then all the expansion pixels are expanded to the left or right boundary or to the top or bottom boundary.

5. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 2, characterized in that, The width-direction segmentation specifically involves: calculating the expanded pixels of each image block using an adaptive formula in a top-to-bottom order; expanding the overlapping boundary of the width-direction segmented image blocks based on the original coordinates of the image blocks and the expanded image block pixels to obtain the expanded image block coordinates; and performing overlapping slicing based on the expanded image block coordinates.

6. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 5, characterized in that, The specific steps for expanding the overlapping boundary of the image blocks segmented in the width direction are as follows: determine whether the overlapping boundary of the image blocks includes both the upper and lower boundaries. If so, increase the expansion pixels by half for both the upper and lower boundaries. If not, expand all the expansion pixels to the upper or lower boundary.

7. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 2, characterized in that, The height-direction segmentation specifically involves: calculating the expanded pixels of each image block from left to right using an adaptive formula; expanding the overlapping boundary of the height-direction segmented image blocks based on the original coordinates of the image blocks and the expanded image block pixels to obtain the expanded image block coordinates; and performing overlapping slicing based on the expanded image block coordinates.

8. The method for adaptive image segmentation based on super-resolution convolution kernel size according to claim 7, characterized in that, The specific steps for expanding the overlapping boundary of the image blocks segmented in the height direction are as follows: determine whether the overlapping boundary of the image blocks includes both left and right boundaries. If so, increase the expansion pixels by half for both left and right boundaries. If not, expand all the expansion pixels to either the left or right boundary.

Images