Multi-scale image processing method, system, device and computer equipment

Abstract

The application relates to a multi-scale image processing method, system, device and computer equipment, and relates to the field of artificial intelligence.The method comprises the following steps: obtaining a corresponding product matrix according to an image matrix corresponding to an original image and an original convolution kernel; determining a plurality of starting point elements (c, d) in the image matrix for any size (U, V) of a hollow convolution, wherein 1<=c<=V, 1<=d<=U, U, V, c and d are all positive integers; reading a target element from the product matrix according to the size (U, V) for any starting point element (c, d), obtaining a first matrix, and determining the target position of each element in a feature image in the first matrix according to the product matrix and the size (U, V); and accumulating the elements located at the same target position in the first matrix corresponding to each starting point element (c, d), so as to obtain a feature image corresponding to the original image under the size (U, V). The method can improve the image processing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-scale image processing method, system, device and computer equipment. BACKGROUND

[0002] As a mainstream technology in the field of artificial intelligence, the convolutional neural network has been widely applied in the field of image processing and recognition. In actual image processing and recognition, different resolution feature extraction may be required for the same image. Therefore, the dilated convolution method is introduced.

[0003] In the traditional dilated convolution method, for a dilated convolution with a size of (U, V), U-1 zeros are supplemented between each two elements of each row in the original convolution kernel, and V-1 zeros are supplemented between each two elements of each column, so as to expand the original convolution kernel into a sparse convolution kernel. Then, the sparse convolution kernel is normally convolved with the image.

[0004] When the above image processing method is used for different resolution feature extraction of images, complete convolution processing is required for each size of dilated convolution, which leads to excessive calculation amount and resource consumption of image processing, reduces the image processing efficiency, and cannot perform multi-resolution feature extraction in actual application process. SUMMARY

[0005] Therefore, it is necessary to provide a multi-scale image processing method, device, computer equipment, computer readable storage medium and computer program product capable of reducing calculation amount and improving image processing efficiency in view of the above technical problems.

[0006] In a first aspect, the present application provides a multi-scale image processing method. The method comprises:

[0007] obtaining a corresponding product matrix according to the image matrix corresponding to the original image and the original convolution kernel;

[0008] For any size (U, V) of dilated convolution, a plurality of starting elements (c, d) are determined in the image matrix, wherein 1≤c≤V, 1≤d≤U, U, V, c and d are all positive integers;

[0009] For any starting element (c, d), a target element is read from the product matrix according to the size (U, V) to obtain a first matrix;

[0010] According to the product matrix and the size (U, V), the target positions of each element in the first matrix in the feature image are determined;

[0011] For the first matrix corresponding to each of the starting point elements (c, d), elements located at the same target position are accumulated to obtain the feature image corresponding to the original image under the size (U, V).

[0012] In one of the embodiments, the method for obtaining the product matrix according to the image matrix corresponding to the original image and the original convolution kernel comprises:

[0013] For the original convolution kernel, elements in each row of the original convolution kernel are read in row order to obtain a convolution row vector;

[0014] The convolution row vector is transposed to obtain a column vector corresponding to the original convolution kernel;

[0015] For any element in the image matrix corresponding to the original image, the element in the image matrix is multiplied by the column vector corresponding to the original convolution kernel to obtain a column vector in the product matrix;

[0016] The product matrix is obtained according to the column vector corresponding to each element in the image matrix.

[0017] In one of the embodiments, the method for reading a target element from the product matrix according to the size (U, V) to obtain the first matrix comprises:

[0018] Each element [x ij *k pq ] in the product matrix is determined, wherein x ij is any element in the image matrix, (i, j) represents the position information of x ij in the image matrix, k pq is any element in the original convolution kernel, and i, j, p, q are all positive integers.

[0019] For the image matrix, starting from the starting point element (c, d), each row is sampled at an interval of U-1 and each column is sampled at an interval of V-1 to obtain the position information (i', j') of each target element in the image matrix.

[0020] According to the position information, a target element [x i＇j＇ *k pq ] is read from the product matrix to obtain the first matrix.

[0021] In one of the embodiments, the method for determining the target position of each element in the first matrix in the feature image according to the product matrix and the size (U, V) comprises:

[0022] For any element [x i＇j＇ *kpq According to the x i＇j＇ The position information (i', j') in the image matrix, and the k pq The target position of the element in the feature image is determined by the position in the original convolution kernel and the size (U, V).

[0023] In one embodiment, the statement based on x i＇j＇ The position information (i', j') in the image matrix, and the k pq Determining the target location of the element in the feature image based on its position in the original convolution kernel and its size (U, V) includes:

[0024] According to the x i＇j＇ The row number i' in the image matrix, the k pq The number of rows p in the original convolution kernel and the size U are used to determine the number of rows of the element in the feature image.

[0025] According to the x i＇j＇ The column number j' in the image matrix, the k pq The number of columns of the element in the feature image is determined by the number of columns q in the original convolution kernel and the size V.

[0026] In one embodiment, the step of accumulating elements located at the same target position in the first matrix corresponding to each of the starting elements (c, d) to obtain the feature image of the original image at the corresponding size (U, V) includes:

[0027] For any of the target locations, the elements to be processed located at the target locations in each of the first matrices are accumulated to obtain the elements at the target locations;

[0028] Based on the elements of each target location, the feature image corresponding to the original image at the specified size (U, V) is constructed.

[0029] Secondly, this application also provides a multi-scale image processing system. The system includes: a pointwise product and storage unit, a parameter generation unit, and a multiplexing computation unit; wherein,

[0030] The pointwise product and storage unit is used to obtain the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel, and to store the product matrix.

[0031] The multiplexing calculation unit is configured to determine a plurality of starting point elements (c, d) in the image matrix for any size (U, V) of the atrous convolution, where 1≤c≤V and 1≤d≤U, and U, V, c and d are positive integers, and for any starting point element (c, d), read a target element from the product matrix stored in the point-by-point product storage unit according to the size (U, V) to obtain a first matrix;

[0032] The parameter generation unit is configured to determine target positions of elements in a feature image in the first matrix corresponding to any starting point element (c, d) according to the product matrix and the size (U, V).

[0033] The multiplexing calculation unit is further configured to read the target positions of the elements in the first matrix corresponding to each starting point element (c, d) from the parameter generation unit respectively, and perform accumulation processing on elements located at the same target position to obtain the feature image corresponding to the original image under the size (U, V).

[0034] In a third aspect, the present application further provides a multi-scale image processing device. The device comprises:

[0035] A product module is configured to obtain a corresponding product matrix according to an image matrix corresponding to an original image and an original convolution kernel.

[0036] A first determination module is configured to determine a plurality of starting point elements (c, d) in the image matrix for any size (U, V) of the atrous convolution, where 1≤c≤V and 1≤d≤U, and U, V, c and d are positive integers.

[0037] A reading module is configured to read a target element from the product matrix according to the size (U, V) for any starting point element (c, d) to obtain a first matrix.

[0038] A second determination module is configured to determine target positions of elements in a feature image in the first matrix according to the product matrix and the size (U, V).

[0039] An accumulation module is configured to perform accumulation processing on elements located at the same target position for the first matrix corresponding to each starting point element (c, d) to obtain the feature image corresponding to the original image under the size (U, V).

[0040] In one embodiment, the product module is further configured to:

[0041] For the original convolution kernel, read elements of each row in the original convolution kernel in row order to obtain a convolution row vector.

[0042] transposing the convolution row vector to obtain a column vector corresponding to the original convolution kernel;

[0043] For any element in the image matrix corresponding to the original image, the element in the image matrix is multiplied by the column vector corresponding to the original convolution kernel to obtain a column vector in the product matrix;

[0044] According to the column vector corresponding to each element in the image matrix, the product matrix is obtained.

[0045] In one of the embodiments, the reading module is further configured to:

[0046] determine each element [x ij *k pq ] in the product matrix, wherein x ij is any element in the image matrix, (i, j) represents the position information of x ij in the image matrix, k pq is any element in the original convolution kernel, and i, j, p, q are all positive integers.

[0047] For the image matrix, starting from the starting element (c, d), each row is sampled at an interval of U-1 and each column is sampled at an interval of V-1 to obtain the position information (i', j') corresponding to each target element in the image matrix.

[0048] According to the position information (i', j'), the target element [x i＇j＇ *k pq ] is read from the product matrix to obtain the first matrix.

[0049] In one of the embodiments, the second determining module is further configured to:

[0050] For any element [x i＇j＇ *k pq ] in the first matrix, according to the position information (i', j') of x i＇j＇ in the image matrix, the position of k pq in the original convolution kernel, and the size (U, V), the target position of the element in the feature image is determined.

[0051] In one of the embodiments, the second determining module is further configured to:

[0052] According to the number i' of rows of x i＇j＇ in the image matrix and the number of k pqa number of rows p in the original convolution kernel and the size U determine a number of rows of the element in the feature image;

[0053] according to the x i＇j＇ a number of columns j' in the image matrix, the k pq a number of columns q in the original convolution kernel and the size V determine a number of columns of the element in the feature image.

[0054] In one embodiment, the accumulation module is further configured to:

[0055] for any target position, elements in the first matrix located at the target position are accumulated to obtain an element of the target position;

[0056] according to the element of each target position, a feature image corresponding to the original image under the size (U, V) is constructed.

[0057] In a fourth aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0058] according to an image matrix corresponding to an original image and an original convolution kernel, a product matrix is obtained;

[0059] for any size (U, V) of the atrous convolution, a plurality of starting elements (c, d) are determined in the image matrix, where 1≤c≤V, 1≤d≤U, U, V, c, and d are all positive integers;

[0060] for any starting element (c, d), according to the size (U, V), a target element is read from the product matrix to obtain a first matrix;

[0061] according to the product matrix and the size (U, V), a target position of each element in the first matrix in a feature image is determined;

[0062] for the first matrix corresponding to each starting element (c, d), elements located at the same target position are accumulated to obtain a feature image corresponding to the original image under the size (U, V).

[0063] In a fifth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0064] according to an image matrix corresponding to an original image and an original convolution kernel, a product matrix is obtained;

[0065] For any dilated convolution size (U, V), multiple starting elements (c, d) are determined in the image matrix, where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers;

[0066] For any of the starting elements (c, d), the target element is read from the product matrix according to the dimensions (U, V) to obtain the first matrix;

[0067] Based on the product matrix and the dimensions (U, V), determine the target position of each element in the first matrix in the feature image;

[0068] For each of the starting point elements (c, d) corresponding to the first matrix, the elements located at the same target position are accumulated to obtain the feature image corresponding to the original image at the size (U, V).

[0069] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0070] Based on the image matrix corresponding to the original image and the original convolution kernel, the corresponding product matrix is ​​obtained;

[0071] For any dilated convolution size (U, V), multiple starting elements (c, d) are determined in the image matrix, where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers;

[0072] For any of the starting elements (c, d), the target element is read from the product matrix according to the dimensions (U, V) to obtain the first matrix;

[0073] Based on the product matrix and the dimensions (U, V), determine the target position of each element in the first matrix in the feature image;

[0074] For each of the starting point elements (c, d) corresponding to the first matrix, the elements located at the same target position are accumulated to obtain the feature image corresponding to the original image at the size (U, V).

[0075] The aforementioned multi-scale image processing methods, systems, devices, computer equipment, storage media, and computer program products obtain a corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel. For any dilated convolution size (U, V), multiple starting elements (c, d) are determined in the image matrix, where 1 ≤ c ≤ V, 1 ≤ d ≤ U, and U, V, c, and d are all positive integers. For any starting element (c, d), target elements are read from the product matrix according to the size (U, V) to obtain a first matrix. Based on the product matrix and the size (U, V), the target positions of each element in the first matrix in the feature image are determined. For each starting element (c, d) corresponding to the first matrix, elements located at the same target position are accumulated to obtain the feature image corresponding to the original image at the stated size (U, V). Based on the aforementioned multi-scale image processing methods, systems, devices, computer equipment, storage media, and computer program products, during multi-scale image processing, for any size (U, V), target elements can be read from the product matrix according to the size (U, V) and the corresponding starting elements (c, d) to form a first matrix. Then, the elements in the first matrix corresponding to each starting element (c, d) are accumulated to obtain the feature image at size (U, V). The product matrix, as a set of reusable data, allows for reuse during multi-scale image processing when performing dilated convolutions at each size. The corresponding elements can be directly read from the product matrix and simply accumulated, eliminating the need to perform a complete convolution operation at each size. This significantly reduces the computational load in multi-scale image processing, shortens the processing time, and improves image processing efficiency. Attached Figure Description

[0076] Figure 1 This is a flowchart illustrating a multi-scale image processing method in one embodiment;

[0077] Figure 2 This is a schematic diagram of convolution calculation in a conventional image processing method in one embodiment;

[0078] Figure 3 This is a flowchart illustrating a multi-scale image processing method in another embodiment;

[0079] Figure 4 This is a flowchart illustrating a multi-scale image processing method in another embodiment;

[0080] Figure 5 This is a flowchart illustrating a multi-scale image processing method in another embodiment;

[0081] Figure 6This is a schematic diagram of the calculation in a multi-scale image processing method in one embodiment;

[0082] Figure 7 This is a flowchart illustrating a multi-scale image processing method in another embodiment;

[0083] Figure 8 This is a schematic diagram of the structure of a multi-scale image processing system in one embodiment;

[0084] Figure 9 This is a schematic diagram of the workflow of the parameter generation unit in one embodiment;

[0085] Figure 10 This is a schematic diagram of the workflow of a multiplexed computing unit in one embodiment;

[0086] Figure 11 This is a schematic diagram of the structure of a multi-scale image processing device in one embodiment;

[0087] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0089] In one embodiment, such as Figure 1 As shown, a multi-scale image processing method is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0090] Step 102: Obtain the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel.

[0091] In this embodiment of the application, the original image is the image to be processed by multi-scale image processing. For example, in the process of multi-scale image processing such as image classification at different resolutions, the image to be classified at different resolutions is the original image.

[0092] For example, the elements of the image matrix corresponding to the original image can be multiplied with the elements of the original convolution kernel to obtain the corresponding product matrix. Each element in the image matrix corresponding to the original image corresponds to a pixel in the original image, and the size of the image matrix corresponds to the size of the original image. The number of rows in the image matrix represents the height of the original image (in pixels), and the number of columns in the image matrix represents the width of the original image (in pixels). For example, if the image matrix is ​​an I-row, J-column matrix, the total number of elements in the image matrix is ​​the product of I and J. Therefore, the total number of pixels in the original image corresponding to the image matrix is ​​the product of I and J.

[0093] In this embodiment, a product matrix can be obtained based on the image matrix corresponding to the original image and the original convolution kernel. This product matrix is ​​obtained by multiplying each element of the image matrix corresponding to the original image with each element of the original convolution kernel. In other words, any element in the product matrix is ​​the product of any element in the image matrix and any element in the original convolution kernel. For example, taking an image matrix X with I rows and J columns and an original convolution kernel K with P rows and Q columns as an example, any element in the image matrix can be represented by x. ij This means that any element in the original convolution kernel can be represented by k. pq If , then any element in the product matrix is ​​[x ij *k pq ].

[0094] Step 104: For any dilated convolution size (U, V), determine multiple starting elements (c, d) in the image matrix, where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers.

[0095] In this embodiment of the application, when processing the original image, different resolutions can correspond to different sizes of dilated convolutions. That is, by changing the size of the dilated convolution, the resolution of the original image during image processing can be changed. For example, when extracting features from the original image at a resolution of A1, the size (U1, V1) is used; when extracting features from the original image at a resolution of A2, the size (U2, V2) is used; and when extracting features from the original image at the highest resolution, the size (1, 1) is used.

[0096] In traditional methods, when processing the original image using convolutional holes of size (U, V), zeros are padded into the original convolutional kernel to obtain a sparse convolutional kernel. This sparse kernel is then convolved with the image matrix corresponding to the original kernel to obtain the feature image at size (U, V). Here, the zero elements in the sparse convolutional kernel convolve with the elements in the image matrix, resulting in a zero. Only the non-zero elements in the sparse convolutional kernel perform normal convolution operations with the elements in the image matrix. This convolution process can be viewed as first sampling the image matrix corresponding to the original image according to size (U, V), and then convolving the sampled matrix with the original convolutional kernel. However, in the above sampling process based on size (U, V), when U > 1 and V > 1, the matrix obtained from a single sampling of the image matrix can only contain a subset of the elements from the original image matrix.

[0097] In this embodiment of the application, in order to ensure that all elements in the image matrix can be processed when using dilated convolution of size (U, V) for image processing, a starting element (c, d) is set, 1≤c≤V, 1≤d≤U, where U, V, c, and d are all positive integers. The starting element (c, d) represents the starting position when sampling the image matrix. According to the size (U, V), multiple starting elements (c, d) are determined in the image matrix. Then, sampling can start from each starting element (c, d) as the starting position. All elements in the sampled matrix corresponding to each starting element (c, d) are all elements in the image matrix.

[0098] For example, taking image processing of a car image as an example, the image matrix corresponding to the car image is: Assuming the dilated convolution has dimensions (2, 3), where U = 2 and V = 3, then among the starting elements (c, d), 1 ≤ c ≤ 3 and 1 ≤ d ≤ 2, we can determine multiple starting elements: (1, 1), (1, 2), (2, 1), (2, 2), (3, 1), and (3, 2). The starting element (1, 1) is the element X. 11 The starting element (1, 2) is element X. 12 The starting element (2, 1) is element X. 21 The starting element (2, 2) is element X. 22 The starting element (3, 1) is element X. 31 The starting element (3, 2) is element X. 32 .

[0099] Step 106: For any starting element (c, d), read the target element from the product matrix according to the size (U, V) to obtain the first matrix.

[0100] In this embodiment, for multiple starting elements (c, d) under the aforementioned size, for any starting element, the target element is read from the product matrix according to the size (U, V) to obtain the first matrix. Taking the above example, there are 6 starting elements (1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2), then 6 first matrices can be obtained accordingly.

[0101] The traditional image processing procedure of dilated convolution can be viewed as follows: first, the image matrix is ​​sampled based on the starting element (c, d) and size (U, V), and then the sampled matrix is ​​convolved with the original convolution kernel. In the original definition of convolution, the image matrix is ​​represented by X, and each element in the image matrix is ​​represented by x(i, j); the convolution kernel is represented by K, and each element in the convolution kernel is represented by k(p, q); the feature image obtained after convolving the original image is represented by Y, and each element in the feature image is represented by y(u, v); the convolution formula is... As can be seen from the above convolution formula, the essence of convolution is to sum the product of any element in the image matrix located at the corresponding position and any element in the original convolution kernel.

[0102] Reference Figure 2 As shown, image processing is performed on an original image with a 3x3 matrix using a 3x3 convolution kernel. Figure 2 It contains 9 subgraphs, each of which is a 5x5 grid. Each grid can be used to represent the element position of the feature image after image processing. A blank grid indicates that there is no element at that position. Here, y(3,3) is the sum of the elements at position (3,3) in the 9 subgraphs.

[0103] Therefore, the process of convolving the sampled matrix with the original convolution kernel can also be transformed into reading the corresponding target elements from the product matrix and accumulating the target elements located at the corresponding positions.

[0104] Step 108: Determine the target position of each element in the first matrix in the feature image based on the product matrix and the size (U, V).

[0105] In this embodiment, the feature image is the image obtained after image processing from the original image. The target position of each element in the first matrix in the feature image is determined based on the product matrix and the size (U, V). For example, the feature image can be represented as a matrix, and the target position of an element can represent the row and column number of that element in the first matrix in the feature image.

[0106] Step 110: For the first matrix corresponding to each starting element (c, d), the elements located at the same target position are accumulated to obtain the feature image corresponding to the original image at size (U, V).

[0107] In this embodiment of the application, a size (U, V) corresponds to multiple starting elements (c, d), where each starting element (c, d) corresponds to a first matrix. Thus, a size (U, V) corresponds to multiple first matrices. For each first matrix corresponding to a starting element (c, d), the elements located at the same target position are accumulated to obtain the feature image under size (U, V).

[0108] After obtaining the feature image at one size (U, V), the above processing can be performed on dilated convolutions at other sizes, and finally, multiple feature images at various sizes can be obtained, which is the image processing result after processing the original image at different resolutions.

[0109] In the aforementioned multi-scale image processing method, during the multi-scale image processing, for any size (U, V), based on the size (U, V) and the corresponding starting element (c, d), the target element can be read from the product matrix to form a first matrix. Then, the elements in the first matrix corresponding to each starting element (c, d) are accumulated to obtain the feature image at size (U, V). The product matrix, as a set of reusable data, allows for reuse during multi-scale image processing when performing dilated convolutions at each size. The corresponding elements can be directly read from the product matrix and a simple accumulation process can be performed, eliminating the need to perform a complete convolution operation at each size. This significantly reduces the computational load in multi-scale image processing, shortens the processing time, and improves image processing efficiency.

[0110] In one embodiment, such as Figure 3 As shown, step 102 includes:

[0111] Step 302: For the original convolution kernel, read the elements of each row in the original convolution kernel in row order to obtain the convolution row vector.

[0112] In this embodiment, for the original convolution kernel, the elements of each row in the original convolution kernel are read sequentially in row order to obtain the convolution row vector. The row order means reading the elements of each row in the original convolution kernel in ascending order of the row number. That is, first, the elements of the first row in the original convolution kernel are read. After all the elements of the first row are read, the elements of the second row in the original convolution kernel are read. After all the elements of the second row are read, the elements of the third row are read, and so on, until all the elements of the last row of the original convolution kernel are read. When reading the elements in each row, the elements are also read according to the column number of the elements in the row.

[0113] For example, the original convolution kernel is When reading elements, first read the element in the first row: K. 11 K 12 K 13 Next, read the element from the second line: K. 21 K 22 K 23 Finally, read the element from the third line: K 31 K 32 K 33 The resulting convolution row vector is (K 11 K 12 K 13 K 21 K 22 K 23 K 31 K 32 K 33 ).

[0114] Step 304: Transpose the convolution row vector to obtain the column vector corresponding to the original convolution kernel.

[0115] In this embodiment, the convolution row vector is transposed. Transposition can convert between column vectors and row vectors. After transposing the convolution row vector, the column vector corresponding to the original convolution kernel can be obtained.

[0116] Step 306: For any element in the image matrix, multiply the element in the image matrix with the column vector corresponding to the original convolution kernel to obtain the column vector in the product matrix.

[0117] In this embodiment, for any element in the image matrix corresponding to the original image, the element in the image matrix is ​​multiplied by the column vector corresponding to the original convolution kernel to obtain the column vector in the product matrix. Then, the number of column vectors in the product matrix (i.e., the number of columns) is equal to the total number of elements in the image matrix, and the number of rows in the product matrix is ​​equal to the total number of elements in the original convolution kernel.

[0118] Step 308: Obtain the product matrix based on the column vectors corresponding to each element in the image matrix.

[0119] In this embodiment, after obtaining the column vectors corresponding to each element in the product matrix, the column vectors in the product matrix are concatenated to obtain the product matrix. During the concatenation process, the row numbers of the elements in each column vector of the product matrix are aligned one by one. For example, taking an image matrix as an example... The original convolution kernel is For example, we can first obtain the column vector (K) corresponding to the original convolution kernel. 11 K 12 K 21 K 22 ) T Then, for any element in the image matrix, such as element X 11 , let X 11 Multiplying the column vectors corresponding to the original convolution kernel yields the first column vector (X) of the product matrix. 11 *K 11 X 11 *K 12 X 11 *K 21 X 11 *K 22 ) T Repeating the above process, we can obtain the second column vector (X) of the product matrix. 12 *K 11 X 12 *K 12 X 12 *K 21 X 12 *K 22 ) T The third column vector of the product matrix (X) 21 *K 11 X 21 *K 12 X 21 *K 21 X 21 *K 22 ) T The fourth column vector of the product matrix (X) 22 *K 11 X 22 *K 12 X 22 *K 21 X 22 *K 22 ) T By concatenating the column vectors of the above product matrix, the product matrix can be obtained.

[0120] In this embodiment, each element in the image matrix corresponding to the original image is first multiplied with each element in the original convolution kernel, and the result of the multiplication operation is stored in the form of a multiplication matrix. The multiplication matrix is ​​stored as a set of reusable data. When performing image processing on each size (U, V) later, the corresponding elements can be directly read from the multiplication matrix and a simple accumulation process can be performed, which greatly reduces the amount of computation in the multi-scale image processing process, shortens the time of multi-scale image processing, and improves the efficiency of image processing.

[0121] In one embodiment, such as Figure 4 As shown, step 106 includes:

[0122] Step 402, determine each element [x] in the product matrix. ij *k pq ], where x ij Let (i, j) be any element in the image matrix, representing x. ij Position information in the image matrix, k pq Let i be any element in the original convolution kernel, where i, j, p, and q are all positive integers.

[0123] In this embodiment of the application, the elements [x] in the product matrix are first determined. ij *k pq The product matrix is ​​obtained by multiplying each element of the image matrix by each element of the original convolution kernel. Therefore, any element in the product matrix should be the product of any element of the image matrix and any element of the original convolution kernel. Thus, each element in the product matrix is ​​[x ij *k pq ], where x ij Let k be any element in the image matrix. pq Let (i, j) be any element in the original convolution kernel, representing x. ij Position information within the image matrix.

[0124] Step 404: For the image matrix, starting from the starting element (c, d), sample each row at interval U-1 and each column at interval V-1 to obtain the position information (i', j') of each target element in the image matrix.

[0125] In this embodiment of the application, for the image matrix, starting from the starting element (c, d), each row is sampled at an interval of U-1 and each column at an interval of V-1 to obtain the position information (i', j') of each target element in the image matrix. Here, one size corresponds to multiple starting elements (c, d). Starting from each starting element (c, d), sampling is performed at an interval of U-1 for each row and an interval of V-1 for each column. The obtained position information (i', j') includes the position information of all elements in the image matrix.

[0126] Using the example above, the image matrix is: The size is (2, 3), and the starting elements are (1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2). Starting from the element X corresponding to (1, 1) 11 Initially, each row is sampled at an interval of 1, and each column at an interval of 2, to obtain the position information (1,1) and (1,3) of each target element in the image matrix; from the element X corresponding to (1,2) 12 Initially, we can obtain the position information (1,2); from the element X corresponding to (2,1) 21 Initially, we can obtain the position information (2,1) and (2,3); from the element X corresponding to (2,2) 22 Initially, the position information (2,2) can be obtained; from the element X corresponding to (3,1) 31 Initially, we can obtain the position information (3,1) and (3,3); from the element X corresponding to (3,2) 32 Initially, the location information (3,2) can be obtained.

[0127] Step 406: Based on the position information (i', j'), read the target element [x] from the product matrix. i＇j＇ *k pq ], thus obtaining the first matrix.

[0128] In this embodiment, the target element [x] can be read from the product matrix based on the position information (i', j'). i＇j＇ *k pq ], thus obtaining the first matrix, where the target element [x] is read based on the position information (i', j'). i＇j＇ *k pq When grouping elements into sets, each starting element (c, d) corresponds to a first matrix. For example, taking the starting element (1, 1) as an example, its corresponding first matrix is ​​(X... 11 *k pq X 13 *k pq ).

[0129] In this embodiment, multiple starting elements (c, d) under a size (U, V) are used as sampling starting points. Sampling is performed according to the size (U, V) to ensure that the final sampled position information (i', j') includes the position information of all elements in the image matrix. This ensures that each part of the original image is processed during image processing at different resolutions, avoiding the phenomenon that some features in the original image cannot be processed after changing the resolution of image processing.

[0130] In one embodiment, step 108 includes:

[0131] For any element [x] in the first matrix i＇j＇ *k pq According to x i＇j＇ The positional information (i', j') and k in the image matrix pq The position and size (U, V) of the element in the original convolution kernel determine the target position of the element in the feature image.

[0132] In this embodiment of the application, for any element [x] in the first matrix... i＇j＇ *k pq According to x i＇j＇ The positional information (i', j') and k in the image matrix pq The position and size (U, V) of the element in the original convolution kernel can determine the target position of the element in the feature image, where the target position represents any element [x] in the first matrix. i＇j＇ *k pq Given the number of rows and columns in the feature image, after obtaining the target position of each element in the first matrix in the feature image, for multiple elements of the first matrix that are at the same target position, simply add these multiple elements together to obtain the element at that target position in the feature image.

[0133] In this embodiment, by determining the position of each element in the first matrix in the feature image, the elements at a certain target position in the feature image can be obtained by simply accumulating the elements located at the same position. That is, this application uses simple accumulation processing to replace a series of multiplication operations in traditional convolution calculation, which greatly reduces the amount of computation in multi-scale image processing and effectively improves the efficiency of image processing.

[0134] In one embodiment, such as Figure 5 As shown, the above is based on x i＇j＇ The positional information (i', j') and k in the image matrix pq The position and size (U, V) of the element in the original convolution kernel determine the target location of the element in the feature image, which may include:

[0135] Step 502, according to x i＇j＇ The row numbers i' and k in the image matrix pq The number of rows p and the size U in the original convolution kernel determine the number of rows of the element in the feature image;

[0136] Step 504, based on x i＇j＇ The column numbers j' and k in the image matrix pq The number of columns q and the size V in the original convolution kernel determine the number of columns of the element in the feature image.

[0137] In this embodiment of the application, x can be used as a basis. i＇j＇ The row numbers i' and k in the image matrix pq The number of rows p and the size U in the original convolution kernel are used to determine the number of rows of an element in the feature image, based on x. i＇j＇ The column numbers j' and k in the image matrix pq The number of columns q and the size V in the original convolution kernel are used to determine the number of columns of the element in the feature image. Then, based on the number of rows and columns of the element in the feature image, the target position of the element in the feature image can be determined.

[0138] For example, the process of determining the number of rows and columns of an element in a feature image can be referred to the following formula (a).

[0139] H (i＇j＇)(pq) =[(p-1)*U+i',(q-1)*V+j'] Formula (I)

[0140] Among them, H i＇j＇(pq) Represents the elements [x] in the product matrix i＇j＇ *k pq The target location in the feature image.

[0141] For example, when processing an original 3x3 image using a convolutional kernel of size (2, 2) and 3x3 columns, refer to... Figure 6 As shown, Figure 6 Each element in the expression can be represented by x(i',j')*k(p,q), or by [x... i＇j＇ *k pq ] indicates that for element [x 11 *k 11 ], where p = q = i' = j' = 1, then H (11)(11) = (1,1), then the element [x 11 *k 11 ] in the first row and first column of the feature image; for element [x 11 *k 12 ], where p = i' = j' = 1, q = 2, then H (11)(12) = (1,3), then the element [x11 *k 12 ] in the 1st row and 3rd column of the feature image; for element [x 13 *k 11 ], where p = q = i' = 1, j' = 3, then H (13)(11) = (1,3), then the element [x 13 *k 11 [In the first row and third column of the feature image.]

[0142] It should be noted that the process of determining the number of rows and columns of an element in the feature image described above is an example in this embodiment. This application embodiment does not specifically limit the process of determining the number of rows and columns of an element in the feature image.

[0143] In this embodiment, by determining the position of each element in the first matrix in the feature image, the elements at a certain target position in the feature image can be obtained by simply accumulating the elements located at the same position. That is, this application uses simple accumulation processing to replace a series of multiplication operations in traditional convolution calculation, which greatly reduces the amount of computation in multi-scale image processing and effectively improves the efficiency of image processing.

[0144] In one embodiment, refer to Figure 7 As shown, step 110 includes:

[0145] Step 702: For any target position, accumulate the elements to be processed in each first matrix that are located at the target position to obtain the element at the target position.

[0146] In this embodiment of the application, for any target location, the elements to be processed located at the target location in each of the first matrices are accumulated to obtain the element at the target location. Taking the above example, refer to... Figure 7 As shown, the element to be processed located at the target position in the first row and first column of the feature image only includes [x 11 *k 11 If ], then the element in the first row and first column of the feature image is [x 11 *k 11 The element to be processed located at the target position in the 1st row and 3rd column of the feature image includes [x] 11 *k 12 ] and [x 13 *k 11 If ], then the element in the 1st row and 3rd column of the feature image is [x 11 *k 12 ]+[x 13 *k 11 ].

[0147] Step 704: Based on the elements at each target location, construct the feature image corresponding to the original image at size (U, V).

[0148] In this embodiment, the target position of each element in the first matrix corresponding to each starting element (c, d) in the feature image must first be determined. Since sampling is performed starting from each starting element (c, d) according to size (U, V), it can be ensured that all elements in the image matrix can be processed subsequently. Therefore, it is necessary to determine the target position of each element in all the first matrices under a size (U, V). Then, based on the elements at each target position, the feature image corresponding to the original image under that size (U, V) can be constructed. That is, the feature image obtained after image processing of the original image at the resolution corresponding to size (U, V) is obtained.

[0149] In an exemplary embodiment, the image matrix corresponding to an original image is used as... The original convolution kernel is Taking a size of (2, 2) as an example, this example reads the elements in each row of the original convolution kernel in row order, that is, it first reads the elements in the first row: K 11 K 12 K 13 Next, read the element from the second line: K. 21 K 22 K 23 Finally, read the element from the third line: K 31 K 32 K 33 The resulting convolution row vector is (K 11 K 12 K 13 K 21 K 22 K 23 K 31 K 32 K 33 The convolutional row vectors are transposed to transform them into column vectors, which are the column vectors corresponding to the original convolutional kernel. Then, any element in the image matrix is ​​multiplied by the column vector corresponding to the original convolutional kernel to obtain multiple column vectors in the product matrix. These multiple column vectors are then concatenated to obtain the final product matrix.

[0150] For the size (2, 2), multiple starting elements (1, 1), (1, 2), (2, 1), and (2, 2) are determined. And for the above product matrix, each element [x] in the product matrix is ​​determined. ij *k pqThen, the image matrix is ​​sampled starting from different starting elements. Taking the starting element (1,1) as an example, sampling is performed with a row interval of 1 and a column interval of 1 to obtain the position information (1,1), (1,3), (3,1), (3,3). The position information corresponding to the starting element (1,2) is (1,2), (3,2), the position information corresponding to the starting element (2,1) is (2,1), (2,3), and the position information corresponding to the starting element (2,2) is (2,2).

[0151] Here, different starting elements correspond to a group. For the starting element (1, 1), the target element [x] can be read from the product matrix based on its corresponding position information. 11 *k pq ]、[x 13 *k pq ]、[x 31 *k pq ]、[x 33 *k pq ], to obtain the first matrix corresponding to the starting element (1, 1). Then, for any element [x] in the first matrix i＇j＇ *k pq According to x i＇j＇ The positional information (i', j') and k in the image matrix pq The position and size (U, V) of the element in the original convolution kernel determine the target position of the element in the feature image. The above is the processing procedure for a set. In practical applications, the above processing can be performed on multiple starting elements simultaneously to finally obtain the target position of each element in the first matrix corresponding to each starting element (c, d).

[0152] Next, for any target position, the elements to be processed in each of the first matrices located at the target position are accumulated to obtain the element at the target position. Taking the target position as row 1, column 1 as an example, the element located in row 1, column 1 is [x 11 *k 11 When the target position is row 1, column 3, the elements to be processed located in row 1, column 3 include [x] 11 *k 12 ] and [x 13 *k 11 If ], then the element in the 1st row and 3rd column of the feature image is [x 11 *k 12 ]+[x 13 *k 11 ].

[0153] After the above series of processing steps, the elements at each position of the feature image can be obtained. Based on the elements at each target position, the feature image corresponding to the original image at size (2,2) can be constructed.

[0154] The multi-scale image processing method of this application reduces the amount of computation in the image processing process, shortens the image processing time, and improves the image processing efficiency. It lays the foundation for the practical application of more size combinations in multi-scale image processing methods and improves the accuracy and effect of multi-resolution learning.

[0155] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0156] This application provides a multi-scale image processing system, referring to... Figure 8 As shown, the system includes: a pointwise product and storage unit, a parameter generation unit, and a multiplexed computation unit; wherein,

[0157] The pointwise product and storage unit is used to obtain the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel, and to store the product matrix.

[0158] The reused computing unit is used to determine multiple starting elements (c, d) in the image matrix for any dilated convolution size (U, V), where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers. For any starting element (c, d), the target element is read from the product matrix stored in the pointwise product and storage unit according to the size (U, V) to obtain the first matrix.

[0159] The parameter generation unit is used to determine the target position of each element in the feature image of the first matrix corresponding to any starting element (c, d) based on the product matrix and the size (U, V).

[0160] The reused computing unit is also used to read the target position of each element in the first matrix corresponding to each starting element (c, d) from the parameter generation unit, and to accumulate the elements located at the same target position to obtain the feature image corresponding to the original image at size (U, V).

[0161] In this embodiment, the pointwise product and storage unit can read the image matrix and the original convolution kernel of the original image from the shared storage, and obtain the corresponding product matrix based on the image matrix and the original convolution kernel of the original image. The product matrix is ​​stored in the pointwise product and storage unit as reusable data for other modules to read.

[0162] The multiplexing computation unit is used to determine multiple starting elements (c, d) in the image matrix for any dilated convolution size (U, V), where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers. For any starting element (c, d), the target element is read from the product matrix stored in the pointwise product and storage unit according to the size (U, V) to obtain the first matrix.

[0163] The parameter generation unit is used to determine the target position of each element in the feature image of the first matrix corresponding to any starting element (c, d) based on the product matrix and the size (U, V).

[0164] The reused computing unit is also used to read the target position of each element in the first matrix from the parameter generation unit for each starting element (c, d), and to accumulate the elements located at the same target position to obtain the feature image corresponding to the original image at size (U, V).

[0165] In this embodiment, the system's operation can also be as follows: the point-by-point multiplication and storage unit first reads the image matrix and the original convolution kernel, reshapes the original convolution kernel K to form the column vector corresponding to the original convolution kernel, and then multiplies each element x(i,j) in the image matrix with the column vector corresponding to the original convolution kernel to obtain I*J storage vectors G. ij =x(i,j)*1≤i≤I,1≤j≤J),K V This represents the column vector corresponding to the original convolution kernel, and the stored vector is stored for later reuse. This part of the stored vector G ij (1≤i≤I, 1≤j≤J) means the result of the multiplication operation will only be calculated once. Subsequent multiplications will not be repeated in multi-scale convolution learning, thus saving the computational cost of multi-scale image processing. The acquisition of each vector in this unit is independent and can be implemented using multiprocessing / threading / coroutines.

[0166] Next, the parameter generation unit generates indices, which represent the target positions of each element in the first matrix within the feature image. For multiple scales (U, V), it needs to generate indices for each (U, V) based on different starting elements (c, d). For a given scale (U, V), there are U*V sets of indexes. Assuming there are Z types of (U, V), then a total of Z*U*V sets of indexes need to be generated. By specifying the indexes, each x(i,j)*K can be specified without modifying / moving the stored results. V Add the elements at different positions in the text.

[0167] Furthermore, the reused computation unit is used to process each (U, V) index generated by the parameter generation unit. First, it is grouped according to different starting elements (c, d) (a total of U*V groups, each group corresponding to one value of the starting point (c, d). Then, the storage vector G within each group is selected for computation in the pointwise product and storage unit. i＇j＇ (can also be expressed as) Then, based on the index The corresponding elements at the same target position are added together to complete the image processing at one scale. Finally, the convolution result at the current (U, V) scale is reconstructed through reshaping. Assuming there are Z (U, V) scales, and the average number of iterations for each scale is O, the above process needs to be executed Z*O times. Each time, no additional multiplication in the convolution is required; only addition is needed, thus significantly reducing computational cost. In this unit, each vector acquisition is independent and can be implemented using multiprocessing / threading / coroutines.

[0168] The workflow of the parameter generation unit is as follows, refer to... Figure 9 As shown,

[0169] Step 902: For each multi-resolution scale (U,V), generate multiple starting elements (c,d);

[0170] Step 904: For each starting element (c,d), generate the position information (i',j') of the target element hit by the "enlarged convolution kernel" used for dilated convolution in the image matrix, where i'=e*U+c, j'=f*V+d, 0≤e≤(ceil(I / U)+P-1)-1, 0≤f≤(ceil(J / V)+Q-1)-1, which means rounding up, for example, ceil(1.5)=2.

[0171] Step 906, for each storage vector G i’j’ At this point, it can be named A corresponding index vector can be generated. Its elements are a two-dimensional array (corresponding to the vertical and horizontal positions of the output feature image Y), that is:

[0172] Where (p-1)*Q+q represents the element in its corresponding storage vector The position of the element in the feature image is (p-1)*U+i', which represents the row number of the element in the feature image, and (q-1)*V+j', which represents the column number of the element in the feature image.

[0173] The workflow of the reused computing unit is as follows, please refer to... Figure 10 As shown,

[0174] Step 1002: Select one scale (U,V) from multiple scales. There are Z candidate scales in total.

[0175] Step 1004: Select a starting element (c,d) (1≤c≤V,1≤d≤U) in the scale (U,V), and there are a total of U*V starting elements.

[0176] Step 1006: Generate the initialization result image Y at the current scale (U,V) and current starting point (c,d). (U,V,c,d) The results of each group calculation will be accumulated into the corresponding position of the initial result image according to the position information. When all starting elements have been processed, the final feature image is obtained.

[0177] Step 1008: Select the stored vectors at the current scale (U,V) and current starting point (c,d). and the corresponding index vector Calculate the corresponding result vector Y (U,V,c,d) The storage vectors can be grouped according to their starting points (c, d), i.e. superscript " (c,d) During calculation, the corresponding index will be used. The elements in the image are accumulated at the target position of the corresponding feature image, and image processing is achieved through superposition.

[0178] Step 1010: Determine whether the image processing at the current scale (U,V) and current starting point (c,d) is complete. If not, return to step 1108; if complete, proceed to step 1012.

[0179] The complete pseudocode for steps 1008 and 1010 is as follows:

[0180]

[0181] The grouping range is i'=e*U+c, j'=f*V+d, 0≤e≤(ceil(I / U)+P-1)-1, 0≤f≤(ceil(J / V)+Q-1)-1, where ceil represents rounding up; [r] indicates The element located in the r-th dimension; [r]1 and [r]2 indicates The first and second elements in the r-th dimension ( The r-th dimension is a 2-dimensional array, representing the row and column numbers of the element in the feature image, respectively. Y represents (U,V,c,d) The position of an element (i.e., the target position).

[0182] Step 1012: Determine whether all starting elements (c,d) corresponding to this scale (U,V) have been calculated. If so, proceed to step 1014; otherwise, proceed to step 1004 and recalculate the starting elements that have not been calculated under this scale.

[0183] Step 1014: For each Y (U,V,c,d) (1≤c≤U, 1≤d≤V) are reshaped. That is, the non-zero values ​​are selected and filled into the feature image Y corresponding to this scale. (U,V) At the corresponding position, the feature image at this scale (U,V) is obtained.

[0184] Step 1016: If all scales (U, V) have been calculated and the corresponding feature images Y are obtained... (U,V) If the result is positive, the multi-scale image processing ends; otherwise, proceed to step 1002 and select other scales for recalculation.

[0185] Based on the aforementioned multi-scale image processing system, during multi-scale image processing, for any size (U, V), the target element can be read from the product matrix according to the size (U, V) and the corresponding starting element (c, d) to form a first matrix. Then, the elements in the first matrix corresponding to each starting element (c, d) are accumulated to obtain the feature image at size (U, V). The product matrix, as a set of reusable data, allows for reuse during dilated convolution at each size in multi-scale image processing. The corresponding elements can be directly read from the product matrix and simply accumulated, eliminating the need to perform a complete convolution operation at each size. This significantly reduces the computational load in multi-scale image processing, shortens the processing time, and improves image processing efficiency.

[0186] Based on the same inventive concept, this application also provides a multi-scale image processing apparatus for implementing the multi-scale image processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the multi-scale image processing apparatus provided below can be found in the limitations of the multi-scale image processing method described above, and will not be repeated here.

[0187] In one embodiment, such as Figure 11 As shown, a multi-scale image processing apparatus is provided, comprising: a product module 1102, a first determination module 1104, a reading module 1106, a second determination module 1108, and an accumulation module 1110, wherein:

[0188] The product module 1102 is used to obtain the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel;

[0189] The first determining module 1104 is used to determine multiple starting elements (c, d) in the image matrix for any dilated convolution size (U, V), where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers.

[0190] The reading module 1106 is used to read the target element from the product matrix according to the size (U, V) for any starting element (c, d) to obtain the first matrix;

[0191] The second determining module 1108 is used to determine the target position of each element in the first matrix in the feature image based on the product matrix and the size (U, V);

[0192] The accumulation module 1110 is used to accumulate the elements located at the same target position in the first matrix corresponding to each starting element (c, d) to obtain the feature image corresponding to the original image at size (U, V).

[0193] In one embodiment, the product module 1102 is further configured to:

[0194] For the original convolution kernel, read the elements of each row in the original convolution kernel in row order to obtain the convolution row vector;

[0195] Transpose the convolution row vector to obtain the column vector corresponding to the original convolution kernel;

[0196] For any element in the image matrix corresponding to the original image, multiply the element in the image matrix with the column vector corresponding to the original convolution kernel to obtain the column vector in the product matrix;

[0197] The product matrix is ​​obtained by using the column vectors corresponding to each element in the image matrix.

[0198] In one embodiment, the reading module 1106 is further configured to:

[0199] Determine the elements [x] in the product matrix ij *k pq ], where x ij Let (i, j) be any element in the image matrix, representing x. ij Position information in the image matrix, kpq Let i be any element in the original convolution kernel, where i, j, p, and q are all positive integers;

[0200] For the image matrix, starting from the starting element (c, d), each row is sampled at an interval of U-1 and each column at an interval of V-1 to obtain the position information (i', j') of each target element in the image matrix;

[0201] Based on the position information (i', j'), read the target element [x] from the product matrix. i＇j＇ *k pq ], thus obtaining the first matrix.

[0202] In one embodiment, the second determining module 1108 is further configured to:

[0203] For any element [x] in the first matrix i＇j＇ *k pq According to x i＇j＇ The positional information (i', j') and k in the image matrix pq The position and size (U, V) of the element in the original convolution kernel determine the feature location of the element in the feature image.

[0204] In one embodiment, the second determining module 1108 is further configured to:

[0205] According to x i＇j＇ The row numbers i' and k in the image matrix pq The number of rows p and the size U in the original convolution kernel determine the number of rows of the element in the feature image;

[0206] According to x i＇j＇ The column numbers j' and k in the image matrix pq The number of columns q and the size V in the original convolution kernel determine the number of columns of the element in the feature image.

[0207] In one embodiment, the accumulation module 1110 is further configured to:

[0208] For any target location, the elements to be processed located at the target location in each of the first matrices are accumulated to obtain the element at the target location;

[0209] Based on the elements at each target location, a feature image corresponding to the original image at size (U, V) is constructed.

[0210] Each module in the aforementioned multi-scale image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0211] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 12 As shown. The computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0212] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0213] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0214] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0215] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0216] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0217] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0218] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0219] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A multi-scale image processing method, characterized in that, The method includes: Based on the image matrix corresponding to the original image and the original convolution kernel, the corresponding product matrix is ​​obtained; For any dilated convolution size (U, V), multiple starting elements (c, d) are determined in the image matrix, where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers; For any of the starting elements (c, d), the target element is read from the product matrix according to the dimensions (U, V) to obtain the first matrix; Based on the product matrix and the dimensions (U, V), determine the target position of each element in the first matrix in the feature image; For each of the starting point elements (c, d) corresponding to the first matrix, the elements located at the same target position are accumulated to obtain the feature image corresponding to the original image at the size (U, V); The step of reading the target element from the product matrix according to the dimensions (U, V) to obtain the first matrix includes: Determine each element [x] in the product matrix ij k pq ], where x ij Let (i, j) be any element in the image matrix, representing x. ij Position information in the image matrix, k pq Let i, j, p, and q be any element in the original convolution kernel, where i, j, p, and q are all positive integers. For the image matrix, starting from the starting element (c, d), each row is sampled at an interval of U-1 and each column at an interval of V-1 to obtain the position information (i', j') of each target element in the image matrix. Based on the position information (i', j'), the target element [x] is read from the product matrix. i＇j＇ k pq ], thus obtaining the first matrix; Determining the target position of each element in the first matrix in the feature image based on the product matrix and the size (U, V) includes: For any element [x] in the first matrix i＇j＇ k pq According to the x i＇j＇ The positional information (i', j') in the image matrix, and the k pq The target position of the element in the feature image is determined by the position in the original convolutional kernel and the size (U, V); According to x i＇j＇ The positional information (i', j') in the image matrix, and the k pq Determining the target location of the element in the feature image based on its position in the original convolutional kernel and its dimensions (U, V) includes: According to the x i＇j＇ The row number i' in the image matrix, the k pq The number of rows p in the original convolution kernel and the size U are used to determine the number of rows of the element in the feature image. According to the x i＇j＇ The column number j' in the image matrix, the k pq The number of columns of the element in the feature image is determined by the number of columns q in the original convolution kernel and the size V.

2. The method according to claim 1, characterized in that, The step of obtaining the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel includes: For the original convolution kernel, the elements of each row in the original convolution kernel are read sequentially in row order to obtain the convolution row vector; The convolution row vector is transposed to obtain the column vector corresponding to the original convolution kernel; For any element in the image matrix corresponding to the original image, the element in the image matrix is ​​multiplied by the column vector corresponding to the original convolution kernel to obtain the column vector in the product matrix; The product matrix is ​​obtained based on the column vectors corresponding to each element in the image matrix.

3. The method according to claim 1, characterized in that, The step of accumulating the elements located at the same target position in the first matrix corresponding to each of the starting elements (c, d) to obtain the feature image of the original image at the corresponding size (U, V) includes: For any of the target locations, the elements to be processed located at the target locations in each of the first matrices are accumulated to obtain the elements at the target locations; Based on the elements of each target location, the feature image corresponding to the original image at the specified size (U, V) is constructed.

4. A multi-scale image processing system, characterized in that, The system includes: a pointwise product and storage unit, a parameter generation unit, and a multiplexed computation unit; wherein... The pointwise product and storage unit is used to obtain the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel, and to store the product matrix. The multiplexing computation unit is used to determine multiple starting elements (c, d) in the image matrix for any dilated convolution size (U, V), where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers. For any starting element (c, d), the unit reads the target element from the product matrix stored in the pointwise product and storage unit according to the size (U, V) to obtain a first matrix. The parameter generation unit is used to determine the target position of each element in the first matrix in the feature image based on the product matrix and the size (U, V) for any of the starting elements (c, d) corresponding to the first matrix. The reuse calculation unit is further configured to read the target position of each element in the first matrix from the parameter generation unit for each of the first matrices corresponding to the starting elements (c, d), and accumulate the elements located at the same target position to obtain the feature image corresponding to the original image at the size (U, V). The reused computation unit is specifically used to determine each element [x] in the product matrix. ij k pq ], where x ij Let (i, j) be any element in the image matrix, representing x. ij Position information in the image matrix, k pq Let i, j, p, and q be any element in the original convolution kernel, where i, j, p, and q are all positive integers. For the image matrix, starting from the starting element (c, d), each row is sampled at an interval of U-1 and each column at an interval of V-1 to obtain the position information (i', j') of each target element in the image matrix. Based on the position information (i', j'), the target element [x] is read from the product matrix. i＇j＇ k pq The parameter generation unit is specifically used to generate the first matrix by generating any element [x] in the first matrix. i＇j＇ k pq According to the x i＇j＇ The positional information (i', j') in the image matrix, and the k pq The target position of the element in the feature image is determined by the position in the original convolutional kernel and the size (U, V); The parameter generation unit is specifically used to generate parameters based on x. i＇j＇ The row number i' in the image matrix, the k pq The number of rows p in the original convolution kernel and the size U are used to determine the number of rows of the element in the feature image. According to the x i＇j＇ The column number j' in the image matrix, the k pq The number of columns of the element in the feature image is determined by the number of columns q in the original convolution kernel and the size V.

5. A multi-scale image processing apparatus, characterized in that, The device includes: The product module is used to obtain the corresponding product matrix based on the image matrix corresponding to the original image and the original convolution kernel; The first determining module is used to determine multiple starting elements (c, d) in the image matrix for any dilated convolution size (U, V), where 1≤c≤V, 1≤d≤U, and U, V, c, and d are all positive integers; The reading module is used to read the target element from the product matrix according to the size (U, V) for any of the starting elements (c, d) to obtain the first matrix; The second determining module is used to determine the target position of each element in the first matrix in the feature image based on the product matrix and the size (U, V); The accumulation module is used to accumulate the elements located at the same target position in the first matrix corresponding to each of the starting elements (c, d) to obtain the feature image corresponding to the original image at the size (U, V). The reading module is specifically used to determine each element [x] in the product matrix. ij k pq ], where x ij Let (i, j) be any element in the image matrix, representing x. ij Position information in the image matrix, k pq Let i, j, p, and q be any element in the original convolution kernel, where i, j, p, and q are all positive integers. For the image matrix, starting from the starting element (c, d), each row is sampled at an interval of U-1 and each column at an interval of V-1 to obtain the position information (i', j') of each target element in the image matrix. Based on the position information (i', j'), the target element [x] is read from the product matrix. i＇j＇ k pq ], thus obtaining the first matrix; The second determining module is specifically used for determining any element [x] in the first matrix. i＇j＇ k pq According to the x i＇j＇ The positional information (i', j') in the image matrix, and the k pq The target position of the element in the feature image is determined by the position in the original convolutional kernel and the size (U, V); The second determining module is specifically used to determine based on the x i＇j＇ The row number i' in the image matrix, the k pq The number of rows p in the original convolution kernel and the size U are used to determine the number of rows of the element in the feature image. According to the x i＇j＇ The column number j' in the image matrix, the k pq The number of columns of the element in the feature image is determined by the number of columns q in the original convolution kernel and the size V.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images