Acceleration system for geometric transformation of digital image

Abstract

The invention provides an acceleration system for geometric transformation of a digital image, which is applied to the field of image processing, and comprises a processor and an accelerator, the processor is used for determining a target transformation matrix, the target transformation matrix is used for geometric transformation between the first original image and the first target image and geometric transformation between the second original image and the second target image; determining each coordinate point on the first target image, wherein each coordinate point on the second target image is the same as each coordinate point on the first target image; the accelerator is used for determining an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image; and determining a first target image and a second target image according to the original coordinate position matrix, the pixel value of each coordinate point on the first original image and the pixel value of each coordinate point on the second original image. The acceleration system can be used for performing geometric transformation on a plurality of digital images at the same time, and the operation efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to an acceleration system for geometric transformation of digital images. Background Art

[0002] Geometric transformations (such as affine transformation, projective transformation) are common algorithms in image processing and are widely used in the field of computer vision, such as in face recognition, image scaling, image rotation, image registration and correction, etc.

[0003] Currently, when using a computing device to perform geometric transformation processing on an image, it is pixel-by-pixel operation in the underlying logic of the computing device, which is also called vector operation (operations such as addition, subtraction, multiplication, and division) in the industry, and the operation efficiency is low. Summary of the Invention

[0004] This application provides an acceleration system for geometric transformation of digital images. The acceleration system performs geometric transformation on multiple digital images simultaneously, which improves the operation efficiency compared with performing geometric transformation pixel by pixel on a single image.

[0005] In a first aspect, this application provides an acceleration system for geometric transformation of digital images. The digital images include a first original image and a second original image. The acceleration system includes a processor and an accelerator.

[0006] The processor is configured to:

[0007] Determine a target transformation matrix, which is used for geometric transformation between the first original image and the first target image, and geometric transformation between the second original image and the second target image, where the first target image and the second target image have the same size;

[0008] Determine each coordinate point on the first target image, and each coordinate point on the second target image is the same as each coordinate point on the first target image;

[0009] The accelerator is configured to:

[0010] According to the target transformation matrix and each coordinate point on the first target image, determine an original coordinate position matrix, where the original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image;

[0011] According to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

[0012] The solution of this application is described by taking a digital image including a first original image and a second original image as an example. A target transformation matrix is calculated. In the underlying logic of the computing device, the target transformation matrix is used to perform geometric transformation operations on the first original image and the second original image simultaneously. Compared with performing geometric transformation operations on the original images one by one, the computing efficiency is improved. It should be noted that the digital image may include a greater number of images. In practical applications, many images can be processed in batches. Each batch includes multiple images. First, calculate the target transformation matrix corresponding to the images in one batch, and use the target transformation matrix of this batch to perform geometric transformation operations on the multiple images in this batch simultaneously. Compared with performing geometric transformation operations on the images one by one, the computing efficiency is improved. In addition, performing geometric transformation on a batch of images simultaneously involves matrix or vector operations. An accelerator is more suitable for matrix or vector operations. Using the accelerator for matrix or vector operations enables the processor and the accelerator to play to their respective strengths. For the geometric transformation scenario of a large number of images, the operation time is shortened and the operation efficiency is further improved.

[0013] Based on the first aspect, in a possible implementation, the accelerator is used for:

[0014] According to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, determine the pixel matrix of adjacent coordinate points. The pixel matrix of adjacent coordinate points includes the pixel values of the adjacent coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image, and the matrix composed of the pixel values of the adjacent coordinate points of the coordinate positions of each coordinate point on the second target image on the second original image;

[0015] Determine the interpolation weight matrix, where the interpolation weight matrix includes the weights of each pixel value in the pixel matrix of adjacent coordinate points;

[0016] According to the pixel matrix of adjacent coordinate points and the interpolation weight matrix, determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

[0017] Based on the first aspect, in a possible implementation, the accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit, where,

[0018] The matrix multiplication acceleration unit is used to determine the original coordinate position matrix and determine the interpolation weight matrix;

[0019] The vector operation acceleration unit is used to determine the pixel matrix of adjacent coordinate points, and to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of adjacent coordinate points and the interpolation weight matrix, so as to obtain the first target image and the second target image.

[0020] The accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit. The matrix multiplication acceleration unit is a parallel operation unit for performing matrix cross multiplication operations, and the vector operation acceleration unit is a parallel operation unit for performing vector operations. Operations involving matrix cross multiplication are executed by the matrix multiplication acceleration unit, and operations involving vectors are executed by the vector operation acceleration unit. By reasonably utilizing the matrix multiplication acceleration unit and the vector operation acceleration unit in the accelerator, the matrix multiplication acceleration unit and the vector operation acceleration unit can each play their own roles and give full play to their strengths, thus improving the operation efficiency.

[0021] Based on the first aspect, in a possible implementation manner, the processor is used to:

[0022] Determine a first transformation matrix, where the first transformation matrix is used for geometric transformation between the first original image and the first target image, and the dimension of the first transformation matrix is k*3;

[0023] Determine a second transformation matrix, where the second transformation matrix is used for geometric transformation between the second original image and the second target image, and the dimension of the second transformation matrix is k*3;

[0024] Stack the first transformation matrix and the second transformation matrix in the row dimension to obtain a target transformation matrix, where the dimension of the target transformation matrix is 2k*3.

[0025] It can be understood that here the first transformation matrix and the second transformation matrix are used as examples for description. When a batch includes multiple images, multiple images correspond to multiple transformation matrices, and a target transformation matrix is determined according to the multiple transformation matrices. The target transformation matrix is used for geometric transformation of multiple images in a batch simultaneously.

[0026] Based on the first aspect, in a possible implementation manner, the dimension of the matrix composed of each coordinate point on the reference image is 3*m, and the dimension of the original coordinate position matrix is 2k*m.

[0027] Based on the first aspect, in a possible implementation manner, the geometric transformation includes one of affine transformation and perspective transformation.

[0028] In a second aspect, the present application provides an acceleration method for geometric transformation of digital images. The digital images include a first original image and a second original image. The method includes:

[0029] The accelerator obtains a target transformation matrix, which is used for geometric transformation between a first original image and a first target image, and geometric transformation between a second original image and a second target image, where the first target image and the second target image have the same size;

[0030] Obtain each coordinate point on the first target image, and each coordinate point on the second target image is the same as each coordinate point on the first target image;

[0031] According to the target transformation matrix and each coordinate point on the first target image, determine the original coordinate position matrix, which includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image;

[0032] According to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

[0033] The solution of this application is introduced by taking the target transformation matrix used for geometric transformation between the first original image and the first target image, and geometric transformation between the second original image and the second target image as an example. In practical applications, the target transformation matrix can be used for geometric transformation of multiple images. Performing geometric transformation operations on multiple images simultaneously using the target transformation matrix improves the calculation efficiency compared to performing geometric transformation operations on the original images one by one.

[0034] Based on the second aspect, in a possible implementation manner, according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, determining the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image, and obtaining the first target image and the second target image, includes:

[0035] According to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, determine the pixel matrix of adjacent coordinate points, where the pixel matrix of adjacent coordinate points includes the pixel values of the adjacent coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image, and the matrix composed of the pixel values of the adjacent coordinate points of the coordinate positions of each coordinate point on the first target image on the second original image;

[0036] Determine the interpolation weight matrix, where the interpolation weight matrix includes the weights of each pixel value in the pixel matrix of adjacent coordinate points;

[0037] Determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of adjacent coordinate points and the interpolation weight matrix, and obtain the first target image and the second target image.

[0038] Based on the second aspect, in a possible implementation, the geometric transformation includes one of an affine transformation and a projective transformation.

[0039] Based on the second aspect, before the accelerator obtains the target transformation matrix, the method further includes:

[0040] The processor determines the target transformation matrix;

[0041] The processor sends the target transformation matrix to the accelerator.

[0042] Based on the second aspect, in a possible implementation, the processor determines the target transformation matrix including:

[0043] The processor determines the first transformation matrix, which is used for the geometric transformation between the first original image and the first target image, and the dimension of the first transformation matrix is k*3;

[0044] Determine the second transformation matrix, which is used for the geometric transformation between the second original image and the second target image, and the dimension of the second transformation matrix is k*3;

[0045] Superimpose the first transformation matrix and the second transformation matrix in the row dimension to obtain the target transformation matrix, and the dimension of the target transformation matrix is 2k*3.

[0046] Based on the second aspect, in a possible implementation, the dimension of the matrix formed by each coordinate point on the reference image is 3*m, and the dimension of the original coordinate position matrix is 2k*m.

[0047] In a third aspect, the present application provides an accelerator, including:

[0048] An acquisition unit, configured to acquire a target transformation matrix, where the target transformation matrix is used for the geometric transformation between the first original image and the first target image, and the geometric transformation between the second original image and the second target image, where the first target image and the second target image have the same size;

[0049] The acquisition unit is further configured to acquire each coordinate point on the first target image, and each coordinate point on the second target image is the same as each coordinate point on the first target image;

[0050] The matrix multiplication acceleration unit is used to determine an original coordinate position matrix according to a target transformation matrix and each coordinate point on a first target image. The original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on a first original image and the coordinate positions of each coordinate point on a second target image on a second original image;

[0051] The matrix multiplication acceleration unit and the vector operation acceleration unit are cooperatively used to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, and obtain the first target image and the second target image.

[0052] Based on the third aspect, in a possible implementation manner,

[0053] The vector operation acceleration unit is used to determine a pixel matrix of neighboring coordinate points according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image. The pixel matrix of neighboring coordinate points includes the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image, and a matrix composed of the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the second target image on the second original image;

[0054] The matrix multiplication acceleration unit is used to determine an interpolation weight matrix, and the interpolation weight matrix includes the weights occupied by each pixel value in the pixel matrix of neighboring coordinate points;

[0055] The vector operation acceleration unit is used to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of neighboring coordinate points and the interpolation weight matrix, and obtain the first target image and the second target image.

[0056] Based on the third aspect, in a possible implementation manner, the geometric transformation includes one of an affine transformation and a projective transformation.

[0057] Each functional module of the third aspect is used to implement the method described on the accelerator side in the second aspect above, and the method described on the accelerator side in any possible implementation manner of the second aspect above.

[0058] In a fourth aspect, the present application provides a host, including:

[0059] A processor, configured to determine a target transformation matrix, where the target transformation matrix is used for geometric transformation between the first original image and the first target image, and geometric transformation between the second original image and the second target image, where the first target image and the second target image have the same size;

[0060] The processor is further configured to send the target transformation matrix to the accelerator, so that the accelerator performs geometric transformation on the first original image and the second original image according to the target transformation matrix.

[0061] Based on the fourth aspect, in a possible implementation, the processor is configured to:

[0062] Determine a first transformation matrix for geometric transformation between the first original image and the first target image, where the dimension of the first transformation matrix is k*3;

[0063] Determine a second transformation matrix for geometric transformation between the second original image and the second target image, where the dimension of the second transformation matrix is k*3;

[0064] Stack the first transformation matrix and the second transformation matrix in the row dimension to obtain a target transformation matrix, where the dimension of the target transformation matrix is 2k*3.

[0065] Based on the fourth aspect, in a possible implementation, the dimension of the matrix formed by each coordinate point on the reference image is 3*m, and the dimension of the original coordinate position matrix is 2k*m.

[0066] Each functional module of the fourth aspect is configured to implement the method described on the processor side in the second aspect above, and the method described on the processor side in any possible implementation of the second aspect above.

[0067] In a fifth aspect, the present application provides a host, including a processor and a memory. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the method described on the processor side in the second aspect above, and the method described on the processor side in any possible implementation of the second aspect above.

[0068] In a sixth aspect, the present application provides an accelerator, including a processor and a memory. The memory is used to store instructions, and the processor is used to execute the instructions stored in the memory to implement the method described on the accelerator side in the second aspect above, and the method described on the accelerator side in any possible implementation of the second aspect above.

[0069] In a seventh aspect, the present application provides a computer storage medium, including program instructions. When the program instructions are executed by a device with computing functions, the device with computing functions is caused to execute the method described by the accelerator and / or the processor in the second aspect above, and the method described by the accelerator and / or the processor in any possible implementation of the second aspect above.

[0070] In an eighth aspect, the present application provides a computer program product including program instructions which, when executed by a computing device, cause the computing device to execute the methods described for the accelerator and / or the processor in the second aspect above, and the methods described for the accelerator and / or the processor in any possible implementation manner of the second aspect above.

[0071] In a ninth aspect, the present application provides a chip that can be used to implement the method described for the accelerator side in the second aspect or any possible implementation manner of the second aspect. Description of the Drawings

[0072] Figure 1A Schematic diagram of the reference image given for the present application;

[0073] Figure 1B Schematic diagram of the image to be processed given for the present application;

[0074] Figure 2 Schematic flowchart of an acceleration method for geometric transformation of digital images provided by the present application;

[0075] Figure 3 An example diagram provided by the present application;

[0076] Figure 4 Schematic flowchart of a method for calculating pixel values at non-integer coordinate positions by the bilinear interpolation method provided by the present application;

[0077] Figure 5 Schematic diagram of the acceleration system structure for geometric transformation of digital images provided by the present application;

[0078] Figure 6 Schematic diagram of the structure of an accelerator provided by the present application;

[0079] Figure 7 Schematic diagram of the structure of a computing device provided by the present application. Detailed Embodiments

[0080] Introduce the technical terms involved in the method embodiments of the present application.

[0081] The vector operation acceleration unit refers to a parallel computing unit specialized for performing vector operations.

[0082] The matrix multiplication acceleration unit refers to a parallel computing unit that can only be used for performing matrix cross multiplication operations.

[0083] The core computing power resource of current mainstream artificial intelligence (AI) accelerators is the matrix multiplication acceleration unit, and the computing power resource of the matrix multiplication acceleration unit accounts for most or even the vast majority of the resources of the AI accelerator.

[0084] Many applications can be implemented through matrix cross multiplication operations, or a part of the operations in the application can be implemented through matrix cross multiplication, but none of them are implemented through the matrix multiplication acceleration unit. Instead, they are implemented through vector operations using vector operation acceleration units, or through other operation forms. This results in the core computing power resources in the accelerator not being reasonably utilized.

[0085] This application provides a method and system for geometric transformation of multiple images based on a reference image. Before introducing the method and system provided by this application, the application scenarios involved in the solution of this application will be introduced first.

[0086] The solution of this application is applicable to the application scenario of geometric transformation of multiple images according to a given reference image, where the geometric transformation can be, for example, an affine transformation or a projective transformation. The affine transformation includes translation, rotation, scaling, shearing, and reflection. For example, in a target recognition application scenario, the target recognition can be face recognition, license plate recognition, etc. It is necessary to first perform geometric transformation on the collected images to transform them into images of the same size, shape, and unified format as the reference image, and then further process the unified format images to achieve target recognition.

[0087] For example, referring to Figure 1A and Figure 1B shown, Figure 1A is a schematic diagram of the reference image given in this application, Figure 1B Images 1, 2, and 3 in are images to be processed. Among them, Images 1, 2, and 3 are images with different sizes and different poses of the targets in the images. It is required to perform geometric transformation on Images 1, 2, and 3 according to the given reference image to obtain target images with the same size, shape, and image placement pose as the reference image. The target images can be further processed to obtain images that meet the application requirements.

[0088] It should be noted that the different sizes of two images mean that the number of rows, or the number of columns, or both the number of rows and columns that make up these two images are different. The different poses of the targets in two images mean that the pose positions of the targets in the two images are different. For example, the pose of the person in Image 1 is different from the pose of the person in Image 2, and the poses of the person in Image 1 and the person in Image 2 are respectively different from the pose of the person in Image 3.

[0089] In Figure 1A and Figure 1BIn the application scenario shown, the affine transformation can be performed on Image 1, Image 2, and Image 3 according to the reference image to obtain target images that are the same size, shape, and image pose as the reference image, such as Figure 1B the target Image 1, target Image 2, and target Image 3 in

[0090] Optionally, the faces in Image 1, the faces in Image 2, and the faces in Image 3 can also be located on one image, that is, one image includes multiple faces, and any one or more of the sizes, shapes, and poses of the multiple faces can be different. Geometric transformations are performed on the multiple faces on this one image respectively to obtain target images that are the same size, shape, and pose as the reference image.

[0091] Next, an acceleration method for geometric transformation of digital images provided by the method of this application will be introduced. See Figure 2 , Figure 2 is a schematic flowchart of an acceleration method for geometric transformation of digital images provided by this application. This method is applied to an acceleration system for geometric transformation of digital images. The acceleration system includes a processor and an accelerator. The method includes but is not limited to the following description.

[0092] It should be noted that the method of this application can be used to perform geometric transformation on multiple images simultaneously. However, for the convenience of description of the solution, the following mainly takes the geometric transformation of two images, namely the first original image and the second original image, as an example for introduction. The method of performing geometric transformation on multiple images simultaneously is similar to this.

[0093] S101. Determine the target transformation matrix, which is used for the geometric transformation between the first original image and the first target image, and the geometric transformation between the second original image and the second target image.

[0094] This step can be executed by the processor in the acceleration system.

[0095] First, determine the first transformation matrix and the second transformation matrix. The first transformation matrix is used for the geometric transformation between the first original image and the first target image, and the second transformation matrix is used for the geometric transformation between the second original image and the second target image; then stack the first transformation matrix and the second transformation matrix in the row dimension to obtain the target transformation matrix. If the dimensions of the first transformation matrix and the second transformation matrix are both k*3, then the dimension of the target transformation matrix is 2k*3. For example, if the geometric transformation is an affine transformation, then k takes the value of 2; if the geometric transformation is a projective transformation, then k takes the value of 3.

[0096] The first target image has the same size as the second target image. In one implementation, both the first target image and the second target image are images that have the same size as a given reference image. In another implementation, the first target image and the second target image have the same size but are different from the size of the reference image, and the same size of the first target image and the second target image can be specifically set according to actual application requirements. For example, if the height of the reference image is H rows, i.e., [0…H], and the width is W columns, i.e., [0…W], in actual applications, if we only need to obtain the [0…H / 2] rows and [0…W / 2] columns of the target image, that is, the output image only needs this part of the area of [0…H / 2] rows and [0…W / 2] columns, then the sizes of the first target image and the second target image can be set to [0…H / 2] rows and [0…W / 2] columns. Here, [0…H / 2] rows and [0…W / 2] columns are only examples, and in actual applications, they can be any other values, which do not constitute a limitation of this application. Taking the affine transformation as an example, if the first transformation matrix is M1 and the second transformation matrix is M2, then the target transformation matrix is M, where,

[0097]

[0098]

[0099]

[0100] In the first transformation matrix, m 00 、m 01 、m 10 and m 11 are used to control the rotation and scaling of the first original image, and m 02 and m 12 are used to control the translation of the first original image. Similarly, in the second transformation matrix, m 20 、m 21 、m 30 and m 31 are used to control the rotation and scaling of the second original image, and m 22 and m 32 are used to control the translation of the second original image.

[0101] The method for determining the first transformation matrix can be to calculate it using the least squares method based on 3 pairs of corresponding points or more than 3 pairs of corresponding points on the reference image and the first original image. Similarly, the method for the second transformation matrix can be to calculate it using the least squares method based on 3 pairs of corresponding points or more than 3 pairs of corresponding points on the reference image and the second original image. The first transformation matrix and the second transformation matrix can also be calculated by other methods, and this application does not limit the calculation methods of the first transformation matrix and the second transformation matrix.

[0102] In this embodiment, the first original image and the second original image are merely examples. In practical applications, geometric transformations are usually performed on a large number of images. When performing geometric transformations on a large number of images, the large number of images can be divided into multiple batches, where each batch includes multiple images. The transformation matrix corresponding to each image in each batch is calculated respectively, and then the target transformation matrix corresponding to each batch is determined. This method can be called batch processing. If the number of original images in a batch is n and the geometric transformation is an affine transformation, the dimension of the target transformation matrix is 2n * 3.

[0103] S102. Determine each coordinate point on the first target image.

[0104] This step can be executed by the processor in the acceleration system. Optionally, this step can also be executed by the vector operation acceleration unit.

[0105] It can be understood that if the first target image to be obtained and the second target image are images of the same size, the positions of each coordinate point included in the first target image are the same as those of each coordinate point on the second target image.

[0106] Determining each coordinate point on the first target image means determining each coordinate point on the second target image. For each original image included in a batch, the size of its corresponding target image is the same as that of the first target image. Therefore, each coordinate point included in the target image is each coordinate point included on the first target image.

[0107] Each coordinate point on the first target image can be generated by means of grid generation. For example, S can be used to represent the matrix composed of each coordinate point on the first target image. S includes src x coordinates, src y coordinates, and homogeneous coordinate 1.

[0108]

[0109] The first target image includes m pixel points. The src x coordinates and src y coordinates in the same column of the S matrix represent the coordinates of a pixel point on the first target image. For example, src x1 and src y1 represent the coordinates of a pixel point on the first target image, and src xm and src ym represent the coordinates of another pixel point on the first target image. The dimension of the S matrix is 3 * m, where m is the number of pixel points included in the first target image or each target image, that is, the product of the width and height of the first target image.

[0110] S103. Determine the original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image.

[0111] This step is executed by the matrix multiplication acceleration unit.

[0112] The original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image.

[0113] Perform a cross product of the target transformation matrix and the matrix formed by each coordinate point on the first target image / second target image to obtain the original coordinate position matrix. Taking affine transformation as an example, if there are n original images in a batch, the target transformation matrix corresponding to this batch is M 2n×3 , assuming that the first target image or each target image includes m pixel points, and the matrix formed by each coordinate point on the first target image is S 3×m , then the original coordinate position matrix is:

[0114] D 2n×m = M 2n×3 × S 3×m (5)

[0115] where D 2n×m is the original coordinate position matrix, the dimension of the original coordinate position matrix is 2n*m, and this original coordinate position matrix includes the corresponding coordinate positions of each coordinate point on the first target image on each of the n original images in this batch.

[0116] S104. Determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, and obtain the first target image and the second target image.

[0117] After determining the coordinate positions of each coordinate point on the first target image on the first original image, assign the pixel value at the corresponding coordinate position on the first original image to the corresponding coordinate point on the first target image, that is, determine the pixel values of each coordinate point on the first target image, and obtain the first target image. Similarly, after determining the coordinate positions of each coordinate point on the second target image on the second original image, assign the pixel value at the corresponding coordinate position on the second original image to the corresponding coordinate point on the second target image, that is, determine the pixel values of each coordinate point on the second target image, and obtain the second target image. It should be noted that the original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image. The pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image are also represented by a matrix, and this matrix is obtained through formula calculation.

[0118] When the coordinate position of each coordinate point on the first target image on the first original image is an integer, the pixel value at the integer coordinate position on the first original image can be directly assigned to the corresponding coordinate point on the first target image. For example, if the coordinate position of the coordinate point (1, 1) on the first target image on the first original image is (8, 8), then the pixel value A at the coordinate position (8, 8) on the first original image can be directly assigned to the coordinate point (1, 1) on the first target image as the pixel value of the (1, 1) coordinate point.

[0119] When the coordinate position of each coordinate point on the first target image on the first original image is a non-integer, it is necessary to calculate the pixel value at the non-integer coordinate position on the first original image. Usually, the linear interpolation method is used to calculate the pixel value at the non-integer coordinate position. For the convenience of understanding, taking the original image as a two-dimensional image and the bilinear interpolation method as an example, the following introduces how to calculate the pixel value at the non-integer coordinate position, and the implementation steps can be referred to Figure 4 the schematic diagram of the method flow shown.

[0120] S1041. Determine the adjacent coordinate points of the non-integer coordinate position and the pixel matrix of the adjacent coordinate points.

[0121] This step can be executed by the vector operation acceleration unit.

[0122] In bilinear interpolation, the adjacent coordinate points of the non-integer coordinate position include the 4 coordinate points or pixel points closest to the non-integer coordinate position. For example, referring to Figure 3 the example diagram shown, the black small dot represents the non-integer coordinate position, and the 4 coordinate points closest to this non-integer coordinate position are as Figure 3As shown by the small light-colored dots, x and y are the coordinate values obtained by rounding down the horizontal and vertical coordinates of a non-integer coordinate point respectively, and ox and oy are the remainders obtained by rounding down the horizontal and vertical coordinates of the non-integer coordinate point. Then the four neighboring coordinate points of this non-integer coordinate point are (x, y), (x + 1, y), (x, y + 1), and (x + 1, y + 1) respectively. The pixel values of the four neighboring coordinate points are p(x, y), p(x + 1, y), p(x, y + 1), and p(x + 1, y + 1) respectively, where p(x, y) represents the pixel value of the coordinate point (x, y) in the original image.

[0123] The pixel matrix of neighboring coordinate points refers to the matrix composed of the pixel values of neighboring coordinate points. If Q is used to represent the pixel matrix of neighboring coordinate points, then

[0124] Q = [p(x, y), p(x + 1, y), p(x, y + 1), p(x + 1, y + 1)] (6)

[0125] Here, it is introduced by taking the neighboring coordinate points of a non-integer coordinate position as an example. In practical applications, when there are n original images in a batch and each original image contains m pixel points, the pixel matrix of the corresponding neighboring coordinate points of this batch is Q nmC×4 , that is, the dimension of Q is nmC * 4, where C is the number of channels of the original image.

[0126] S1042. Determine the interpolation weight matrix.

[0127] This step of determining the interpolation weight matrix can be executed by the matrix multiplication acceleration unit.

[0128] The interpolation weight matrix refers to the matrix composed of the weights occupied by the pixel values of each neighboring coordinate point. In bilinear interpolation, the interpolation weight matrix can be obtained by the cross product of the bilinear interpolation factor matrix K and the distance matrix N of bilinear interpolation, that is, N × K, where K is the matrix expression coefficient of the bilinear interpolation algorithm and N is the variable of the matrix expression of the bilinear interpolation algorithm.

[0129]

[0130]

[0131] Here, it is introduced by taking the distance matrix of a non-integer coordinate position as an example. In practical applications, when there are n original images in a batch and each original image contains m pixel points, the distance matrix corresponding to this batch is N nm×4 , that is, the dimension of N is nm * 4.

[0132] S1043. Determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of adjacent coordinate points and the interpolation weight matrix.

[0133] This step can be executed by a vector operation acceleration unit.

[0134] Dot-multiply the pixel matrix of adjacent coordinate points with the interpolation weight matrix to obtain the pixel values at non-integer coordinate positions in the first original image and the pixel values at non-integer coordinate positions in the second original image, that is, obtain the pixel values of the corresponding coordinate points on the first target image and the pixel values of the corresponding coordinate points on the second target image, that is,

[0135] P n×m×C = reduce_sum(Q nmC×4 ·(N nm×4 ×K 4×4 ), axis = 1) (9)

[0136] where reduce_sum is a function for calculating the sum of tensor elements, and axis = 1 means performing a summation operation in the first dimension to obtain the first target image and the second target image.

[0137] Here, the first original image and the second original image are taken as examples for introduction, and the first target image and the second target image are obtained. If geometric transformation is performed on a batch of original images, the target images corresponding to each original image in this batch are obtained. It can be seen that the present application provides an acceleration method for geometric transformation of digital images. In the method of the present application, multiple images can be divided into multiple batches, each batch includes several images, calculate the target transformation matrix corresponding to the images in this batch, and use this target transformation matrix to perform affine transformation on the images in this batch together, and at the same time obtain the target images corresponding to the images in this batch. Compared with performing frame-by-frame operations at the bottom layer of the computing device, the method of processing in batches (batch) improves the speed of geometric transformation of multiple images, shortens the time required for geometric transformation of multiple images, and improves the operation efficiency. Divide multiple images into multiple batches. When performing geometric transformation on the images in each batch, for operations involving matrix cross-multiplication, use the matrix multiplication acceleration unit in the accelerator for operation, so that the core computing power resources in the accelerator are reasonably and efficiently utilized. In the geometric transformation of images, reasonably using the matrix multiplication acceleration unit and the vector operation acceleration unit for operation can improve the computing efficiency.

[0138] The present application provides an acceleration system for geometric transformation of digital images, as Figure 5 shown, Figure 5Schematic structural diagram of an acceleration system for geometric transformation of digital images provided by this application. The acceleration system includes a host and an accelerator. The accelerator can be plugged into the host in the form of a smart network card, or the accelerator can be integrated and deployed on the host. The host includes a processor, and the accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit.

[0139] The processor is used to form multiple images into multiple batches and determine the target transformation matrix corresponding to each batch, and is also used to determine each coordinate point on the reference image. Specifically, the processor can be used to execute steps S101 and S102 in the above method embodiments.

[0140] The matrix multiplication acceleration unit is used to determine the original coordinate position matrix according to the target transformation matrix and each coordinate point on the reference image. Specifically, the matrix multiplication acceleration unit can be used to execute step S103 in the above method embodiments.

[0141] The vector operation acceleration unit is used to determine the pixel matrix of adjacent coordinate points according to multiple image data. Specifically, the vector operation acceleration unit can be used to execute step S1041 in the above method embodiments. The matrix multiplication acceleration unit is also used to determine the interpolation weight matrix, which can be used to execute step S1042 in the above method embodiments; the vector operation acceleration unit is also used to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of adjacent coordinate points and the interpolation weight matrix, and obtain the target images of each batch, which can be used to execute step S1043 in the above method embodiments.

[0142] In this application, multiple images are divided into multiple batches. When performing geometric transformation on the images in each batch, for operations involving matrix cross multiplication, the matrix multiplication acceleration unit in the accelerator is used for the operation, so that the core computing power resources in the accelerator are reasonably and efficiently utilized. In the geometric transformation of images, reasonably using the matrix multiplication acceleration unit and the vector operation acceleration unit for operations can improve the computing efficiency.

[0143] The system structure and method embodiments are described above. Next, a virtual device corresponding to the above system and method will be introduced.

[0144] See Figure 6 , Figure 6 Schematic structural diagram of an accelerator 600 provided by this application. The accelerator 600 includes:

[0145] An acquisition unit 610 is configured to acquire a target transformation matrix, where the target transformation matrix is used for geometric transformation between a first original image and a first target image, and geometric transformation between a second original image and a second target image, and the first target image and the second target image have the same size;

[0146] The acquisition unit 610 is further configured to acquire each coordinate point on the first target image / second target image;

[0147] A matrix multiplication acceleration unit 620 is configured to determine an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image / second target image, where the original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image respectively and the coordinate positions of each coordinate point on the second target image on the second original image respectively;

[0148] The matrix multiplication acceleration unit 620 and a vector operation acceleration unit 630 cooperate to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, so as to obtain the first target image and the second target image.

[0149] In a possible implementation manner,

[0150] The vector operation acceleration unit 630 is configured to determine a pixel matrix of neighboring coordinate points according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, where the pixel matrix of neighboring coordinate points includes the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the reference image on the first original image respectively, and the matrix formed by the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the reference image on the second original image respectively;

[0151] The matrix multiplication acceleration unit 620 is configured to determine an interpolation weight matrix, where the interpolation weight matrix includes the weights of each pixel value in the pixel matrix of neighboring coordinate points;

[0152] The vector operation acceleration unit 630 is configured to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of neighboring coordinate points and the interpolation weight matrix, so as to obtain the first target image and the second target image.

[0153] In a possible implementation manner, the geometric transformation includes one of an affine transformation and a projective transformation.

[0154] Each functional module in the accelerator 600 is used to implement the methods described on the accelerator side in the above method embodiments. For details, please refer to the description of the above method embodiments. For the sake of brevity of the specification, it will not be repeated here.

[0155] Each functional module in the accelerator 600 can be implemented by software or hardware. The division of each functional module is only an example. In actual applications, the network device can be divided into more or fewer functional modules, which is not limited in this application.

[0156] This application also provides a device with computing functions, such as Figure 7 shown Figure 7 FIG. 700 is a schematic structural diagram of a device 700 with computing functions provided by this application. The device 700 with computing functions can be the accelerator in the above method embodiments and is used to implement the methods described on the accelerator side in the above method embodiments; the device 700 with computing functions can also be a physical machine, and the physical machine includes a processor for implementing the methods described on the processor side in the above method embodiments; the device 700 with computing functions can also be a physical machine, and the physical machine includes a processor and an accelerator, and the physical machine is used to implement all steps in the above method embodiments.

[0157] The device 700 with computing functions includes: a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other through the bus 702. It should be understood that this application does not limit the number of processors and memories in the device 700 with computing functions.

[0158] The bus 702 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 7 only one line is shown in, but it does not mean that there is only one bus or one type of bus. The bus 702 can include a path for transmitting information between various components (such as the memory 706, the processor 704, and the communication interface 708) in the device 700 with computing functions.

[0159] The processor 704 may include any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a micro processor (MP), or a digital signal processor (DSP).

[0160] The memory 706 may include volatile memory, such as random access memory (RAM). The processor 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0161] Executable code is stored in the memory 706. When the device 700 is configured as the accelerator in the method embodiment, the processor 704 executes the executable code to implement the functions of the foregoing acquisition unit 610, matrix multiplication acceleration unit 620, and vector operation acceleration unit 630, so as to implement the method described on the accelerator side in the acceleration method for geometric transformation of multiple images based on a reference image. That is, instructions for executing the method described on the accelerator side in the acceleration method for geometric transformation of digital images are stored on the memory 706.

[0162] When the device 700 is configured as a physical machine and the physical machine does not include an accelerator, the processor 704 executes the executable code to implement the functions of the processor in the method embodiment, so as to implement the method described on the processor side in the acceleration method for geometric transformation of multiple images based on a reference image. That is, instructions for executing the method described on the processor side in the acceleration method for geometric transformation of digital images are stored on the memory 706.

[0163] Optionally, when the device 700 is configured as a physical machine and the physical machine includes an accelerator, the processor 704 executes the executable code to implement the functions of the processor and the accelerator in the method embodiment, so as to implement all steps of the method described in the acceleration method for geometric transformation of multiple images based on a reference image. That is, instructions for executing all the methods described in the acceleration method for geometric transformation of digital images are stored on the memory 706.

[0164] The communication interface 708 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the device 700 with computing capabilities and other devices or communication networks.

[0165] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions that can run on a device with computing capabilities or be stored in any available medium. When the computer program product runs on at least one device with computing capabilities, it causes at least one device with computing capabilities to execute the methods described in the method for accelerating geometric transformation of digital images.

[0166] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a device with computing capabilities or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that direct a device with computing capabilities to execute the methods described in the method for accelerating geometric transformation of digital images.

[0167] The embodiments of the present application also provide a chip that can be used to implement some or all of the steps in the above method embodiments. For example, the chip can include a matrix multiplication acceleration unit and a vector operation acceleration unit, and the matrix multiplication acceleration unit and the vector operation acceleration unit can be respectively used to implement some steps in the above method embodiments.

[0168] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An acceleration system for geometric transformation of digital images, characterized in that, the digital image includes at least a first original image and a second original image, and the acceleration system includes a processor and an accelerator, the processor is configured to: determine a target transformation matrix for geometric transformation between the first original image and a first target image, and between the second original image and a second target image, wherein the first target image and the second target image have the same size; determine each coordinate point on the first target image, and each coordinate point on the second target image is the same as each coordinate point on the first target image; the accelerator is configured to: determine an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image, where the original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image; determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, and obtain the first target image and the second target image.

2. The acceleration system according to claim 1, characterized in that, the accelerator is configured to: determine a pixel matrix of neighboring coordinate points according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, where the pixel matrix of neighboring coordinate points includes the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image, and the matrix formed by the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the second target image on the second original image; determine an interpolation weight matrix, where the interpolation weight matrix includes the weights of each pixel value in the pixel matrix of neighboring coordinate points; determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of neighboring coordinate points and the interpolation weight matrix, and obtain the first target image and the second target image.

3. The acceleration system according to claim 2, characterized in that, the accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit, wherein, the matrix multiplication acceleration unit is configured to determine the original coordinate position matrix and determine the interpolation weight matrix; the vector operation acceleration unit is configured to determine the pixel matrix of neighboring coordinate points, and determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of neighboring coordinate points and the interpolation weight matrix, and obtain the first target image and the second target image.

4. The acceleration system according to any one of claims 1 to 3, wherein, the processor is configured to: determine a first transformation matrix for geometric transformation between the first original image and the first target image, the dimension of the first transformation matrix being k*3; determine a second transformation matrix for geometric transformation between the second original image and the second target image, the dimension of the second transformation matrix being k*3; superimpose the first transformation matrix and the second transformation matrix in the row dimension to obtain the target transformation matrix, the dimension of the target transformation matrix being 2k*3.

5. The acceleration system according to claim 4, wherein, the dimension of the matrix formed by each coordinate point on the reference image is 3*m, and the dimension of the original coordinate position matrix is 2k*m.

6. The acceleration system according to any one of claims 1 to 5, wherein, the geometric transformation includes one of an affine transformation and a projective transformation.

7. An acceleration method for geometric transformation of a digital image, wherein, the digital image includes at least a first original image and a second original image, and the method includes: obtaining a target transformation matrix for geometric transformation between the first original image and the first target image and between the second original image and the second target image, wherein the first target image and the second target image have the same size; obtaining each coordinate point on the first target image, and each coordinate point on the second target image being the same as each coordinate point on the first target image; determining an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image, the original coordinate position matrix including the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image; determining the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, to obtain the first target image and the second target image.

8. The method according to claim 7, wherein, the determining the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, to obtain the first target image and the second target image, includes: Determine a pixel matrix of neighboring coordinate points based on the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image. The pixel matrix of neighboring coordinate points includes a matrix composed of the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image, and the pixel values of the neighboring coordinate points of the coordinate positions of each coordinate point on the second target image on the second original image; Determine an interpolation weight matrix, where the interpolation weight matrix includes the weights of each pixel value in the pixel matrix of neighboring coordinate points; Determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the pixel matrix of neighboring coordinate points and the interpolation weight matrix, and obtain the first target image and the second target image.

9. The method according to claim 7 or 8, wherein, the geometric transformation includes one of an affine transformation and a projective transformation.

10. An acceleration device for geometric transformation of digital images, wherein, the digital image includes at least a first original image and a second original image, and includes: an acquisition unit for acquiring a target transformation matrix, where the target transformation matrix is used for the geometric transformation between the first original image and the first target image, and the geometric transformation between the second original image and the second target image, wherein the first target image and the second target image have the same size; the acquisition unit is further configured to acquire each coordinate point on the first target image, and each coordinate point on the second target image is the same as each coordinate point on the first target image; a matrix multiplication acceleration unit is configured to determine an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image, where the original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image; the matrix multiplication acceleration unit and the vector operation acceleration unit cooperate to determine the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, and obtain the first target image and the second target image.

11. The device according to claim 10, wherein, The vector operation acceleration unit is configured to determine a pixel matrix of neighboring coordinate points according to the original coordinate position matrix, pixel values of each coordinate point on the first original image, and pixel values of each coordinate point on the second original image, where the pixel matrix of neighboring coordinate points includes a matrix formed by pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image and pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the second target image on the second original image; The matrix multiplication acceleration unit is configured to determine an interpolation weight matrix, where the interpolation weight matrix includes weights occupied by each pixel value in the pixel matrix of neighboring coordinate points; The vector operation acceleration unit is configured to determine pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the pixel matrix of neighboring coordinate points and the interpolation weight matrix, so as to obtain the first target image and the second target image.

12. The apparatus according to claim 10 or 11, wherein, the geometric transformation includes one of an affine transformation and a projective transformation.

13. A chip, wherein, the chip is configured to implement the method according to any one of claims 7 to 9.

Images