An accelerated system for geometric transformation of digital images

Abstract

The application provides an acceleration system for geometric transformation of digital images, applied to the field of image processing. The acceleration system comprises a processor and an accelerator. The processor is configured to determine a target transformation matrix, which is used for geometric transformation between a first original image and a first target image, and geometric transformation between a second original image and a second target image; and determine each coordinate point on the first target image, each coordinate point on the second target image being the same as each coordinate point on the first target image. The accelerator is configured to determine an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image; and determine the first target image and the second target image according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image. The acceleration system in the application can be used for geometric transformation of multiple digital images at the same time, and the operation efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an acceleration system for geometric transformation of digital images. BACKGROUND

[0002] Geometric transformation (such as affine transformation, perspective transformation) is a common algorithm in image processing, and is widely used in the field of computer vision, such as in face recognition, image scaling, image rotation, image registration correction, etc.

[0003] At present, when using a computing device to perform geometric transformation on an image, the underlying logic of the computing device is pixel-by-pixel operation, which is also referred to as vector operation (addition, subtraction, multiplication, and division operations) in the industry, and the operation efficiency is low. SUMMARY

[0004] The present application provides an acceleration system for geometric transformation of digital images, which simultaneously performs geometric transformation on multiple digital images, thereby improving the operation efficiency compared to performing geometric transformation on a single image pixel by pixel.

[0005] In a first aspect, the present application provides an acceleration system for geometric transformation of digital images, the digital images including a first original image and a second original image, the acceleration system including a processor and an accelerator,

[0006] The processor is configured to:

[0007] determine a target transformation matrix, the target transformation matrix being used for geometric transformation between the first original image and a first target image, and geometric transformation between the second original image and a second target image, wherein the first target image and the second target image are of the same size;

[0008] determine each coordinate point on the first target image, each coordinate point on the second target image being the same as each coordinate point on the first target image;

[0009] The accelerator is configured to:

[0010] determine an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image, the original coordinate position matrix including coordinate positions of each coordinate point on the first target image on the first original image and coordinate positions of each coordinate point on the second target image on the second original image;

[0011] determine pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, and obtain the first target image and the second target image.

[0012] The scheme of the present application is described by taking the first original image and the second original image in the digital image as an example. The target transformation matrix is calculated. In the bottom logic of the computing device, the target transformation matrix is used to simultaneously perform geometric transformation operation on the first original image and the second original image. Compared with performing geometric transformation operation on the original images one by one, the calculation efficiency is improved. It should be noted that the digital image can include a larger number of images. In actual application, a plurality of images can be processed in batches. A batch includes a plurality of images. The target transformation matrix corresponding to the images in the batch is calculated first. The target transformation matrix of the batch is used to simultaneously perform geometric transformation operation on the plurality of images in the batch. Compared with performing geometric transformation operation on the images one by one, the calculation efficiency is improved. In addition, simultaneously performing geometric transformation on a batch of images involves matrix or vector operation. The accelerator is more suitable for matrix or vector operation. Using the accelerator to perform matrix or vector operation makes the processor and the accelerator each perform its function and each perform its best. For the geometric transformation scene of a large number of images, the operation time is shortened, and the operation efficiency is further improved.

[0013] Based on the first aspect, in possible implementation manners, the accelerator is configured to:

[0014] According to the original coordinate position matrix, the pixel values of the coordinate points on the first original image, and the pixel values of the coordinate points on the second original image, a pixel matrix of adjacent coordinate points is determined. The pixel matrix of adjacent coordinate points includes a matrix composed of the pixel values of the adjacent coordinate points of the coordinate positions of the coordinate points on the first target image on the first original image and the pixel values of the adjacent coordinate points of the coordinate positions of the coordinate points on the second target image on the second original image.

[0015] An interpolation weight matrix is determined. The interpolation weight matrix includes the weights of the pixel values in the pixel matrix of adjacent coordinate points.

[0016] According to the pixel matrix of adjacent coordinate points and the interpolation weight matrix, the pixel values of the coordinate points on the first target image and the pixel values of the coordinate points on the second target image are determined, and the first target image and the second target image are obtained.

[0017] Based on the first aspect, in possible implementation manners, the accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit, wherein,

[0018] The matrix multiplication acceleration unit is configured to determine the original coordinate position matrix and determine the interpolation weight matrix.

[0019] The vector operation acceleration unit is configured to determine a pixel matrix of the neighboring coordinate points, and determine the pixel value of each coordinate point on the first target image and the pixel value of each coordinate point on the second target image according to the pixel matrix of the neighboring coordinate points and the interpolation weight matrix, so as to obtain the first target image and the second target image.

[0020] The accelerator comprises the matrix multiplication acceleration unit and the vector operation acceleration unit, the matrix multiplication acceleration unit is a parallel operation unit for performing matrix cross multiplication operation, and the vector operation acceleration unit is a parallel operation unit for performing vector operation. The matrix multiplication acceleration unit and the vector operation acceleration unit in the accelerator are reasonably utilized, so that the matrix multiplication acceleration unit and the vector operation acceleration unit can perform their respective functions and play their respective roles, and the operation efficiency is improved.

[0021] In a possible implementation manner based on the first aspect, the processor is configured to:

[0022] determine a first transformation matrix, the first transformation matrix being used for geometric transformation between the first original image and the first target image, and the first transformation matrix having a dimension of k*3;

[0023] determine a second transformation matrix, the second transformation matrix being used for geometric transformation between the second original image and the second target image, and the second transformation matrix having a dimension of k*3;

[0024] stack the first transformation matrix and the second transformation matrix in a row dimension to obtain a target transformation matrix, and the target transformation matrix having a dimension of 2k*3.

[0025] It can be understood that the first transformation matrix and the second transformation matrix are taken as examples for description, when a batch includes multiple images, the multiple images correspond to multiple transformation matrices, and the target transformation matrix is determined according to the multiple transformation matrices, and the target transformation matrix is used for simultaneously performing geometric transformation on the multiple images in the batch.

[0026] In a possible implementation manner based on the first aspect, a matrix formed by the coordinate points on the reference image has a dimension of 3*m, and the original coordinate position matrix has a dimension of 2k*m.

[0027] In a possible implementation manner based on the first aspect, the geometric transformation includes one of affine transformation and transmission transformation.

[0028] In a possible implementation manner based on the first aspect, the geometric transformation includes one of affine transformation and transmission transformation.

[0029] The accelerator obtains a target transformation matrix, the target transformation matrix being used for geometric transformation between the first original image and the first target image and geometric transformation between the second original image and the second target image, wherein the first target image and the second target image have the same size;

[0030] The accelerator obtains each coordinate point on the first target image, each coordinate point on the second target image being the same as each coordinate point on the first target image;

[0031] The accelerator determines an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image, the original coordinate position matrix including coordinate positions of each coordinate point on the first target image on the first original image and coordinate positions of each coordinate point on the first target image on the second original image;

[0032] The accelerator determines pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image and the pixel values of each coordinate point on the second original image, and obtains the first target image and the second target image.

[0033] The scheme is introduced by taking the target transformation matrix as an example for geometric transformation between the first original image and the first target image and geometric transformation between the second original image and the second target image. In actual application, the target transformation matrix can be used for geometric transformation of multiple images. Compared with performing geometric transformation on original images one by one, the use of the target transformation matrix for geometric transformation of multiple images at the same time improves the calculation efficiency.

[0034] Based on the second aspect, in possible implementation manners, the accelerator determines pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image and the pixel values of each coordinate point on the second original image, and obtains the first target image and the second target image, including:

[0035] The accelerator determines a pixel matrix of adjacent coordinate points according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image and the pixel values of each coordinate point on the second original image, the pixel matrix of adjacent coordinate points including a matrix formed by pixel values of adjacent coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image and pixel values of adjacent coordinate points of the coordinate positions of each coordinate point on the first target image on the second original image.

[0036] The accelerator determines an interpolation weight matrix, the interpolation weight matrix including weights of each pixel value in the pixel matrix of adjacent coordinate points.

[0037] According to the pixel matrix of the adjacent coordinate points and the interpolation weight matrix, the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image are determined, and the first target image and the second target image are obtained.

[0038] In a possible implementation manner based on the second aspect, the geometric transformation includes one of an affine transformation and a perspective transformation.

[0039] Based on the second aspect, before the accelerator obtains the target transformation matrix, the method further includes:

[0040] The processor determines the target transformation matrix;

[0041] The processor sends the target transformation matrix to the accelerator.

[0042] In a possible implementation manner based on the second aspect, the processor determining the target transformation matrix includes:

[0043] The processor determines a first transformation matrix, the first transformation matrix being used for the geometric transformation between the first original image and the first target image, and the first transformation matrix having a dimension of k*3;

[0044] A second transformation matrix is determined, the second transformation matrix being used for the geometric transformation between the second original image and the second target image, and the second transformation matrix having a dimension of k*3;

[0045] The first transformation matrix and the second transformation matrix are superimposed in a row dimension to obtain the target transformation matrix, and the target transformation matrix has a dimension of 2k*3.

[0046] In a possible implementation manner based on the second aspect, the matrix composed of each coordinate point on the reference image has a dimension of 3*m, and the original coordinate position matrix has a dimension of 2k*m.

[0047] In a third aspect, the present application provides an accelerator, including:

[0048] An obtaining unit is configured to obtain a target transformation matrix, the target transformation matrix being used for geometric transformation between a first original image and a first target image and geometric transformation between a second original image and a second target image, wherein the first target image and the second target image have the same size.

[0049] The obtaining unit is further configured to obtain each coordinate point on the first target image, and each coordinate point on the second target image is the same as each coordinate point on the first target image.

[0050] The matrix multiplication acceleration unit is configured to determine, according to the target transformation matrix and each coordinate point on the first target image, a raw coordinate position matrix, the raw coordinate position matrix including coordinate positions of each coordinate point on the first target image on the first raw image and coordinate positions of each coordinate point on the second target image on the second raw image;

[0051] The matrix multiplication acceleration unit and the vector operation acceleration unit are cooperatively configured to determine, according to the raw coordinate position matrix, pixel values of each coordinate point on the first raw image and pixel values of each coordinate point on the second raw image, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

[0052] In possible implementation manners based on the third aspect,

[0053] The vector operation acceleration unit is configured to determine, according to the raw coordinate position matrix, pixel values of each coordinate point on the first raw image and pixel values of each coordinate point on the second raw image, a neighboring coordinate point pixel matrix, the neighboring coordinate point pixel matrix including a matrix formed by pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first raw image and pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the second target image on the second raw image.

[0054] The matrix multiplication acceleration unit is configured to determine an interpolation weight matrix, the interpolation weight matrix including weights of each pixel value in the neighboring coordinate point pixel matrix.

[0055] The vector operation acceleration unit is configured to determine, according to the neighboring coordinate point pixel matrix and the interpolation weight matrix, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

[0056] In possible implementation manners based on the third aspect, the geometric transformation includes one of an affine transformation and a perspective transformation.

[0057] Each functional module of the third aspect is configured to implement the method described on the accelerator side in the second aspect above, and the method described on the accelerator side in any one of the possible implementation manners of the second aspect above.

[0058] In a fourth aspect, the present application provides a host, including:

[0059] A processor is configured to determine a target transformation matrix, the target transformation matrix being used for geometric transformation between the first raw image and the first target image and geometric transformation between the second raw image and the second target image, wherein the first target image and the second target image are of the same size.

[0060] The processor is further configured to send the target transformation matrix to the accelerator, so that the accelerator performs geometric transformation on the first original image and the second original image according to the target transformation matrix.

[0061] According to the fourth aspect, in possible implementation manners, the processor is configured to:

[0062] determine a first transformation matrix, the first transformation matrix being used for geometric transformation between the first original image and the first target image, the first transformation matrix having a dimension of k*3;

[0063] determine a second transformation matrix, the second transformation matrix being used for geometric transformation between the second original image and the second target image, the second transformation matrix having a dimension of k*3;

[0064] stack the first transformation matrix and the second transformation matrix in a row dimension to obtain a target transformation matrix, the target transformation matrix having a dimension of 2k*3.

[0065] According to the fourth aspect, in possible implementation manners, a matrix formed by each coordinate point on the reference image has a dimension of 3*m, and the original coordinate position matrix has a dimension of 2k*m.

[0066] Each functional module of the fourth aspect is configured to implement the method described on the processor side in the second aspect above, and the method described on the processor side in any possible implementation manner of the second aspect above.

[0067] In the fifth aspect, the present application provides a host including a processor and a memory, the memory being configured to store instructions, and the processor being configured to execute the instructions stored in the memory to implement the method described on the processor side in the second aspect above, and the method described on the processor side in any possible implementation manner of the second aspect above.

[0068] In the sixth aspect, the present application provides an accelerator including a processor and a memory, the memory being configured to store instructions, and the processor being configured to execute the instructions stored in the memory to implement the method described on the accelerator side in the second aspect above, and the method described on the accelerator side in any possible implementation manner of the second aspect above.

[0069] In the seventh aspect, the present application provides a computer storage medium including program instructions, when the program instructions are executed by a device having a computing function, the device having the computing function executes the method described by the accelerator and / or the processor in the second aspect above, and the method described by the accelerator and / or the processor in any possible implementation manner of the second aspect above.

[0070] In an eighth aspect, the present application provides a computer program product, comprising program instructions, which, when executed by a device having computing function, cause the device having computing function to perform the method described by the accelerator and / or the processor in the second aspect above, and the method described by the accelerator and / or the processor in any possible implementation manner of the second aspect above.

[0071] In a ninth aspect, the present application provides a chip, which can be used to implement the method described by the accelerator side in the second aspect above or any possible implementation manner of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1A A schematic diagram of a reference image given in the present application;

[0073] Figure 1B A schematic diagram of a to-be-processed image given in the present application;

[0074] Figure 2 A flowchart of an accelerated method for geometric transformation of a digital image provided in the present application;

[0075] Figure 3 An example diagram provided in the present application;

[0076] Figure 4 A flowchart of a method for calculating a pixel value at a non-integer coordinate position by using a bilinear interpolation method provided in the present application;

[0077] Figure 5 A structural diagram of an accelerated system for geometric transformation of a digital image provided in the present application;

[0078] Figure 6 A structural diagram of an accelerator provided in the present application;

[0079] Figure 7 A structural diagram of a device having computing function provided in the present application. DETAILED DESCRIPTION

[0080] Some technical terms related to the method embodiments of the present application are introduced.

[0081] Vector operation acceleration unit refers to a parallel computing unit specialized in vector operation.

[0082] Matrix multiplication acceleration unit refers to a parallel computing unit that can only be used for matrix cross multiplication operation.

[0083] The core computing resource of the current mainstream artificial intelligence (AI) accelerator is a matrix multiplication acceleration unit, and the computing resource of the matrix multiplication acceleration unit accounts for most or even the majority of the AI accelerator resource.

[0084] Many applications can be implemented through matrix cross multiplication operations, or part of the operations in the applications can be implemented through matrix cross multiplication, but none of them is implemented through the matrix multiplication acceleration unit, but through the vector operation acceleration unit using vectors or through other operation forms, which makes the core computing resource in the accelerator not be reasonably utilized.

[0085] The present application provides a method and system for geometric transformation of multiple images based on a reference image. Before introducing the method and system provided by the present application, the application scenario involved in the present application scheme will be introduced.

[0086] The present application scheme is applicable to the application scenario of geometric transformation of multiple images based on a given reference image, where the geometric transformation may be, for example, affine transformation or perspective transformation, and the affine transformation includes translation, rotation, scaling, skewing and reflection. For example, in a target recognition application scenario, target recognition can be face recognition, license plate recognition, etc., and the collected images need to be geometrically transformed into images of the same size, the same shape and the same uniform format as the reference image, and then further processed to achieve target recognition.

[0087] For example, as shown in Figure 1A and Figure 1B , Figure 1A is a schematic diagram of the reference image given by the present application, Figure 1B image 1, image 2 and image 3 are images to be processed, where image 1, image 2 and image 3 are images of different sizes and different poses of the target in the image, and it is required to perform geometric transformation on image 1, image 2 and image 3 according to the given reference image to obtain a target image of the same size, the same shape and the same pose as the reference image. The target image can be further processed to obtain an image that meets the application requirements.

[0088] It should be noted that the sizes of two images are different, which means that the number of rows or the number of columns or both of the two images are different. The poses of the targets in the two images are different, which means that the poses of the targets in the two images are different, for example, the pose of the person in image 1 is different from that in image 2, and the poses of the person in image 1 and image 2 are different from that in image 3.

[0089] In Figure 1A and Figure 1BIn the shown application scenario, the image 1, the image 2 and the image 3 can be subjected to affine transformation according to the reference image to obtain target images 1, 2 and 3 of the same size, shape and image placement as the reference image. Figure 1B

[0090] Alternatively, the faces in the image 1, the image 2 and the image 3 can also be located on one image, i.e., one image includes multiple faces, and any one or more of the size, line shape and posture of the multiple faces can be different. Geometric transformation is performed on the multiple faces on the one image respectively to obtain target images of the same size, shape and placement as the reference image.

[0091] The following describes an acceleration method for geometric transformation of digital images provided by the present application, which is described with reference to Figure 2 Figure 2 FIG. 1 is a flowchart of an acceleration method for geometric transformation of digital images provided by the present application. The method is applied to an acceleration system for geometric transformation of digital images, and the acceleration system includes a processor and an accelerator. The method includes but is not limited to the following description.

[0092] It should be noted that the method of the present application can be used to simultaneously perform geometric transformation on multiple images. However, for the convenience of description of the scheme, the following mainly takes the example of simultaneously performing geometric transformation on two images, i.e., a first original image and a second original image, and the method of simultaneously performing geometric transformation on multiple images is similar to this.

[0093] S101, determine a target transformation matrix, which is used for geometric transformation between the first original image and the first target image, and geometric transformation between the second original image and the second target image.

[0094] This step can be performed by the processor in the acceleration system.

[0095] First, determine a first transformation matrix and a second transformation matrix. The first transformation matrix is used for geometric transformation between the first original image and the first target image, and the second transformation matrix is used for geometric transformation between the second original image and the second target image. Then, the first transformation matrix and the second transformation matrix are superimposed in the row dimension to obtain the target transformation matrix. If the dimensions of the first transformation matrix and the second transformation matrix are both k*3, the dimension of the target transformation matrix is 2k*3. For example, if the geometric transformation is affine transformation, k takes the value of 2, and if the geometric transformation is perspective transformation, k takes the value of 3.

[0096] ​​The first target image and the second target image have the same size. In one implementation, the first target image and the second target image are images having the same size as the given reference image. In another implementation, the first target image and the second target image have the same size but different from the reference image, and the size of the first target image and the second target image can be set according to actual application requirements. For example, if the height of the reference image is H rows, i.e., [0…H], and the width is W columns, i.e., [0…W], in actual application, we only need to obtain [0…H / 2] rows and [0…W / 2] columns of the target image, i.e., the output image only needs [0…H / 2] rows and [0…W / 2] columns of the target image, then the size of the first target image and the second target image can be set as [0…H / 2] rows and [0…W / 2] columns. Here, [0…H / 2] rows and [0…W / 2] columns are only examples, and in actual application, they can be any other values and do not limit the present application. Taking an affine transformation as an example, if the first transformation matrix is M1 and the second transformation matrix is M2, then the target transformation matrix is M, where,

[0097]

[0098]

[0099]

[0100] In the first transformation matrix, m 00 , m 01 , m 10 and m 11 are used to control the rotation and scaling of the first original image, m 02 and m 12 are used to control the translation of the first original image. Similarly, in the second transformation matrix, m 20 , m 21 , m 30 and m 31 are used to control the rotation and scaling of the second original image, m 22 and m 32 are used to control the translation of the second original image.

[0101] The method for determining the first transformation matrix can be that, according to the corresponding 3 pairs of points or 3 pairs of points or more on the reference image and the first original image, the least square method is used to calculate, and similarly, the method for determining the second transformation matrix can be that, according to the corresponding 3 pairs of points or 3 pairs of points or more on the reference image and the second original image, the least square method is used to calculate. The first transformation matrix and the second transformation matrix can also be calculated by other methods, and the present application does not limit the calculation method of the first transformation matrix and the second transformation matrix.

[0102] In this embodiment, the first original image and the second original image are only an example, and in actual application, geometric transformation is usually performed on a plurality of images. When geometric transformation is performed on a large number of images, the large number of images can be divided into a plurality of batches, each batch including a plurality of images, and a transformation matrix corresponding to each image in each batch is calculated respectively, and then a target transformation matrix corresponding to each batch is determined. This method can be referred to as batch processing. If the number of original images in a batch is n and the geometric transformation is affine transformation, the dimension of the target transformation matrix is 2n*3.

[0103] S102, determine each coordinate point on the first target image.

[0104] This step can be performed by a processor in an acceleration system. Alternatively, this step can also be performed by a vector operation acceleration unit.

[0105] It can be understood that the first target image and the second target image to be obtained are images of the same size, and each coordinate point position included in the first target image is the same as each coordinate point position on the second target image.

[0106] Determination of each coordinate point on the first target image is determination of each coordinate point on the second target image. For each original image included in a batch, the size of the corresponding target image is the same as the size of the first target image, and therefore each coordinate point included in the target image is each coordinate point included in the first target image.

[0107] Each coordinate point on the first target image can be generated by a grid generation method. For example, S can be used to represent a matrix of each coordinate point on the first target image, S includes src x coordinates, src y coordinates and homogeneous coordinates 1,

[0108]

[0109] The first target image includes m pixel points, and src x coordinates and src y coordinates in the same column in the S matrix represent the coordinates of a pixel point on the first target image, for example, src x1 and src y1 represent the coordinates of a pixel point on the first target image, src xm and src ym represent the coordinates of another pixel point on the first target image, and the dimension of the S matrix is 3*m, m is the number of pixel points included in the first target image or each target image, that is, the product of the width and height of the first target image.

[0110] S103, determining an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image.

[0111] This step is performed by the matrix multiplication acceleration unit.

[0112] The original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image.

[0113] The target transformation matrix is cross-multiplied with the matrix composed of each coordinate point on the first target image / second target image to obtain the original coordinate position matrix. Taking an affine transformation as an example, if n original images are included in a batch, the target transformation matrix corresponding to the batch is M 2n×3 , assuming that m pixel points are included in the first target image or each target image, the matrix composed of each coordinate point on the first target image is S 3×m , and the original coordinate position matrix is:

[0114] D 2n×m =M 2n×3 ×S 3×m (5)

[0115] wherein D 2n×m is the original coordinate position matrix, the dimension of the original coordinate position matrix is 2n*m, and the original coordinate position matrix includes what are the corresponding coordinate positions of each coordinate point on the first target image on each original image in the batch of n original images.

[0116] S104, determining the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image according to the original coordinate position matrix, the pixel values of each coordinate point on the first original image, and the pixel values of each coordinate point on the second original image, and obtaining the first target image and the second target image.

[0117] After the coordinate positions of each coordinate point on the first target image on the first original image are determined, the pixel value of the corresponding coordinate position on the first original image is assigned to the corresponding coordinate point on the first target image, i.e., the pixel value of each coordinate point on the first target image is determined, and the first target image is obtained. Similarly, after the coordinate positions of each coordinate point on the second target image on the second original image are determined, the pixel value of the corresponding coordinate position on the second original image is assigned to the corresponding coordinate point on the second target image, i.e., the pixel value of each coordinate point on the second target image is determined, and the second target image is obtained. It should be noted that the original coordinate position matrix includes the coordinate positions of each coordinate point on the first target image on the first original image and the coordinate positions of each coordinate point on the second target image on the second original image, and the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image are also represented by a matrix, which is obtained by formula calculation.

[0118] For the coordinate positions of each coordinate point on the first target image on the first original image being an integer, the pixel value of the integer coordinate position on the first original image can be directly assigned to the corresponding coordinate point on the first target image. For example, the coordinate position of the coordinate point (1, 1) on the first target image on the first original image is (8, 8), and the pixel value A of the coordinate position (8, 8) on the first original image can be directly assigned to the coordinate point (1, 1) on the first target image as the pixel value of the coordinate point (1, 1).

[0119] For the coordinate positions of each coordinate point on the first target image on the first original image being a non-integer, the pixel value of the non-integer coordinate position on the first original image needs to be calculated. Usually, the linear interpolation method is used to calculate the pixel value of the non-integer coordinate position. In order to facilitate understanding, the following introduces how to calculate the pixel value of the non-integer coordinate position by taking the original image as a two-dimensional image and taking the bilinear interpolation method as an example. The implementation steps can refer to the method flow diagram shown in Figure 4

[0120] S1041, determine the adjacent coordinate points of the non-integer coordinate position and the pixel matrix of the adjacent coordinate points.

[0121] This step can be executed by a vector operation acceleration unit.

[0122] In the bilinear interpolation, the adjacent coordinate points of the non-integer coordinate position include the four coordinate points or pixels closest to the non-integer coordinate position. For example, referring to the example diagram shown in Figure 3 Figure 3 ​​The small circle of middle light color is shown, x and y are the horizontal and vertical coordinates of the non-integer coordinate point, and the coordinates are taken down to the integer value, ox and oy are the remainders of the horizontal and vertical coordinates of the non-integer coordinate point taken down to the integer value, then the four adjacent coordinate points of the non-integer coordinate point are (x, y), (x+1, y), (x, y+1), (x+1, y+1), and the pixel values of the four adjacent coordinate points are p(x, y), p(x+1, y), p(x, y+1), p(x+1, y+1), wherein p(x, y) represents the pixel value of the coordinate point (x, y) in the original image.

[0123] The pixel matrix of the adjacent coordinate point refers to the matrix composed of the pixel values of the adjacent coordinate point, if Q represents the pixel matrix of the adjacent coordinate point, then

[0124] Q = [p(x, y), p(x+1, y), p(x, y+1), p(x+1, y+1)] (6)

[0125] Here, the adjacent coordinate point of a non-integer coordinate position is taken as an example for introduction, in actual application, when a batch includes n original images, each original image includes m pixel points, the pixel matrix of the adjacent coordinate point corresponding to the batch is Q nmC×4 , that is, the dimension of Q is nmC*4, wherein C is the number of channels of the original image.

[0126] S1042, determine the interpolation weight matrix.

[0127] The step of determining the interpolation weight matrix can be executed by the matrix multiplication acceleration unit.

[0128] The interpolation weight matrix refers to the matrix composed of the weight occupied by the pixel value of each adjacent coordinate point. In the bilinear interpolation, the interpolation weight matrix can be obtained by the cross multiplication of the bilinear interpolation factor matrix K and the distance matrix N of the bilinear interpolation, that is, N x K, wherein K is the matrix expression coefficient of the bilinear interpolation algorithm, and N is the variable of the matrix expression of the bilinear interpolation algorithm,

[0129]

[0130]

[0131] Here, the distance matrix of a non-integer coordinate position is taken as an example for introduction, in actual application, when a batch includes n original images, each original image includes m pixel points, the distance matrix corresponding to the batch is N nm×4 , that is, the dimension of N is nm*4.

[0132] S1043, determining the pixel value of each coordinate point on the first target image and the pixel value of each coordinate point on the second target image according to the pixel matrix of the adjacent coordinate points and the interpolation weight matrix.

[0133] This step can be performed by a vector operation acceleration unit.

[0134] The pixel matrix of the adjacent coordinate points is multiplied by the interpolation weight matrix to obtain the pixel value of the non-integer coordinate position in the first original image and the pixel value of the non-integer coordinate position in the second original image, that is, the pixel value of the corresponding coordinate point on the first target image and the pixel value of the corresponding coordinate point on the second target image, that is,

[0135] P n×m×C = reduce_sum(Q nmC×4 ·(N nm×4 ×K 4×4 ),axis=1) (9)

[0136] wherein reduce_sum is a function for calculating the sum of tensor elements, and axis=1 means that the summation operation is performed on the first dimension, thereby obtaining the first target image and the second target image.

[0137] Here, the first original image and the second original image are taken as examples for introduction, and the first target image and the second target image are obtained. If a batch of original images is subjected to geometric transformation, the target images corresponding to the original images in the batch are obtained. It can be seen that the application provides an acceleration method for geometric transformation of digital images. In the method of the application, a plurality of images can be divided into a plurality of batches, a plurality of images are included in each batch, a target transformation matrix corresponding to the batch of images is calculated, and the batch of images is subjected to affine transformation using the target transformation matrix, and the target images corresponding to the batch of images are obtained. Compared with the frame-by-frame operation at the bottom of the computing device, the use of the batch processing method improves the speed of geometric transformation of multiple images, shortens the time required for geometric transformation of multiple images, and improves the operation efficiency. The plurality of images are divided into a plurality of batches, and when the images in each batch are subjected to geometric transformation, the matrix multiplication operation is performed by using the matrix multiplication acceleration unit in the accelerator, so that the core computing resource in the accelerator is reasonably and efficiently utilized. In the geometric transformation of the image, the matrix multiplication acceleration unit and the vector operation acceleration unit are reasonably utilized for operation, which can improve the computing efficiency.

[0138] The application provides an acceleration system for geometric transformation of digital images, as shown in Figure 5 Figure 5 ​An acceleration system structure diagram for geometric transformation of digital images is provided in the present application, the acceleration system includes a host and an accelerator, the accelerator can be plugged in the form of an intelligent network card on the host, and the accelerator can also be integrated and deployed on the host. The host includes a processor, and the accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit.

[0139] The processor is configured to group multiple images into batches, determine a target transformation matrix corresponding to each batch, and determine coordinate points on a reference image. Specifically, the processor can be configured to perform steps S101 and S102 in the method embodiments described above.

[0140] The matrix multiplication acceleration unit is configured to determine an original coordinate position matrix based on the target transformation matrix and the coordinate points on the reference image. Specifically, the matrix multiplication acceleration unit can be configured to perform step S103 in the method embodiments described above.

[0141] The vector operation acceleration unit is configured to determine a pixel matrix of adjacent coordinate points based on multiple image data. Specifically, the vector operation acceleration unit can be configured to perform step S1041 in the method embodiments described above. The matrix multiplication acceleration unit is further configured to determine an interpolation weight matrix, which can be configured to perform step S1042 in the method embodiments described above. The vector operation acceleration unit is further configured to determine pixel values of coordinate points in a first target image and pixel values of coordinate points in a second target image based on the pixel matrix of adjacent coordinate points and the interpolation weight matrix, and obtain target images of each batch, which can be configured to perform step S1043 in the method embodiments described above.

[0142] In the present application, multiple images are divided into batches, and when performing geometric transformation on images in each batch, the matrix multiplication acceleration unit in the accelerator is used to perform operations involving matrix cross multiplication, so that the core computing resources in the accelerator are reasonably and efficiently utilized. In the geometric transformation of images, the matrix multiplication acceleration unit and the vector operation acceleration unit are reasonably utilized to perform operations, which can improve the computing efficiency.

[0143] The system structure and method embodiments are described above, and the virtual devices corresponding to the above system and method are introduced below.

[0144] Referring to Figure 6 , Figure 6 A structure diagram of an accelerator 600 is provided in the present application, the accelerator 600 includes:

[0145] The acquisition unit 610 is configured to acquire a target transformation matrix, the target transformation matrix being used for geometric transformation between the first original image and the first target image and geometric transformation between the second original image and the second target image, wherein the first target image and the second target image have the same size.

[0146] The acquisition unit 610 is further configured to acquire each coordinate point on the first target image / second target image.

[0147] The matrix multiplication acceleration unit 620 is configured to determine, according to the target transformation matrix and each coordinate point on the first target image / second target image, an original coordinate position matrix, the original coordinate position matrix including coordinate positions of each coordinate point on the first target image in the first original image and coordinate positions of each coordinate point on the second target image in the second original image.

[0148] The matrix multiplication acceleration unit 620 and the vector operation acceleration unit 630 are jointly configured to determine, according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, to obtain the first target image and the second target image.

[0149] In a possible implementation manner,

[0150] The vector operation acceleration unit 630 is configured to determine, according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, a neighboring coordinate point pixel matrix including a matrix formed by pixel values of neighboring coordinate points of each coordinate point on the first original image and pixel values of neighboring coordinate points of each coordinate point on the second original image.

[0151] The matrix multiplication acceleration unit 620 is configured to determine an interpolation weight matrix including weights of each pixel value in the neighboring coordinate point pixel matrix.

[0152] The vector operation acceleration unit 630 is configured to determine, according to the neighboring coordinate point pixel matrix and the interpolation weight matrix, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, to obtain the first target image and the second target image.

[0153] In a possible implementation manner, the geometric transformation includes one of an affine transformation and a perspective transformation.

[0154] The various functional modules in the accelerator 600 are used to implement the methods described on the accelerator side in the above method embodiments. For a specific description, refer to the description of the above method embodiments. For the sake of brevity of the description, no further description is given here.

[0155] The various functional modules in the accelerator 600 can be implemented by software or hardware. The division of the various functional modules is only an example. In actual applications, the network device can be divided into a larger or smaller number of functional modules, which is not limited in the present application.

[0156] The present application also provides a device with computing function, such as Figure 7 as shown in the figure, Figure 7 Fig. 7 is a structural schematic diagram of a device with computing function provided in the present application. The device with computing function 700 can be an accelerator in the above method embodiments, used to implement the methods described on the accelerator side in the above method embodiments. The device with computing function 700 can also be a physical machine including a processor, used to implement the methods described on the processor side in the above method embodiments. The device with computing function 700 can also be a physical machine including a processor and an accelerator, and the physical machine is used to implement all the steps in the above method embodiments.

[0157] The device with computing function 700 includes a bus 702, a processor 704, a memory 706 and a communication interface 708. The processor 704, the memory 706 and the communication interface 708 communicate through the bus 702. It should be understood that the number of processors and memories in the device with computing function 700 is not limited in the present application.

[0158] The bus 702 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of brevity of the description, Figure 7 only one line is used to represent it, but it does not mean that there is only one bus or only one type of bus. The bus 702 can include a path for transmitting information between various components (for example, the memory 706, the processor 704, the communication interface 708) of the device with computing function 700.

[0159] The processor 704 can include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), among other processors.

[0160] The memory 706 can include volatile memory, such as random access memory (RAM), among other types of volatile memory. The processor 704 can also include non-volatile memory, such as read-only memory (ROM), flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), among other types of non-volatile memory.

[0161] The executable code stored in the memory 706. When the device 700 is configured as an accelerator in the method embodiment, the processor 704 executes the executable code to implement the functions of the aforementioned acquisition unit 610, the matrix multiplication acceleration unit 620, and the vector operation acceleration unit 630, thereby implementing the method described in the accelerator side of the acceleration method for geometric transformation of a plurality of images based on a reference image. That is, the memory 706 stores instructions for executing the method described in the accelerator side of the acceleration method for geometric transformation of a digital image.

[0162] When the device 700 is configured as a physical machine, the physical machine does not include an accelerator, and the processor 704 executes the executable code to implement the functions of the processor in the method embodiment, thereby implementing the method described in the processor side of the acceleration method for geometric transformation of a plurality of images based on a reference image. That is, the memory 706 stores instructions for executing the method described in the processor side of the acceleration method for geometric transformation of a digital image.

[0163] Optionally, when the device 700 is configured as a physical machine, the physical machine includes an accelerator, and the processor 704 executes the executable code to implement the functions of the processor and the accelerator in the method embodiment, thereby implementing all steps of the method described in the acceleration method for geometric transformation of a plurality of images based on a reference image. That is, the memory 706 stores all instructions for executing the method described in the acceleration method for geometric transformation of a digital image.

[0164] The communication interface 708 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, etc., to enable communication between the computing-enabled device 700 and other devices or communication networks.

[0165] The embodiments of the present application also provide a computer program product containing instructions. The computer program product can be a software or program product containing instructions, which can be run on a computing-enabled device or stored in any available medium. When the computer program product is run on at least one computing-enabled device, the at least one computing-enabled device is caused to perform the method described in the method for accelerating geometric transformation of digital images.

[0166] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing-enabled device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state disk), etc. The computer-readable storage medium contains instructions, which instruct the computing-enabled device to perform the method described in the method for accelerating geometric transformation of digital images.

[0167] The embodiments of the present application also provide a chip, which can be used to implement part or all of the steps of the above-described method embodiments. For example, the chip can include a matrix multiplication acceleration unit and a vector operation acceleration unit, which can be used to implement part of the steps in the above-described method embodiments.

[0168] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An accelerated system for geometric transformation of digital images, characterized by, The digital images include at least a first original image and a second original image, the acceleration system includes a processor and an accelerator, The processor is configured to: determine a target transformation matrix, the target transformation matrix being used for geometric transformation between the first original image and a first target image, and geometric transformation between the second original image and a second target image, wherein the first target image and the second target image have the same size; determine each coordinate point on the first target image, each coordinate point on the second target image being the same as each coordinate point on the first target image; The accelerator is configured to: determine, according to the target transformation matrix and each coordinate point on the first target image, an original coordinate position matrix, the original coordinate position matrix including coordinate positions of each coordinate point on the first target image on the first original image and coordinate positions of each coordinate point on the second target image on the second original image; determine, according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

2. The acceleration system of claim 1, wherein, The accelerator is configured to: determine, according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, a neighboring coordinate point pixel matrix, the neighboring coordinate point pixel matrix including a matrix composed of pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image and pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the second target image on the second original image; determine an interpolation weight matrix, the interpolation weight matrix including weights of each pixel value in the neighboring coordinate point pixel matrix; determine, according to the neighboring coordinate point pixel matrix and the interpolation weight matrix, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

3. The acceleration system of claim 2, wherein, The accelerator includes a matrix multiplication acceleration unit and a vector operation acceleration unit, wherein The matrix multiplication acceleration unit is configured to determine the original coordinate position matrix and determine the interpolation weight matrix; The vector operation acceleration unit is configured to determine the neighboring coordinate point pixel matrix, and determine, according to the neighboring coordinate point pixel matrix and the interpolation weight matrix, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, and obtain the first target image and the second target image.

4. The acceleration system according to any one of claims 1 to 3, characterized in that The processor is configured to: determine a first transformation matrix, the first transformation matrix being used for geometric transformation between the first original image and the first target image, the first transformation matrix having a dimension of k*3; determining a second transformation matrix, the second transformation matrix being used for geometric transformation between the second original image and the second target image, the second transformation matrix having a dimension of k*3; stacking the first transformation matrix and the second transformation matrix in a row dimension to obtain the target transformation matrix, the target transformation matrix having a dimension of 2k*3.

5. The acceleration system of claim 4, wherein, the matrix composed of each coordinate point on the first target image has a dimension of 3*m, and the original coordinate position matrix has a dimension of 2k*m.

6. The acceleration system of claim 1, wherein, The geometric transformation includes one of an affine transformation and a perspective transformation.

7. An accelerated method of geometric transformation of digital images, characterized in that, The digital image at least includes a first original image and a second original image, and the method comprises: obtaining a target transformation matrix, the target transformation matrix being used for geometric transformation between the first original image and the first target image and geometric transformation between the second original image and the second target image, wherein the first target image and the second target image are of the same size; obtaining each coordinate point on the first target image, each coordinate point on the second target image being the same as each coordinate point on the first target image; determining an original coordinate position matrix according to the target transformation matrix and each coordinate point on the first target image, the original coordinate position matrix including coordinate positions of each coordinate point on the first target image on the first original image and coordinate positions of each coordinate point on the second target image on the second original image; determining pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, to obtain the first target image and the second target image.

8. The method of claim 7, wherein, The method for determining pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, to obtain the first target image and the second target image, comprises: determining a neighboring coordinate point pixel matrix according to the original coordinate position matrix, pixel values of each coordinate point on the first original image and pixel values of each coordinate point on the second original image, the neighboring coordinate point pixel matrix including a matrix composed of pixel values of neighboring coordinate points of coordinate positions of each coordinate point on the first target image on the first original image and pixel values of neighboring coordinate points of coordinate positions of each coordinate point on the second target image on the second original image; determining an interpolation weight matrix, the interpolation weight matrix including weights of each pixel value in the neighboring coordinate point pixel matrix; determining pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image according to the neighboring coordinate point pixel matrix and the interpolation weight matrix, to obtain the first target image and the second target image.

9. The method according to claim 7 or 8, characterized in that, The geometric transformation includes one of an affine transformation and a perspective transformation.

10. An acceleration device for geometric transformation of digital images, characterized in that The digital images include at least a first original image and a second original image, and the apparatus includes: an obtaining unit configured to obtain a target transformation matrix, the target transformation matrix being used for geometric transformation between the first original image and a first target image and geometric transformation between the second original image and a second target image, wherein the first target image and the second target image are of the same size; the obtaining unit is further configured to obtain each coordinate point on the first target image, each coordinate point on the second target image being the same as each coordinate point on the first target image; a matrix multiplication acceleration unit configured to determine, according to the target transformation matrix and each coordinate point on the first target image, an original coordinate position matrix, the original coordinate position matrix including coordinate positions of each coordinate point on the first target image on the first original image and coordinate positions of each coordinate point on the second target image on the second original image; a vector operation acceleration unit, which is used in cooperation with the matrix multiplication acceleration unit to determine, according to the original coordinate position matrix, pixel values of each coordinate point on the first target image and pixel values of each coordinate point on the second target image, and to obtain the first target image and the second target image.

11. The apparatus of claim 10, wherein the vector operation acceleration unit is configured to determine, according to the original coordinate position matrix, the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image, a neighboring coordinate point pixel matrix including a matrix of pixel values of neighboring coordinate points of the coordinate positions of each coordinate point on the first target image on the first original image and of the coordinate positions of each coordinate point on the second target image on the second original image; the matrix multiplication acceleration unit is configured to determine an interpolation weight matrix including weights of each pixel value in the neighboring coordinate point pixel matrix; the vector operation acceleration unit is configured to determine, according to the neighboring coordinate point pixel matrix and the interpolation weight matrix, the pixel values of each coordinate point on the first target image and the pixel values of each coordinate point on the second target image, and to obtain the first target image and the second target image.

12. The apparatus of claim 10 or 11, wherein, The geometric transformation includes one of an affine transformation and a perspective transformation.

13. A computer program product, characterised in that, program instructions, which, when executed on a device having computing functionality, cause the device having computing functionality to perform the method of any one of claims 7 to 9.

Images