An image processing method, apparatus, electronic device, and storage medium

Abstract

Embodiments of the present specification provide an image processing method and device, electronic equipment and storage medium, wherein the image processing method determines the mapping coordinates of the pixel points in a pixel block in units of pixel blocks. In this process, only the first pixel in the pixel block needs to be subjected to homographic transformation based on a homographic transformation formula to obtain its projection coordinates, and other pixel points (i.e., second pixels) in the pixel block can be determined through the positional relationship of the second pixels and the first pixel in the pixel block. In this way, when determining the mapping coordinates of the pixel points in the pixel block, the number of homographic transformations in the operation process can be greatly reduced (if a pixel block includes M pixel points, the number of homographic transformations can be reduced to 1 / M times), which is conducive to reducing the calculation amount required by the entire image processing method and improving the operation efficiency of the image processing method.

Description

Technical Field

[0001] This specification relates to homography transformation technology in the field of image processing, and more specifically, to an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Homography, a concept in projective geometry also known as projective transformation, is widely used in image registration and correction. Homography plays a crucial role in image registration, image stitching, camera pose estimation, and Simultaneous Localization and Mapping (SLAM).

[0003] Homography is a transformation that maps a point (a three-dimensional homogeneous vector) on one projective plane to another. Homography is a linear transformation of a three-dimensional homogeneous vector and can be represented by a 3×3 non-singular matrix H, which is called the homography matrix. Using this matrix, a point on one projective plane can be projected onto another plane.

[0004] Currently, image correction based on homography transformation suffers from high computational complexity and long processing time. It is necessary to improve the computational efficiency of image correction and other image processing processes. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an image processing method, apparatus, electronic device, and storage medium to reduce the computational load of the image processing method and improve computational efficiency.

[0006] To achieve the above technical objectives, the embodiments of this application provide the following technical solutions:

[0007] Firstly, embodiments of this specification provide an image processing method, including:

[0008] Acquire a first image, the first image comprising multiple pixel blocks, each pixel block comprising a first pixel and multiple second pixels;

[0009] Based on the homography transformation formula, the mapping coordinates of the first pixel are obtained, and the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image;

[0010] Based on the mapping coordinates and relative position relationship of the first pixel, the mapping coordinates of the second pixel are obtained. The relative position relationship is the positional relationship between the second pixel and the first pixel in the pixel block. The mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image.

[0011] Based on the mapped coordinates of pixels in the first image, the second image is interpolated to obtain the corrected target image.

[0012] Optionally, obtaining the mapping coordinates of the second pixel based on the mapping coordinates and relative positional relationship of the first pixel includes:

[0013] The mapping coordinates of the first pixel are superimposed with the relative distance to obtain the mapping coordinates of the second pixel, wherein the relative distance is the distance between the second pixel and the first pixel in the pixel block.

[0014] Optionally, the pixel block comprises m rows × n columns of pixels, where m is 4, 8, or 16, and n is 4, 8, or 16.

[0015] The step of interpolating the second image based on the mapped coordinates of pixels in the first image to obtain the corrected target image includes:

[0016] Based on the mapped coordinates of pixels in the first image, the second image is interpolated using single instruction stream multiple data stream (SILMS) technology to obtain a corrected target image. The mapped coordinates of the pixels include the mapped coordinates of the first pixel and the mapped coordinates of the second pixel.

[0017] Optionally, the step of interpolating the second image based on the mapped coordinates of pixels in the first image and using single-instruction-multiple-data-stream technology to obtain the corrected target image includes:

[0018] Based on the single instruction stream multiple data stream technology, the second image is subjected to linear interpolation in the first direction, taking a row of pixels in the pixel block as a unit, so as to obtain the interpolation points of a row of pixels in the pixel block in the first direction.

[0019] Based on the single instruction stream multiple data stream technique, the second image is linearly interpolated in the second direction using the interpolation points of a row of pixels in the pixel block in the first direction as units, so as to obtain the corrected target image, wherein the first direction and the second direction intersect.

[0020] Optionally, the step of performing linear interpolation processing in the second image in the first direction, based on the single instruction stream multiple data stream technique and taking a row of pixels in the pixel block as the unit, includes:

[0021] Obtain the first adjacent coordinates, which include the coordinates of the top left, top right, bottom left and bottom right pixels of the preset mapping coordinates, wherein the preset mapping coordinates are the mapping coordinates of the first pixel of a row of pixels in the pixel block;

[0022] N consecutive pixel values ​​starting from the first adjacent coordinates are loaded into the first vector register corresponding to the first adjacent coordinates; N is equal to the number of pixels in a row of pixels in the first image;

[0023] The first offset value and the second offset value are loaded into the second vector register and the third vector register, respectively. The first offset value is the offset of the interpolation point relative to the top-left pixel adjacent to the interpolation point in the first direction. The second offset value is the offset of the interpolation point relative to the top-left pixel adjacent to the interpolation point in the second direction. The interpolation point corresponds one-to-one with the pixels in the first image.

[0024] Using the values ​​loaded in the first vector register and the second vector register, the interpolation points of a row of pixels in the first image in the first direction are calculated according to the principle of linear interpolation.

[0025] Optionally, the step of performing linear interpolation processing on the second image in the second direction based on the single instruction stream multiple data stream technique, using the interpolation points of a row of pixels in the pixel block in the first direction as units, includes:

[0026] Using the value loaded in the third vector register and the interpolation points of a row of pixels in the first image in the first direction, the interpolation points of a row of pixels in the first image in the second direction are calculated according to the principle of linear interpolation.

[0027] Optionally, acquiring the first image includes:

[0028] A first image is obtained comprising multiple pixel blocks of the same size, wherein the first pixel in each pixel block is located at the same position in the pixel block.

[0029] Secondly, embodiments of this specification also provide an image processing apparatus, comprising:

[0030] An image acquisition module is used to acquire a first image, the first image including multiple pixel blocks, the pixel blocks including first pixels and multiple second pixels;

[0031] The homography transformation module is used to obtain the mapping coordinates of the first pixel based on the homography transformation formula, wherein the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image;

[0032] The coordinate calculation module is used to obtain the mapping coordinates of the second pixel based on the mapping coordinates and relative positional relationship of the first pixel, wherein the relative positional relationship is the positional relationship between the second pixel and the first pixel in the pixel block, and the mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image;

[0033] The image correction module is used to perform interpolation processing on the second image based on the mapping coordinates of pixels in the first image to obtain the corrected target image.

[0034] Thirdly, embodiments of this specification also provide an electronic device, including: a memory and a processor;

[0035] The memory is connected to the processor and is used to store programs;

[0036] The processor is configured to implement the image processing method as described above by running a program stored in the storage.

[0037] Fourthly, embodiments of this specification provide a storage medium storing a computer program, which, when executed by a processor, implements the image processing method as described in any of the preceding claims.

[0038] Fifthly, embodiments of this specification provide a computer program product or computer program, the computer program product or computer program including computer instructions stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the steps of the above-described image processing method.

[0039] As can be seen from the above technical solutions, the embodiments of this application provide an image processing method, apparatus, electronic device, and storage medium. The image processing method determines the mapping coordinates of pixels within a pixel block on a pixel block basis. In this process, only the first pixel within the pixel block needs to undergo a homography transformation based on the homography transformation formula to obtain its projected coordinates. The other pixels within the pixel block (i.e., the second pixel) can be determined through the positional relationship between the second pixel and the first pixel within the pixel block. Thus, when determining the mapping coordinates of pixels within a pixel block, the number of homography transformations, which involves complex computation, can be greatly reduced (if a pixel block contains M pixels, the number of homography transformations can be reduced to 1 / M times), which helps to reduce the computational load required by the entire image processing method and improve its computational efficiency. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0041] Figure 1 A schematic diagram of a linear interpolation provided in the embodiments of this specification;

[0042] Figure 2 This is a schematic diagram of a bilinear interpolation method provided in the embodiments of this specification;

[0043] Figure 3 This is a schematic flowchart of an image processing method provided in an embodiment of this specification;

[0044] Figure 4 A schematic diagram illustrating a method of dividing a first image as provided in an embodiment of this specification;

[0045] Figure 5 This is a schematic diagram illustrating the positional relationship between two pixels provided in an embodiment of this specification.

[0046] Figure 6 A schematic flowchart illustrating another image processing method provided in the embodiments of this specification;

[0047] Figure 7 This is a schematic diagram illustrating how to load N consecutive pixel values ​​starting from the first adjacent coordinates into a first vector register, as provided in an embodiment of this specification.

[0048] Figure 8 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this specification;

[0049] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0050] Unless otherwise defined, the technical or scientific terms used in the embodiments of this specification shall have the ordinary meaning understood by one of ordinary skill in the art to which this specification pertains. The terms "first," "second," and similar terms used in the embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to avoid confusion of constituent elements.

[0051] Unless the context otherwise requires, throughout this specification, "a plurality of" means "at least two," and "including" is interpreted as open-ended or encompassing, that is, "including, but not limited to." In the description of this specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this specification. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example.

[0052] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0053] The image processing method provided in the embodiments of this specification can be applied to the image correction process in application scenarios such as image registration, image stitching, camera pose estimation, and visual SLAM. By dividing the image into blocks to calculate the mapping coordinates and using single instruction stream multiple data stream technology for image correction, the computational load required by the image processing method is reduced, and the computation and execution efficiency of the image processing method is improved.

[0054] First, let's explain some of the terms used in this manual:

[0055] Homography transformation, as mentioned above, is a concept in projective geometry, also known as projective transformation. Homography transformation maps a point (a three-dimensional homogeneous vector) on one projective plane to another. It is a linear transformation of a three-dimensional homogeneous vector, which can be represented by a 3×3 non-singular matrix H, called the homography matrix. Using this matrix, a point on one projective plane can be projected onto another plane. Two points on the two planes with a projection relationship are called corresponding points. The homography matrix is ​​a transformation matrix that maps a point from one image to another. Traditional homography transformation methods typically calculate the mapping coordinates of each pixel in the reference image relative to the image to be corrected based on the homography transformation formula shown in formula (1). With the mapping coordinates, interpolated pixel values ​​can be obtained on the image to be corrected, completing the image correction process.

[0056] in,

[0057] In formula (1), (x1, y1, 1) and (x2, y2, 1) represent the homogeneous coordinates of corresponding points on the two images, respectively, and h ij , i = 1, 2, 3; j = 1, 2, 3 represent the parameters in the homography matrix.

[0058] Linear interpolation is a method of determining the value of an unknown quantity by using a straight line connecting two known quantities. (Reference) Figure 1Suppose we know the coordinates (x0, y0) and (x1, y1), and to find the value of x at a certain position within the interval on the straight line, we obtain the two-point form of the straight line equation:

[0059]

[0060] Assuming the value on both sides of the equation is α, then this value is the interpolation coefficient (i.e., the ratio of the distance from x0 to x to the distance from x0 to x1). Since the value of x is known, the value of α can be obtained from the formula:

[0061]

[0062] Thus, formula (2) above can be expressed as:

[0063] y=(1-α)y0+αy1 (4)

[0064] Thus, y can be obtained through α. The process of finding x given y is similar to the above, and will not be elaborated upon here.

[0065] Bilinear interpolation, also known as bilinear interpolation, is mathematically an extension of the linear interpolation function of two variables. Its core idea is to perform linear interpolation in two directions (e.g., the x-axis and y-axis).

[0066] refer to Figure 2 , Figure 2 China Q 11 Q 12 Q 21 and Q 22 These are four known pixels, with point P being the point to be interpolated.

[0067] In bilinear interpolation, a linear interpolation can be performed first in one direction, for example, linear interpolation can be performed first in the X direction, and then in the Q direction... 12 and Q 22 Insert interpolation point R2 in Q 11 and Q 21 Insert the blue dot R1 in the middle.

[0068] Then, linear interpolation is performed in the Y direction, and point P is calculated by interpolating R1 and R2 calculated in the first step in the Y direction.

[0069] Bicubic interpolation is a more complex interpolation method that produces smoother image edges than bilinear interpolation. In bicubic interpolation, the value of the function f at the point (x,y) is obtained by weighted averaging of the sixteen nearest samples in a rectangular grid. This requires the use of two polynomial cubic interpolation functions, one for each direction.

[0070] Bicubic interpolation, also known as bicubic interpolation, is a method used to "interpolate" or increase the number / density of "pixels" in an image. Interpolation techniques are typically used to increase graphic data so that when it is printed or output in other forms, the print area and / or resolution can be increased.

[0071] Regardless of the interpolation method used, it is necessary to obtain the mapping coordinates of each pixel in the reference image to the image to be corrected. However, the computational load required to perform a homography transformation on each pixel using a relatively complex homography matrix is ​​large, which makes the image processing time long. This results in poor applicability of related products for application scenarios with high timeliness requirements or relatively low computing power.

[0072] To address this issue, this specification provides an image processing method. As described above, it simplifies the calculation process of mapping coordinates by dividing the image into blocks, thereby reducing the computational load required for the image processing method and improving processing efficiency.

[0073] Exemplary methods

[0074] This specification provides an image processing method through its embodiments, such as... Figure 3 As shown, the image processing method includes:

[0075] S101: Acquire a first image, the first image comprising multiple pixel blocks, the pixel blocks comprising a first pixel and multiple second pixels.

[0076] The first image is the image awaiting transformation into the second image, or it can also be called the reference image. Before obtaining the mapping coordinates of the first image, it can be divided into pixel blocks. Each pixel block can include a first pixel and multiple second pixels. The first pixel can be called the base point of the pixel block. In the pixel block, the positional relationship between the second pixel and the first pixel can be used as the basis for subsequently determining the mapping coordinates of the second pixel.

[0077] Optionally, the pixels (first pixel and second pixel) in the pixel block can be arranged in a matrix. More specifically, the pixels in the pixel block can be arranged in a rectangular matrix of m rows × n columns. The pixel blocks arranged in a matrix make the shape of the pixel block more regular, which facilitates the complete division of the first image and makes it easier to determine the positional relationship between the second pixel and the first pixel.

[0078] For example, such as Figure 4 As shown, Figure 4 This illustrates a method for dividing a first image, and the arrangement of pixels within a pixel block. Figure 4 In the first image, the image is divided into four pixel blocks (pixel block 1, pixel block 2, pixel block 3, and pixel block 4). Each pixel block is the same size, and the pixels within each block are arranged in a 4x4 rectangular matrix. These pixels are assigned the number P. i，j The first image is represented as follows, where i = 0, 1, 2, 3; j = 0, 1, 2, 3. It is understood that the size of the pixel blocks in the first image can vary, and the arrangement of pixels within each pixel block can also be different (e.g., a circular matrix) to meet the requirements of different application scenarios. For example, when the first image is an irregular image, including pixel blocks of different sizes and shapes may be more beneficial for dividing the pixels in the first image. Therefore, this specification does not limit the size of the pixel blocks in the first image or the arrangement of pixels within each pixel block; the specific arrangement depends on the actual situation.

[0079] To simplify the determination of subsequent relative positional relationships and reduce the complexity of coordinate calculations, in one embodiment of this specification, step S101 includes:

[0080] A first image is obtained comprising multiple pixel blocks of the same size, wherein the first pixel in each pixel block is located at the same position in the pixel block.

[0081] When the pixel blocks in the first image are of the same size (i.e., the width and height of the pixel blocks are the same), the transformation accuracy of the mapped coordinates can remain consistent, and the block division logic of the first image is simplified, which is beneficial to improving the calculation speed. Furthermore, having the first pixel in each pixel block located at the same position within the pixel block (e.g., at the top left corner of the pixel block) ensures that the positional relationship between the second pixel and the first pixel in each pixel block is fixed. This simplifies the subsequent calculation of the mapped coordinates of the second pixel based on the mapped coordinates of the first pixel and their relative positional relationship, thus improving computational efficiency.

[0082] S102: Based on the homography transformation formula, obtain the mapping coordinates of the first pixel, where the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image.

[0083] As mentioned above, the homography matrix can be used to perform a homography transformation on the first pixel based on the homography transformation formula in the form of formula (1) to obtain the mapping coordinates of the first pixel.

[0084] S103: Based on the mapping coordinates and relative position relationship of the first pixel, obtain the mapping coordinates of the second pixel, wherein the relative position relationship is the positional relationship between the second pixel and the first pixel in the pixel block, and the mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image.

[0085] It's easy to understand that once the pixel block of the first pixel is determined, the positional relationships of the individual pixels within that block are also determined. The positions of each pixel can be represented using coordinates in an XY coordinate system or a polar coordinate system. The positional relationship between two pixels can be determined by their coordinates in the same coordinate system. (Reference) Figure 5 , Figure 5 The diagram shows the coordinates of two pixels in the XY coordinate system. The coordinates of each pixel can be represented by the coordinates of its center (O1 and O2). Figure 5 The coordinates of two pixels O1 and O2 are (x1, y1) and (x2, y2), respectively. The positional relationship between the two pixels can be expressed as follows: the horizontal distance between pixels O1 and O2 is x2 - x1, and the vertical distance between them is y2 - y1. Furthermore, the positional relationship between these two pixels can also be expressed as the linear distance between them: This manual does not limit the specific way in which the positional relationship between pixels is represented; the specific method depends on the actual situation.

[0086] Once the mapping coordinates of the first pixel are determined, the mapping coordinates of the second pixel can be determined based on the relative positional relationship and the mapping coordinates of the first pixel. This process avoids repeating the complex homography transformation, reducing the required computation, lowering the computational requirements, and improving computational efficiency. It's easy to understand that if each pixel block contains m×n pixels, the number of homography operations required for the first image can be reduced to 1 / (m×n) of the original number.

[0087] Optionally, one embodiment of this specification provides a feasible method for obtaining the mapping coordinates of the second pixel based on the mapping coordinates and relative positional relationship of the first pixel, specifically including:

[0088] S1031: The mapping coordinates of the first pixel are superimposed with the relative distance to obtain the mapping coordinates of the second pixel, wherein the relative distance is the distance between the second pixel and the first pixel in the pixel block.

[0089] The relative distance can include the relative distance along each coordinate axis in the coordinate system containing the first and second pixels. Still using... Figure 5 For example, assuming pixel O1 is the first pixel and pixel O2 is the second pixel, the relative distance between the first and second pixels includes the X-axis distance and the Y-axis distance. Assuming the coordinates of the first pixel are (x1, y1) and the coordinates of the second pixel are (x2, y2), then the X-axis distance is x2 - x1, and the Y-axis distance is y2 - y1. If the mapped coordinates of the first pixel are (xi, yi), then the mapped coordinates of the second pixel can be calculated based on the above X-axis and Y-axis distances. That is, the mapped coordinates of the second pixel can be expressed as: (xi + x2 - x1, yi + y2 - y1). This method allows for a relatively convenient calculation of the mapped coordinates of the second pixel, which helps to simplify the algorithm's complexity.

[0090] S104: Based on the mapping coordinates of the pixels in the first image, perform interpolation processing on the second image to obtain the corrected target image.

[0091] After obtaining the mapped coordinates of the pixels in the first image, in order to resolve the misalignment between the first and second images, it is necessary to perform interpolation processing on the second image based on the mapped coordinates of each pixel to correct the pixel positions and resolve the misalignment problem between the first and second images.

[0092] As mentioned above, the interpolation process can be performed using linear interpolation, bilinear interpolation, bicubic interpolation, or other interpolation methods, and this specification does not limit the specific interpolation method used.

[0093] To further optimize the interpolation process, embodiments of this specification also provide an image processing method, such as... Figure 6 As shown, it includes:

[0094] S201: Acquire a first image, the first image comprising multiple pixel blocks, each pixel block comprising a first pixel and multiple second pixels, the first pixel and multiple second pixels being arranged in an m-row × n-column configuration, where m is 4, 8 or 16 and n is 4, 8 or 16.

[0095] In order to meet the bit width requirements of the Single Instruction Stream Multiple Data Stream (SIMDL) technology, the size of the pixel blocks in the first image is limited in step S201.

[0096] S202: Based on the homography transformation formula, obtain the mapping coordinates of the first pixel, where the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image.

[0097] S203: Based on the mapping coordinates and relative position relationship of the first pixel, obtain the mapping coordinates of the second pixel, wherein the relative position relationship is the positional relationship between the second pixel and the first pixel in the pixel block, and the mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image.

[0098] S204: Based on the mapping coordinates of pixels in the first image, interpolation processing is performed on the second image using single instruction stream multiple data stream technology to obtain a corrected target image. The mapping coordinates of the pixels include the mapping coordinates of the first pixel and the mapping coordinates of the second pixel.

[0099] Single Instruction Multiple Data (SIMD) technology is a technique that uses a single controller to control multiple processors, which simultaneously perform the same operation on each of a set of data (also known as a "data vector") to achieve spatial parallelism.

[0100] In this embodiment, the interpolation process is based on the single instruction stream multiple data stream technique, which allows for parallel processing of multiple mapped coordinates of pixels in the first pixel, thus improving the algorithm's running efficiency.

[0101] Specifically, step S204 may include:

[0102] S2041: Based on the single instruction stream multiple data stream technology, the second image is subjected to linear interpolation in the first direction, taking a row of pixels in the pixel block as a unit, to obtain the interpolation points of a row of pixels in the pixel block in the first direction.

[0103] S2042: Based on the single instruction stream multiple data stream technology, the second image is linearly interpolated in the second direction using the interpolation points of a row of pixels in the pixel block in the first direction as units, so as to obtain the corrected target image, wherein the first direction and the second direction intersect.

[0104] In this embodiment, based on the single instruction stream multiple data stream technique, linear interpolation in the first or second direction can be performed simultaneously on a row of pixels in a pixel block. Taking an 8-row × 8-column pixel block as an example, bilinear interpolation of a row of 8 pixels can be completed at once, which helps to improve the algorithm's running efficiency.

[0105] The following uses the ARM NEON instruction set as an example to illustrate the specific feasible execution methods of steps S2041 and S2042.

[0106] Optionally, step S2041 includes:

[0107] S20411: Obtain the first adjacent coordinates, the first adjacent coordinates including the coordinates of the top left, top right, bottom left and bottom right pixels of the preset mapping coordinates, the preset mapping coordinates being the mapping coordinates of the first pixel of a row of pixels in the pixel block.

[0108] In step S20411, firstly, the first pixel in a row of pixels in the pixel block is found. Then, the mapped coordinates of the first pixel are rounded to obtain the coordinates of the top-left pixel adjacent to (or immediately next to) the first pixel's interpolation position. For example, it can be denoted as P. m，n Then, by performing simple calculations on the horizontal and vertical coordinates based on the coordinates of the pixel at the top left corner, the top right, top left, and bottom right pixels with preset mapping coordinates can be obtained. These pixels can be denoted as P. m+1，n P m，n+1 P m+1，n+1 For example, suppose the mapped coordinates of the first pixel in a row of a pixel block are (7.05, 8.73). Rounding down gives the coordinates of the top-left pixel (7,8). Adding 1 to the ordinate of (7,8) gives the coordinates of the bottom-left pixel (7,9). Adding 1 to the x-coordinate of (7,8) gives the coordinates of the top-right pixel (8,8). Adding 1 to both the x and ordinate of (7,8) gives the coordinates of the bottom-right pixel (8,9).

[0109] S20412: Load N consecutive pixel values ​​starting from the first adjacent coordinates into the first vector register corresponding to the first adjacent coordinates; N is equal to the number of pixels in a row of pixels in the first image.

[0110] Still assuming a pixel block comprises 8 rows x 8 columns of pixels, with P being the first neighboring coordinate of the first pixel in a given row. m，n P m+1，n P m，n+1 P m+1，n+1 Taking these four pixels as an example, refer to Figure 7 , respectively with P m，n P m+1，n P m，n+1 P m+1，n+1 The eight consecutive pixel values ​​starting from these four pixels are respectively loaded into the first vector register V of the four NEON modules. LU V RU VLD and V RD In China, it will soon be P m，n The first vector register V is loaded with 8 consecutive pixel values ​​starting from the first vector register V. LU In the middle, P will be used m+1，n The first vector register V is loaded with 8 consecutive pixel values ​​starting from the first vector register V. RU In the middle, P will be used m，n+1 The first vector register V is loaded with 8 consecutive pixel values ​​starting from the first vector register V. LD In the middle, P will be used m+1，n+1 The first vector register V is loaded with 8 consecutive pixel values ​​starting from the first vector register V. RD Since image pixel values ​​are typically 8-bit, the first vector register can be an int8 vector register.

[0111] S20413: Load the first offset value and the second offset value into the second vector register and the third vector register respectively. The first offset value is the offset of the interpolation point relative to the top-left pixel adjacent to the interpolation point in the first direction. The second offset value is the offset of the interpolation point relative to the top-left pixel adjacent to the interpolation point in the second direction. The interpolation point corresponds one-to-one with the pixels in the first image.

[0112] As described above regarding the linear interpolation principle, the offsets dx and dy of the interpolation point of a row of pixels relative to its immediate upper-left corner pixel are equal in the X and Y directions. Therefore, the first adjacent coordinate is still taken as P. m，n P m+1，n P m，n+1 P m+1，n+1 Taking these four pixels as an example, if P m，n The mapped coordinates are (X m,n ,Y m,n ),but

[0113] dx=X m,n -m;

[0114] dy = Y m,n -n;

[0115] Since the offsets dx and dy are both floating-point types, the second vector register and the third vector register can both be float32 vector registers of the NEON module.

[0116] S20414: Using the values ​​loaded in the first vector register and the second vector register, calculate the interpolation point of a row of pixels in the first image in the first direction according to the principle of linear interpolation.

[0117] Based on the linear interpolation principle described above, specifically, step S20414 may include:

[0118] V x1 =(1-V dx V LU +V dx V RU

[0119] V x2 =(1-V dx V LD +V dx V RD

[0120] Among them, V x1 V x2 This represents two interpolation points of a row of pixels in the pixel block in the first direction. After obtaining the interpolation points of a row of pixels in the first direction, step S20421 is executed to perform linear interpolation in the second direction, completing the quadratic linear interpolation process of a row of pixels.

[0121] S20421: Using the value loaded in the third vector register and the interpolation points of a row of pixels in the first image in the first direction, calculate the interpolation points of a row of pixels in the first image in the second direction according to the principle of linear interpolation.

[0122] Similarly, based on the linear interpolation principle described above, step S20421 may specifically include:

[0123] V R =(1-V dy V x1 +V dy V x2

[0124] This completes the bilinear interpolation of one row of pixels in the pixel block. The other rows of pixels in the pixel block can be calculated in the same way. This method allows the pixel interpolation process to be performed in parallel, which helps to improve the execution efficiency of the algorithm.

[0125] To verify the image processing method provided in the embodiments of this specification, in a specific embodiment of this specification, image processing (correction) was performed on a 4000×3000 YUV420 format image on the Qualcomm 765G platform based on the Android system environment. Eight consecutive corrections were performed using both existing methods and the image processing method provided in the embodiments of this specification. The experimental data are shown in Table 1.

[0126] Table 1 Comparative experimental data

[0127]

[0128] As can be seen from Table 1, the image processing method provided in the embodiments of this specification can significantly shorten the algorithm running time and improve the algorithm running efficiency compared with the existing image correction method (existing method). This makes the image processing method provided in the embodiments of this specification applicable to devices with relatively weak computing power or application scenarios with high requirements for algorithm real-time performance, thus improving the applicability of the image processing method.

[0129] Exemplary device

[0130] Corresponding to the image processing method described above, this application also provides an image processing apparatus, see [link to relevant documentation]. Figure 8 As shown, the device includes:

[0131] The image acquisition module 100 is used to acquire a first image, the first image including multiple pixel blocks, each pixel block including a first pixel and multiple second pixels.

[0132] The homography transformation module 200 is used to obtain the mapping coordinates of the first pixel based on the homography transformation formula, wherein the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image.

[0133] The coordinate calculation module 300 is used to obtain the mapping coordinates of the second pixel based on the mapping coordinates and relative position relationship of the first pixel, wherein the relative position relationship is the positional relationship between the second pixel and the first pixel in the pixel block, and the mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image.

[0134] The image correction module 400 is used to perform interpolation processing on the second image based on the mapping coordinates of pixels in the first image to obtain the corrected target image.

[0135] The image processing apparatus provided in this embodiment belongs to the same concept as the image processing method provided in the above embodiments of this application. It can execute the image processing method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the image processing method. Technical details not described in detail in this embodiment can be found in the specific processing content of the image processing method provided in the above embodiments of this application, and will not be repeated here.

[0136] Exemplary electronic devices

[0137] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 9As shown, an exemplary embodiment of this specification also provides an electronic device, including: a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the steps of the image processing method according to various embodiments of this specification described above.

[0138] The internal structure of the electronic device can be as follows: Figure 9 As shown, the electronic device includes a processor, memory, network interface, and input device connected via a system bus. The processor provides computing and control capabilities. The memory of the central control device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface of the electronic device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it follows the steps of the image processing method according to various embodiments of this specification as described in the above embodiments.

[0139] The processor may include the main processor, as well as baseband chips, modems, etc.

[0140] The memory stores a program that executes the technical solution of this invention, and may also store an operating system and other critical business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0141] The processor can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0142] Input devices may include devices that receive data and information input by the user, such as keyboards, mice, cameras, scanners, light pens, voice input devices, touch screens, pedometers, or gravity sensors.

[0143] Output devices may include devices that allow information to be output to the user, such as displays, printers, speakers, etc.

[0144] The communication interface may include any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0145] The processor executes the program stored in the memory and calls other devices, which can be used to implement the various steps of any of the image processing methods provided in the above embodiments of this application.

[0146] The electronic device may also include a display component and a voice component. The display component may be a liquid crystal display screen or an e-ink display screen. The input device of the electronic device may be a touch layer covering the display component, or a button, trackball or touchpad set on the casing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0147] Those skilled in the art will understand that Figure 9 The structures shown are merely block diagrams of a portion of the structure related to the scheme described in this specification, and do not constitute a limitation on the electronic devices to which the scheme described in this specification is applied. Specific electronic devices may include more or fewer components than those shown in the figures, or may combine certain components, or may have different component arrangements.

[0148] Exemplary computer program products and storage media

[0149] In addition to the methods and devices described above, the image processing methods provided in the embodiments of this specification can also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image processing methods according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0150] The computer program product described herein can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments described herein. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0151] Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor of the steps in the image processing methods according to various embodiments of this specification as described in the "Exemplary Methods" section above.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementation methods outlined in this specification. While the descriptions are specific and detailed, they should not be construed as limiting the scope of the solutions provided in this specification. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this specification, and these all fall within the scope of protection of this specification. Therefore, the scope of protection for this patent should be determined by the appended claims.

Claims

1. An image processing method, characterized in that, include: A first image is acquired, the first image comprising multiple pixel blocks, each pixel block comprising a first pixel and multiple second pixels; the pixel block is a two-dimensional pixel block comprising m rows × n columns of pixels; Based on the homography transformation formula, the mapping coordinates of the first pixel are obtained, and the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image; Based on the mapping coordinates and relative position relationship of the first pixel, the mapping coordinates of the second pixel are obtained. The relative position relationship is the positional relationship between the second pixel and the first pixel in the pixel block. The mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image. Based on the mapped coordinates of pixels in the first image, the second image is interpolated to obtain the corrected target image; The step of interpolating the second image based on the mapped coordinates of pixels in the first image to obtain the corrected target image includes: Based on the single instruction stream multiple data stream technology, the second image is subjected to linear interpolation in the first direction, taking a row of pixels in the pixel block as a unit, so as to obtain the interpolation points of a row of pixels in the pixel block in the first direction. Based on the single instruction stream multiple data stream technique, the second image is linearly interpolated in the second direction using the interpolation points of a row of pixels in the pixel block in the first direction as units, so as to obtain the corrected target image, wherein the first direction and the second direction intersect.

2. The method according to claim 1, characterized in that, Obtaining the mapping coordinates of the second pixel based on the mapping coordinates and relative position relationship of the first pixel includes: The mapping coordinates of the first pixel are superimposed with the relative distance to obtain the mapping coordinates of the second pixel, wherein the relative distance is the distance between the second pixel and the first pixel in the pixel block.

3. The method according to claim 1, characterized in that, The value of m is 4, 8, or 16, and the value of n is 4, 8, or 16. The mapped coordinates of the pixel include the mapped coordinates of the first pixel and the mapped coordinates of the second pixel.

4. The method according to claim 3, characterized in that, The method of performing linear interpolation in the first direction on the second image based on the single instruction stream multiple data stream technique, taking a row of pixels in the pixel block as the unit, includes: Obtain the first adjacent coordinates, which include the coordinates of the top left, top right, bottom left and bottom right pixels of the preset mapping coordinates, wherein the preset mapping coordinates are the mapping coordinates of the first pixel of a row of pixels in the pixel block; N consecutive pixel values ​​starting from the first adjacent coordinates are loaded into the first vector register corresponding to the first adjacent coordinates; N is equal to the number of pixels in a row of pixels in the first image; The first offset value and the second offset value are loaded into the second vector register and the third vector register, respectively. The first offset value is the offset of the interpolation point relative to the top-left pixel adjacent to the interpolation point in the first direction. The second offset value is the offset of the interpolation point relative to the top-left pixel adjacent to the interpolation point in the second direction. The interpolation point corresponds one-to-one with the pixels in the first image. Using the values ​​loaded in the first vector register and the second vector register, the interpolation points of a row of pixels in the first image in the first direction are calculated according to the principle of linear interpolation.

5. The method according to claim 4, characterized in that, The method of performing linear interpolation processing on the second image in the second direction based on the single instruction stream multiple data stream technique, using the interpolation points of a row of pixels in the pixel block in the first direction as units, includes: Using the value loaded in the third vector register and the interpolation points of a row of pixels in the first image in the first direction, the interpolation points of a row of pixels in the first image in the second direction are calculated according to the principle of linear interpolation.

6. The method according to any one of claims 1-5, characterized in that, The acquisition of the first image includes: A first image is obtained comprising multiple pixel blocks of the same size, wherein the first pixel in each pixel block is located at the same position in the pixel block.

7. An image processing apparatus, characterized in that, include: An image acquisition module is used to acquire a first image, the first image comprising multiple pixel blocks, each pixel block comprising a first pixel and multiple second pixels; the pixel block is a two-dimensional pixel block comprising m rows × n columns of pixels; The homography transformation module is used to obtain the mapping coordinates of the first pixel based on the homography transformation formula, wherein the mapping coordinates of the first pixel are the projection coordinates of the first pixel onto the second image; The coordinate calculation module is used to obtain the mapping coordinates of the second pixel based on the mapping coordinates and relative positional relationship of the first pixel, wherein the relative positional relationship is the positional relationship between the second pixel and the first pixel in the pixel block, and the mapping coordinates of the second pixel are the projection coordinates of the second pixel onto the second image; The image correction module is used to perform interpolation processing on the second image based on the mapped coordinates of pixels in the first image to obtain the corrected target image; The image correction module is specifically used for: Based on the single instruction stream multiple data stream technology, the second image is subjected to linear interpolation in the first direction, taking a row of pixels in the pixel block as a unit, so as to obtain the interpolation points of a row of pixels in the pixel block in the first direction. Based on the single instruction stream multiple data stream technique, the second image is linearly interpolated in the second direction using the interpolation points of a row of pixels in the pixel block in the first direction as units, so as to obtain the corrected target image, wherein the first direction and the second direction intersect.

8. An electronic device, characterized in that, include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the image processing method as described in any one of claims 1-6 by running a program stored in the memory.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1-6.

Images