Image transformation method, system, chip and storage medium

Abstract

The application discloses an image transformation method, system, chip and storage medium. The application improves the calculation performance by performing coordinate transformation on output pixel points in an output image in advance. Specifically, the corresponding target input pixel points are obtained by performing coordinate transformation on the output pixel points in advance, and the target input pixel points obtained by transformation are used to manage the local cache so that the corresponding target image data is stored in the local cache. Therefore, when subsequent target image processing is required for the output pixel points, the target image data corresponding to the output pixel points has been stored in the local cache, so it is not necessary to wait for the local cache to read and store the corresponding target image data from a storage entity, time delay is reduced, processing efficiency is improved, and the design index of high performance can be met.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image transformation method, system, chip, and storage medium. Background Technology

[0002] Image transformations are widely used in 3D vision technology. Image geometric transformations mainly include stretching, shrinking, distorting, and rotating, and are primarily categorized into affine transformations and perspective transformations. Image geometric transformations, also known as image space transformations, are used to map coordinate positions in one image to new coordinate positions in another image. This mapping can be divided into forward mapping and backward mapping. For backward mapping, when the transformed coordinate position is obtained, the image data of the pixel at the transformed coordinate position and its surrounding area generally needs to be read from storage entities such as DDR (Double Data Rate Synchronous Dynamic Random Access Memory) or Shared Memory, cached locally, and then the pixel at the original coordinate position is directly assigned a value or assigned a value after interpolation based on the cached image data.

[0003] Because the hardware system requires a certain latency to read image data from storage, each read of the transformed pixel location and its surrounding image data wastes time, resulting in a significant degrade in the overall hardware system's computational performance and making it difficult to meet high-performance design specifications. Furthermore, for backward mapping, each pixel in the output image is mapped to a pixel in the input image. This mapping relationship can be one-to-one or many-to-one. When the mapping relationship is many-to-one, the image data of a certain point in the input image needs to be cached locally multiple times from storage. Moreover, it's common practice to cache a certain point in the input image and its surrounding image data multiple times locally, resulting in an even larger amount of repeatedly cached data. This repeated caching severely wastes system bandwidth, leading to an increase in overall bandwidth usage. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of low performance and large bandwidth caused by the caching strategy used in the prior art for image transformation, and to provide an image transformation method, system, chip and storage medium.

[0005] The present invention solves the above-mentioned technical problems through the following technical solutions.

[0006] An image transformation method includes: performing pre-transformation processing on output pixels in an output image according to a preset order, wherein the pre-transformation processing includes: determining target input pixels in an input image corresponding to the output pixels in order N according to a mapping relationship; managing a local cache according to the target input pixels; and, if the target image data corresponding to the target input pixels is not stored in the local cache, writing the target image data corresponding to the target input pixels stored in a storage entity into the local cache, so that the local cache stores the target image data corresponding to the target input pixels, wherein N is a natural number, and the mapping relationship is represented by... To characterize the coordinate transformation relationship between the output image and the input image, the target image data corresponding to the target input pixel includes data of several neighboring input pixels located in the vicinity of the target input pixel in the input image; image processing is performed on the output pixels in the output image according to the preset order, wherein the image processing includes: reading the target image data corresponding to the target input pixel from the local cache, and performing target image processing on the output pixels in order N according to the target image data corresponding to the target input pixel, wherein the progress of the pre-transformation processing is no later than the progress of the image processing.

[0007] Preferably, the image processing further includes: determining whether N is not greater than the most recent order of the local cache notification, wherein the most recent order represents the order of the output pixels corresponding to the most recent input pixel used to manage the local cache; if so, then performing the step of reading the target image data corresponding to the target input pixel from the local cache.

[0008] Preferably, the local cache stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels. Reading the target image data corresponding to the target input pixel from the local cache includes: determining the target input pixel in the input image corresponding to the output pixel in the order N according to the mapping relationship; and reading the candidate image data whose coordinates are the same as those of the target input pixel from the local cache as the target image data.

[0009] Preferably, determining the target input pixel corresponding to the output pixels in the input image in order N based on the mapping relationship includes: determining the coordinates of the target input pixel based on the mapping relationship and the coordinates of the output pixels in order N; or, determining the coordinates of the target input pixel based on the relative positions of the output pixels in order N in the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image, wherein: the output image is divided into multiple output blocks, each output block includes multiple output pixels, and the vertex coordinates of the target input block are determined according to the mapping relationship. The mapping relationship and the vertex coordinates of the target output block are determined. Alternatively, the output image is divided into multiple output blocks, each output block includes multiple output pixels, the output image corresponds to an output grid map including multiple grid points, the input image corresponds to an input grid map including multiple grid points, the output grid map and the input grid map satisfy the mapping relationship, the vertex coordinates of the target input block are determined according to the coordinates of several grid points in the vicinity of the target grid point corresponding to the vertex of the target input block, and the target grid point is determined according to the position of the corresponding vertex in the target output block in the output grid map.

[0010] Preferably, the start time of the pre-transformation process is earlier than the start time of the image processing, and / or the throughput of the pre-transformation process is higher than the throughput of the image processing.

[0011] Preferably, the local cache stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels. Managing the local cache according to the target input pixel includes: determining whether the local cache stores target image data corresponding to the target input pixel based on the coordinates of the target input pixel and the coordinates stored in the local cache; if not, writing the target image data corresponding to the target input pixel stored in the storage entity into the local cache; or, the local cache stores candidate image data corresponding to candidate input pixels and the order of the output pixels corresponding to the candidate input pixels. Managing the local cache according to the target input pixel includes: determining whether the local cache stores target image data corresponding to the target input pixel based on N and the order stored in the local cache; if not, writing the target image data corresponding to the target input pixel stored in the storage entity into the local cache.

[0012] Preferably, the step of managing the local cache according to the target input pixel includes: managing the local cache according to a PLRU policy or an LRU policy; and / or, the local cache stores candidate image data corresponding to candidate input pixels and the order of the output pixels corresponding to the candidate input pixels, and the step of reading the target image data corresponding to the target input pixel from the local cache includes: reading the candidate image data with the corresponding order N from the local cache as the target image data.

[0013] An image transformation system includes: a pre-transformation processing module, configured to pre-transform output pixels in an output image according to a preset order, specifically configured to determine target input pixels in an input image corresponding to the output pixels in order N according to a mapping relationship, wherein N is a natural number, and the mapping relationship is used to characterize the coordinate transformation relationship between the output image and the input image; and a local cache, configured to manage data based on the target input pixels, so that when the target image data corresponding to the target input pixels is not stored, the target image data corresponding to the target input pixels stored in the storage entity is written to the cache, so that the cached target image data is stored to the target input pixels. The target image data corresponding to the target input pixel includes data of several neighboring input pixels located in the vicinity of the target input pixel in the input image; the image processing module is used to perform image processing on the output pixels in the output image according to the preset order, specifically to read the target image data corresponding to the target input pixel from the local cache, and to perform target image processing on the output pixels in order N according to the target image data corresponding to the target input pixel, wherein the progress of the pre-transformation processing module is no later than the progress of the image processing module.

[0014] A chip includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements any of the above-described image transformation methods.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described image transformation methods.

[0016] The positive and progressive effects of this invention are as follows: This invention improves computational performance by pre-transforming the coordinates of the output pixels in the output image. Specifically, the corresponding target input pixels are obtained by pre-transforming the coordinates of the output pixels. The transformed target input pixels are used to manage the local cache so that the corresponding target image data is stored therein. Thus, when the output pixel needs to be processed for target image processing in the future, the target image data corresponding to the output pixel has already been stored in the local cache. Therefore, there is no need to wait for the local cache to read and store the corresponding target image data from the storage entity, which reduces time delay, speeds up processing efficiency, and thus meets the design specifications for high performance. Attached Figure Description

[0017] Figure 1 This is a flowchart of the image transformation method according to Embodiment 1 of the present invention.

[0018] Figure 2 This is a schematic diagram corresponding to Method 3 in the image transformation method of Embodiment 1 of the present invention.

[0019] Figure 3 This is a schematic diagram of the image transformation system according to Embodiment 2 of the present invention. Detailed Implementation

[0020] The present invention will be further illustrated below by way of embodiments, but the invention is not limited to the scope of the embodiments described herein.

[0021] This embodiment provides an image transformation method, and further, an image geometric transformation method for mapping coordinate positions in one image to new coordinate positions in another image. Specifically, it can be applied to scenarios such as lens distortion correction, image registration, panoramic stitching, EIS image stabilization, and SLAM. In this embodiment, the image transformation method is preferably based on backward mapping. Figure 1 A flowchart of this embodiment is shown.

[0022] Reference Figure 1 The image transformation method of this embodiment includes: S11, performing pre-transformation processing on the output pixels in the output image according to a preset order, wherein the pre-transformation processing includes: determining the target input pixel in the input image corresponding to the output pixel in the order of N according to the mapping relationship, managing the local cache according to the target input pixel, so that if the target image data corresponding to the target input pixel is not stored in the local cache, the target image data corresponding to the target input pixel stored in the storage entity is written into the local cache, so that the target image data corresponding to the target input pixel is stored in the local cache.

[0023] In this embodiment, the output pixels in the output image are pre-transformed according to a preset order. Specifically: Based on the mapping relationship, the target input pixel 0 corresponding to output pixel 0 (i.e., the output pixel in sequence 0) in the input image is determined. The local cache is managed based on target input pixel 0, so that the local cache stores the target image data corresponding to target input pixel 0. Based on the mapping relationship, the target input pixel 1 corresponding to output pixel 1 (i.e., the output pixel in sequence 1) in the input image is determined. The local cache is managed based on target input pixel 1, so that the local cache stores the target image data corresponding to target input pixel 1… Based on the mapping relationship, the target input pixel N corresponding to output pixel N (i.e., the output pixel in sequence N) in the input image is determined. The local cache is managed based on target input pixel N, so that the local cache stores the target image data corresponding to target input pixel N…

[0024] Reference Figure 1 The image transformation method in this embodiment further includes: S12, performing image processing on the output pixels in the output image according to a preset order, wherein the image processing includes: reading the target image data corresponding to the target input pixel from the local cache, and performing target image processing on the output pixels in order N according to the target image data corresponding to the target input pixel.

[0025] In this embodiment, image processing is performed on the output pixels in the output image according to a preset order. Specifically: target image data corresponding to target input pixel 0 is read from the local cache, and target image processing is performed on output pixel 0 based on the target image data corresponding to target input pixel 0; target image data corresponding to target input pixel 1 is read from the local cache, and target image processing is performed on output pixel 1 based on the target image data corresponding to target input pixel 1... target image data corresponding to target input pixel N is read from the local cache, and target image processing is performed on output pixel N based on the target image data corresponding to target input pixel N...

[0026] In this embodiment, the order of each output pixel is determined by the aforementioned preset order, which may be, for example, a coordinate order from left to right and from top to bottom. In this embodiment, the progress of the pre-transformation process is no later than the progress of the image processing. For example, in this embodiment, when the throughput of the pre-transformation process and the image processing are similar, the start time of the pre-transformation process may be earlier than the start time of the image processing. For example, when the pre-transformation process for the output pixel with order U is completed, image processing may only be performed on the output pixel with order V, where U and V are natural numbers and U ≥ V. Furthermore, the difference in start time can also reflect the difference in the order of the processed output pixels. Further, in this embodiment, the processing time period corresponding to the pre-transformation process and the processing time period corresponding to the image processing preferably partially overlap to accelerate processing efficiency and improve computational performance. For another example, in this embodiment, when the throughput of the pre-transformation process is significantly greater than the throughput of the image processing, the start time of the pre-transformation process may be earlier than the start time of the image processing, or it may be synchronized with the start time of the image processing, or it may even be slightly later than the start time of the image processing.

[0027] In this embodiment, the output image includes multiple output pixels, and the input image includes multiple input pixels. The mapping relationship is used to characterize the coordinate transformation relationship between the output image and the input image. Furthermore, the mapping relationship can have different uses in different application scenarios. For example, it can be used to correct lens distortion in a lens distortion correction scenario, or it can be used to correct image motion in an EIS image stabilization scenario.

[0028] Furthermore, in this embodiment, the mapping relationship can be calculated based on processing units such as DSP (Digital Signal Processor), and can be specifically implemented as a global homography matrix method or a pixel-level optical flow-based mesh interpolation method. In addition, it can be embodied in different forms. For example, the mapping relationship can be embodied as a mapping formula, through which the coordinate position of the output pixel in the output image in the input image can be calculated. Or, for example, the mapping relationship can also be embodied as a mapping table, through which the coordinate position of the output pixel in the output image in the input image can be found.

[0029] It should be understood that the coordinates of the output pixels in the output image obtained based on the mapping relationship are usually non-integer, that is, not the coordinates of the input pixels. Based on this, the input pixel closest to the coordinates obtained based on the mapping relationship can be used as the target input pixel corresponding to the output pixel. Then, the corresponding target image data can be read based on this target input pixel. Furthermore, it should be understood that the correspondence between output pixels and input pixels can be either one-to-one or many-to-one.

[0030] In this embodiment, the target image data corresponding to the target input pixel includes the data of several neighboring input pixels located in the neighboring region of the target input pixel in the input image. The neighboring region of the target input pixel may be, for example, a 2*2 range or a 4*4 range around the target input pixel. Further, the 2*2 range may be a range of coordinates from (0,-1) to (1,0) from the target input pixel, and the 4*4 range may be a range of coordinates from (-1,-1) to (2,2) from the target input pixel. In addition, the data of the neighboring input pixels may include, but is not limited to, the pixel values ​​of the neighboring input pixels, which may be determined according to the actual application scenario.

[0031] In this embodiment, the image transformation method includes two independent processing flows: a pre-transformation processing flow and an image processing flow, in order to improve computational performance. Specifically, the progress of the pre-transformation processing flow is no later than that of the image processing flow. The pre-transformation processing flow aims to pre-transform the coordinates of the output pixels in sequence N according to the mapping relationship, and determine the target input pixels based on the coordinate pre-transformation results to achieve data management of the local cache. This allows the local cache to pre-store the target image data for the output pixels in sequence N to perform target image processing. Thus, in the image processing flow, the target image data for the output pixels in sequence N to perform target image processing is already stored in the local cache, so the target image data can be directly read from the local cache to perform target image processing on the output pixels in sequence N. That is, in the image processing flow, there is at least no need to wait for the local cache to read and store the corresponding target image data from the storage entity, reducing time delay, speeding up processing efficiency, and thus meeting the high-performance design specifications.

[0032] In this embodiment, in order to ensure that the corresponding target image data can be read when performing image processing on output pixels in sequence N, the image processing may further include: determining whether N is not greater than the most recent order of the local cache notification; if so, then performing the step of reading the target image data corresponding to the target input pixel from the local cache.

[0033] Specifically, in this embodiment, the recent order represents the order of the output pixels corresponding to the input pixels used to manage the local cache most recently. For example, when the local cache was managed based on the target input pixel 0 most recently, the local cache notifies that the recent order is 0; when the local cache was managed based on the target input pixel 1 most recently, the local cache notifies that the recent order is 1... when the local cache was managed based on the target input pixel N most recently, the local cache notifies that the recent order is N... When performing image processing on the output pixels with the order N, if N is not greater than the recent order notified by the local cache, it indicates that the output pixels with the order N have already completed the pre-transformation processing. In other words, the local cache has already stored the target image data of the output pixels with the order N for the target image processing.

[0034] Furthermore, in this embodiment, a local cache is managed based on the target input pixel. If the target image data corresponding to the target input pixel is not stored in the local cache, the target image data corresponding to the target input pixel stored in the storage entity is written into the local cache. This aims to ensure that the target image data corresponding to the target input pixel is stored in the local cache. Specifically, this can be implemented by determining whether the target image data corresponding to the target input pixel is stored in the local cache. If yes, otherwise, the target image data corresponding to the target input pixel stored in the storage entity (e.g., DDR) is written into the local cache, ensuring that the target image data corresponding to the target input pixel is stored in the local cache. If yes, the local cache is not updated. For example, when determining the target input pixel 0 in the input image corresponding to output pixel 0 (i.e., the output pixel with sequence number 0) based on the mapping relationship, since target output pixel 0 is the first output pixel in the output image to undergo pre-transformation processing, the local cache will not store the target image data corresponding to input pixel 0. In this case, the target image data corresponding to target input pixel 0 will be written from the storage entity to the local cache, so that the local cache stores the target image data corresponding to target input pixel 0. As another example, when determining the target input pixel 1 in the input image corresponding to output pixel 1 (i.e., the output pixel with sequence number 1) based on the mapping relationship, since the local cache stores the target image data corresponding to target input pixel 0, it is necessary to determine whether target input pixel 1 and target input pixel 0 correspond to the same target image data. If they do, the local cache will not be updated; that is, the target image data corresponding to target input pixel 1 will not be written from the storage entity to the local cache. If not, then the target image data corresponding to target input pixel 1 will be written from the storage entity to the local cache. Based on this, after processing an entire output image in a preset order, bandwidth resources can be greatly saved.

[0035] Therefore, for an output pixel with sequence N, after pre-transformation, the local cache stores at least the target image data for subsequent target image processing. In addition, the local cache may also store the coordinates of its corresponding target input pixel, its corresponding sequence N, etc., for indexing its corresponding target image data. Based on this, the data type stored in the local cache includes not only the candidate image data corresponding to the candidate input pixel, but also at least one of the following: the coordinates of the candidate input pixel, the sequence of the output pixel corresponding to the candidate input pixel, etc. In short, the data type stored in the local cache includes not only candidate image data, but also at least one of the following: coordinates, sequence, etc., that have a corresponding relationship with the candidate image data. It should be understood that the candidate input pixels and candidate image data in this embodiment are intended to simplify the description. The candidate image data is intended to represent the image data stored in the local cache, specifically including the target image data corresponding to different output pixels, and the candidate input pixels are intended to represent the input pixels that have a corresponding relationship with the candidate image data.

[0036] Based on this, if the local cache of this embodiment stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels, then in the pre-transformation process, it can be determined whether the local cache stores target image data corresponding to the target input pixels based on the coordinates of the target input pixels and the coordinates stored in the local cache. Specifically, it can be determined whether the local cache stores target image data corresponding to the target input pixels by comparing the coordinates of the output pixels in order N corresponding to the target input pixels with the coordinates already stored in the local cache, thereby realizing the management of the local cache. Furthermore, if a match can be made, it indicates that the local cache stores target image data corresponding to the target input pixels; if a match cannot be made, it indicates that the local cache does not store target image data corresponding to the target input pixels. It should be understood that under this data management strategy, the coordinates stored in the local cache and their corresponding candidate image data will not be duplicated. That is, when different output pixels correspond to the same target input pixels, the local cache will not repeatedly store the coordinates and target image data corresponding to the target input pixels of multiple output pixels.

[0037] For example, when the local cache is implemented as a cache, managing the local cache based on the target input pixel can be specifically implemented as follows: when the cache receives the target input pixel determined in the pre-transformation process, it can determine whether the target image data corresponding to the target input pixel has been cached locally based on the local tag. If the determination result is miss, the target image data corresponding to the target input pixel stored in the storage entity is written locally. If the determination result is hit, no update is made locally. Here, tag represents the absolute coordinates of the target input pixel on the input image each time the cache is accessed, miss represents the state of the internal cache logic that the target image data corresponding to the target input pixel currently accessed is not yet cached locally, and hit represents the state of the internal cache logic that the target image data corresponding to the target input pixel currently accessed is already cached locally.

[0038] Accordingly, when the local cache stores candidate image data corresponding to candidate input pixels and the coordinates of those candidate input pixels, the image processing step of reading the target image data corresponding to the target input pixel from the local cache can be specifically implemented as follows: first, determine the target input pixel in the input image corresponding to the output pixel of order N according to the mapping relationship; then, read the candidate image data with the same coordinates as the target input pixel from the local cache as the target image data. It should be understood that since pre-transformation processing and image processing are two independent processing methods, the target input pixel corresponding to the output pixel of order N obtained through pre-transformation processing cannot be processed. That is, under this data management strategy, the output pixel of order N needs to undergo coordinate transformation in both pre-transformation processing and image processing, i.e., it needs to undergo two identical coordinate transformations. Furthermore, the first coordinate transformation aims to obtain the target input pixel for managing the local cache so that it stores the corresponding target image data, and the second coordinate transformation aims to obtain the target input pixel for reading the corresponding target image data from the local cache for target image processing.

[0039] If the local cache of this embodiment stores candidate image data corresponding to candidate input pixels and the order of output pixels corresponding to candidate input pixels, then in the pre-transformation process, it can be determined whether the local cache stores target image data corresponding to target input pixels based on N and the order stored in the local cache. Specifically, it can be determined whether the local cache stores target image data corresponding to target input pixels by comparing N with the order already stored in the local cache, thereby realizing the management of the local cache. Further, if a match is found, it indicates that the local cache stores target image data corresponding to target input pixels; if a match is not found, it indicates that the local cache does not store target image data corresponding to target input pixels. It should be understood that under this data management strategy, candidate image data corresponding to different orders stored in the local cache may be duplicated. That is, when the target input pixels corresponding to different output pixels are the same, the local cache will store the target image data corresponding to the target input pixels of the target input pixels corresponding to multiple output pixels multiple times because the order of the different output pixels is different.

[0040] Accordingly, in image processing, reading the target image data corresponding to the target input pixel from the local cache can be specifically implemented by reading the candidate image data in order N from the local cache as the target image data. Under this data management strategy, since the candidate image data in the local cache can be indexed according to the order of the currently processed output pixels, the output pixels in order N only need to undergo coordinate transformation in the pre-transformation process.

[0041] Based on a comprehensive consideration of computational and storage resource requirements, this embodiment preferably adopts the former data management strategy. That is, the local cache prioritizes storing the candidate image data corresponding to the candidate input pixel and the coordinates of that candidate input pixel, rather than the order of the candidate image data corresponding to the candidate input pixel and the output pixel corresponding to that candidate input pixel. Specifically, although the former data management strategy requires two identical coordinate transformations for the same output pixel, while the latter only requires one, considering the potential many-to-one correspondence between output and input pixels, the former data management strategy, compared to the latter, avoids repeated readings and writings of the same target image data when multiple output pixels correspond to the same target input pixel. This prevents waste of storage resources and an increase in overall bandwidth.

[0042] Furthermore, in this embodiment, the coordinate transformation of the output pixel can be implemented in multiple ways. Specifically, for output pixels of sequence N, determining the target input pixel in the input image corresponding to the output pixel of sequence N according to the mapping relationship can be implemented in at least the following three ways.

[0043] Method 1: Determine the coordinates of the target input pixel based on the mapping relationship and the coordinates of the output pixels in order N. Specifically, Method 1 can determine the coordinates of the target input pixel based on (u N, v N )=T{(x N, y N )} to achieve, where (x N, y N ) represents the coordinates of the output pixels in order N, and T represents the mapping relationship, (u N, v N The characterization represents the output pixel (x) obtained based on the mapping relationship T. N, y N The coordinates of the corresponding location in the input image can then be used to determine the distance coordinates (u). N, v N The most recent input pixel is used as the output pixel (x) N, y N The target input pixel corresponding to ) in this case is the distance from the coordinate position (u N, v N The nearest integer coordinate is the output pixel (x). N, y N The coordinates of the target input pixel can be obtained, and then the coordinates of several neighboring input pixels located in the vicinity of the target input pixel can be determined.

[0044] Method 2: Determine the coordinates of the target input pixels based on the relative positions of the output pixels in sequence N within the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image. Method 2 introduces output and input blocks with coordinate transformations that satisfy a mapping relationship, building upon Method 1. Specifically, the output image is divided into multiple output blocks, each containing multiple output pixels. The coordinates of the four vertices of each output block are known relative to the output image. Furthermore, the order of each output block can be obtained according to the preset left-to-right and top-to-bottom order described above. For example, output pixels in sequence N can be included in the target output block in sequence M. In this embodiment, the top-left vertex coordinates of the output block can be used to represent its own block coordinate position. Thus, the coordinates of the four vertices of the target output block can be determined based on the block coordinate positions of the four output blocks. Specifically, the top-left vertex coordinates of the target output block are its own block coordinate positions, the bottom-left vertex coordinates of the target output block are the block coordinate positions of the first output block located below and adjacent to it, the top-right vertex coordinates of the target output block are the block coordinate positions of the second output block located to its right and adjacent to it, and the bottom-right vertex coordinates of the target output block are the block coordinate positions of the third output block that is adjacent to both the first and second output blocks.

[0045] Furthermore, in Method 2, the vertex coordinates of the target input block can be determined based on the mapping relationship and the vertex coordinates of the target output block. Specifically, for the target output block with sequence M, it can be determined based on (u M, v M )=T{(b M, b M )} to achieve, where, (b M, b M ) represents the block coordinate position of the target output block in the output image with order M, and T represents the mapping relationship, (u M, v M The coordinates of the target output block obtained based on the mapping relationship T are represented in the input image. Therefore, by traversing each output block obtained from the output image, the coordinates of the four vertices of the target output block in the input image can be obtained. The block enclosed by these four vertices in the input image is the target input block corresponding to the target output block in the input image. Subsequently, based on the output pixel (x... N, y N The relative position of the pixel in the target output block and the coordinates of the four vertices of the target input block are used for interpolation and other operations to calculate the output pixel (x). N, y N The corresponding coordinate position (u) in the input image N, vN This allows us to determine the distance coordinates (u) N, v N The most recent input pixel is used as the output pixel (x) N, y N The target input pixel corresponding to ) in this case is the distance from the coordinate position (u N, v N The nearest integer coordinate is the output pixel (x). N, y N The coordinates of the target input pixel corresponding to the output pixel in the output image can be determined by traversing all the output blocks and obtaining their corresponding coordinates in the input image, or by traversing all the output blocks and obtaining their coordinates.

[0046] Method 3: Determine the coordinates of the target input pixels based on the relative positions of the N output pixels in the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image. Method 3, building upon Method 2, introduces an output grid map and an input grid map whose coordinate transformation relationship satisfies a mapping relationship. The output image corresponds to an output grid map containing multiple grid points, and the coordinate position of each grid point in the output grid map within the output image is known. Similarly, the input image corresponds to an input grid map containing multiple grid points, and the coordinate position of each grid point in the input grid map within the input image is known. Further, when the output block is a square block, the output image and the output grid map preferably satisfy the following relationship: h size / block size <=mesh nx and v size / block size <=mesh ny , where h size The magnitude of the row direction of the output image, v size Characterizes the size of the column orientation of the output image, block size The mesh represents the row and column size of the output block. nx The mesh represents the number of rows in the output mesh graph. ny This represents the number of columns in the output grid. It should be understood that the output grid preferably completely covers the output image, and there may be a positional offset between the output grid and the output image. Specifically, Figure 2 The correspondence between the output image, the output grid (i.e., Grid_uniform), the input image, and the input grid (i.e., Grid_warp) is shown.

[0047] Based on this, in addition to representing the position of the output block according to the output image, the output block can also be represented by mapping it onto the output mesh map. Specifically, it can be represented according to the mesh. _ x=init x +step x *block _ x and mesh _ y=init y +step y *block _ y is implemented, where block _ x, block _ y represents the relative position of the output block on the output image. For example, following the order from left to right and top to bottom, block(x,y) represents the output block located at the x-th position in the row direction and the y-th position in the column direction in the output image; mesh _ x, mesh _ y represents the row and column positions of the output block mapped onto the output mesh graph, respectively; that is, the position of the output block block(x,y) mapped onto the output mesh graph is mesh(x,y); step x step y These represent the row and column distances required to move from the original output block's row and column positions on the output grid to the current output block's row and column positions on the output grid when moving from one output block to another on the output image; init x init y These respectively characterize the offset of the output image relative to the output grid in the row and column directions. For example, when init... x init y When both are 0, it indicates that there is no offset between the top left vertex position of the output image and the output mesh image; that is, the output block (0,0) corresponds to the mesh point mesh (0,0) in the output mesh image. However, when init... x =1 and init y When the value is 2, the output block (0,0) corresponds to the grid point mesh (1,2) in the output mesh graph. It should be understood that `init`... x init y step x step y All of these may be non-integers. Therefore, the position mesh(x,y) that the output block(x,y) maps to in the output mesh graph may also be non-integer.

[0048] Furthermore, since the coordinate transformation relationship between the output mesh and the input mesh satisfies a mapping relationship, it should be understood that the output block (x,y) maps to the position mesh (x,y) in the output mesh, and also maps to the position mesh (x,y) in the input mesh. Therefore, the corresponding coordinate position of the position mesh (x,y) in the input mesh in the input image is the block coordinate position of the output block (x,y) in the input image corresponding to the input block. Since the position mesh (x,y) is usually a non-integer position, that is, the position of a non-mesh point, the coordinate position of the output block (x,y) corresponding to the input block in the input image can be determined based on several mesh points in the neighboring region of the nearest mesh point in the input mesh. Specifically, this can be obtained by interpolation and other operations based on the relative position of the position mesh (x,y) in the corresponding neighboring region and the coordinates of several mesh points in the corresponding neighboring region of the position mesh (x,y) in the input image. The description of the neighboring region can be found above and will not be repeated here.

[0049] Therefore, in Method 3, the vertex coordinates of the target input block can be determined based on the coordinates of several grid points in the vicinity of the target grid point corresponding to the vertex of the target input block, and the target grid point can be determined based on the position of the corresponding vertex in the target output block in the output grid graph. Specifically, for a target output block of sequence M, the target output block can be first... M (x, y) is mapped to the position mesh on the output mesh graph. M (x,y), thus allowing the distance from the input mesh to the location mesh. M The nearest grid point (x, y) is determined as the target grid point. Therefore, the mesh can be spun based on its location. M The relative position of (x, y) in the neighborhood of the target grid point and the coordinates of several grid points in the neighborhood of the target grid point in the input image are interpolated to obtain the position mesh in the input grid image. M The coordinates (x, y) in the input image correspond to the target output block. M The coordinates (x, y) in the input image correspond to the given coordinates. By traversing each output block of the output image, the coordinates of the four vertices of the target output block in the input image can be obtained. The block enclosed by these four vertices in the input image is the target input block corresponding to the target output block in the input image. Then, based on method two, the coordinates of the target input pixels can be determined according to the relative positions of the N output pixels in the corresponding target output blocks and the vertex coordinates of the target input blocks corresponding to the target output blocks in the input image.

[0050] In this embodiment, after determining the coordinates of the target input pixel, the coordinates of several neighboring input pixels located in the vicinity of the target input pixel can be determined. That is, the target image data corresponding to the target input pixel for subsequent target image processing can be determined. In this embodiment, target image processing may include, but is not limited to, pixel interpolation. For example, in this embodiment, if the data type stored in the local cache includes image data and its corresponding coordinates, the output pixel (x) can be determined first based on the mapping relationship during image processing. N, y N The corresponding coordinate position (u) in the input image N, v N ), and then in the input image, based on the coordinate position (u N, v N The target input pixel is determined and its coordinates are used to extract the target image data. Based on this, the coordinate position (u) in the input image can be used to extract the target image data. N, v N The extracted target image data is then interpolated to obtain the output pixel (x) in the output image. N, y N The corresponding pixel value. For example, if the data type stored in the local cache includes image data and its order, then in image processing, the target image data can be extracted based on the order N of the output pixels. Furthermore, the output pixels (x, y, z) in the output image can be directly processed based on the pixel values ​​corresponding to the target input pixels in the target image data. N, y N You can assign values ​​to the x-pixels in the output image, or perform custom calculations based on the target image data. N, y N The corresponding pixel value.

[0051] Furthermore, in this embodiment, managing the local cache based on the target input pixels can be specifically implemented by managing the local cache according to the PLRU (Pseudo-Least Recently Used) strategy or the LRU (Least Recently Used) strategy. Therefore, in this embodiment, the local cache preferably adopts a replacement strategy of retaining the "most recently used" and replacing the "least recently used" to avoid repeated reading and writing of the same image data as much as possible, thereby reducing the overall bandwidth as much as possible.

[0052] Furthermore, to minimize the repeated reading and writing of the same image data, this embodiment can also obtain a local cache configuration scheme with the lowest overall bandwidth by simulating real-world shooting data for all scenes in advance. Here, "all scenes" refers to the actual scenarios that this embodiment can apply to, including but not limited to lens distortion correction, image registration, panoramic stitching, EIS image stabilization, SLAM, and other practical application scenarios. For example, when the local cache is implemented as a cache, the cache configuration scheme with the lowest overall bandwidth obtained based on simulation can include, but is not limited to, related configurations such as cache size, cacheline size and layout, set size and layout, and way size. Here, cache size represents the size of the cache, cacheline represents the basic data unit that the cache fills or replaces each time (e.g., 32*4), set represents a logical grouping unit and includes several cachelines (e.g., 32 cachelines), and way represents the number of parallel logical groups (e.g., the cache can include 32 ways).

[0053] Based on test data, this embodiment can achieve high performance of 4K 120fps for any angle between 0° and 360° of image rotation; for typical scenarios such as EIS image stabilization and image registration, this embodiment can achieve high performance of 4K 120fps; for special scenarios such as panoramic stitching and horizon correction, this embodiment can also achieve high performance of 4K 60fps.

[0054] This embodiment provides an image transformation system, and more specifically, an image geometric transformation system for mapping coordinate positions in one image to new coordinate positions in another image. This system can be applied to scenarios such as lens distortion correction, image registration, panoramic stitching, EIS image stabilization, and SLAM. In this embodiment, the image transformation system is preferably based on backward mapping. Figure 3 A schematic diagram of the modules in this embodiment is shown.

[0055] Reference Figure 3The image transformation system of this embodiment includes: a pre-transformation processing module 21, used to pre-transform the output pixels in the output image according to a preset order, specifically used to determine the target input pixels in the input image corresponding to the output pixels in the order of N according to the mapping relationship; a local cache 22, used to manage data according to the target input pixels, so that if the target image data corresponding to the target input pixels is not stored, the target image data corresponding to the target input pixels stored in the storage entity is written to ensure that the target image data corresponding to the target input pixels is stored; and an image processing module 23, used to perform image processing on the output pixels in the output image according to a preset order, specifically used to read the target image data corresponding to the target input pixels from the local cache 22, and perform target image processing on the output pixels in the order of N according to the target image data corresponding to the target input pixels.

[0056] In this embodiment, the pre-transformation processing module 21 is specifically used to: determine the target input pixel 0 in the input image corresponding to the output pixel 0 (i.e., the output pixel with sequence 0) according to the mapping relationship; determine the target input pixel 1 in the input image corresponding to the output pixel 1 (i.e., the output pixel with sequence 1) according to the mapping relationship; determine the target input pixel N in the input image corresponding to the output pixel N (i.e., the output pixel with sequence N) according to the mapping relationship; ... In this embodiment, the local cache 22 is specifically used to: manage the local cache according to the target input pixel 0 to store the target image data corresponding to the target input pixel 0; manage the local cache according to the target input pixel 1 to store the target image corresponding to the target input pixel 1. The data... is managed in a local cache based on the target input pixel N to store the target image data corresponding to the target input pixel N... In this embodiment, the image processing module 23 is specifically used for: reading the target image data corresponding to the target input pixel 0 from the local cache 22, and performing target image processing on the output pixel 0 based on the target image data corresponding to the target input pixel 0; reading the target image data corresponding to the target input pixel 1 from the local cache 22, and performing target image processing on the output pixel 1 based on the target image data corresponding to the target input pixel 1... and reading the target image data corresponding to the target input pixel N from the local cache 22, and performing target image processing on the output pixel N based on the target image data corresponding to the target input pixel N...

[0057] In this embodiment, the order of each output pixel is determined by the aforementioned preset order, which may be, for example, a coordinate order from left to right and from top to bottom. In this embodiment, the progress of the pre-transformation processing module is no later than that of the image processing module. For example, in this embodiment, when the throughput of the pre-transformation processing module and the image processing module is similar, the start time of the pre-transformation processing module may be earlier than that of the image processing module. For instance, image processing may only be performed on output pixels with order U after the pre-transformation processing of output pixels with order U is completed, where U and V are natural numbers and U ≥ V. Furthermore, the difference in start time can also reflect the difference in the order of the processed output pixels. Further, in this embodiment, the processing time period of the pre-transformation processing module and the processing time period of the image processing module preferably partially overlap to accelerate processing efficiency and improve computational performance. For example, in this embodiment, when the throughput of the pre-transformation processing module is significantly greater than that of the image processing module, the start time of the pre-transformation processing module can be earlier than the start time of the image processing module, or synchronized with the start time of the image processing module, or even slightly later than the start time of the image processing module.

[0058] In this embodiment, the output image includes multiple output pixels, and the input image includes multiple input pixels. The mapping relationship is used to characterize the coordinate transformation relationship between the output image and the input image. Furthermore, the mapping relationship can have different uses in different application scenarios. For example, it can be used to correct lens distortion in a lens distortion correction scenario, or it can be used to correct image motion in an EIS image stabilization scenario.

[0059] Furthermore, in this embodiment, the mapping relationship can be calculated based on processing units such as DSP (Digital Signal Processor), and can be specifically implemented as a global homography matrix method or a pixel-level optical flow-based mesh interpolation method. In addition, it can be embodied in different forms. For example, the mapping relationship can be embodied as a mapping formula, through which the coordinate position of the output pixel in the output image in the input image can be calculated. Or, for example, the mapping relationship can also be embodied as a mapping table, through which the coordinate position of the output pixel in the output image in the input image can be found.

[0060] It should be understood that the coordinates of the output pixels in the output image obtained based on the mapping relationship are usually non-integer, that is, not the coordinates of the input pixels. Based on this, the input pixel closest to the coordinates obtained based on the mapping relationship can be used as the target input pixel corresponding to the output pixel. Then, the corresponding target image data can be read based on this target input pixel. Furthermore, it should be understood that the correspondence between output pixels and input pixels can be either one-to-one or many-to-one.

[0061] In this embodiment, the target image data corresponding to the target input pixel includes the data of several neighboring input pixels located in the neighboring region of the target input pixel in the input image. The neighboring region of the target input pixel may be, for example, a 2*2 range or a 4*4 range around the target input pixel. Further, the 2*2 range may be a range of coordinates from (0,-1) to (1,0) from the target input pixel, and the 4*4 range may be a range of coordinates from (-1,-1) to (2,2) from the target input pixel. In addition, the data of the neighboring input pixels may include, but is not limited to, the pixel values ​​of the neighboring input pixels, which may be determined according to the actual application scenario.

[0062] In this embodiment, the image transformation system includes two independent processing modules, a pre-transformation processing module 21 and an image processing module 23, to improve computational performance. Specifically, the progress of the pre-transformation processing module 21 is no later than that of the image processing module 23. The pre-transformation processing module 21 aims to pre-transform the coordinates of the output pixels in sequence N according to the mapping relationship to determine the corresponding target input pixels. This allows the local cache 22 to manage data based on the target input pixels, thereby pre-storing the target image data for target image processing of the output pixels in sequence N. As a result, the target image data for target image processing of the output pixels in sequence N is already stored in the local cache 22 for the image processing module 23. The image processing module 23 can then directly read the target image data from the local cache 22 to perform target image processing on the output pixels in sequence N. In other words, the image processing module 23 does not need to wait for the local cache 22 to read and store the corresponding target image data from the storage entity, reducing time delay, accelerating processing efficiency, and thus meeting the high-performance design specifications.

[0063] In this embodiment, the image processing module 23 can also be specifically used to: determine whether N is not greater than the most recent order notified by the local cache 22; if so, read the target image data corresponding to the target input pixel from the local cache 22.

[0064] Specifically, in this embodiment, the recent order represents the order of the output pixels corresponding to the most recent input pixels used for data management. For example, when the most recent data management was performed based on the target input pixel 0, the local cache 22 notifies that the recent order is 0; when the most recent data management was performed based on the target input pixel 1, the local cache 22 notifies that the recent order is 1... when the most recent data management was performed based on the target input pixel N, the local cache 22 notifies that the recent order is N... When performing image processing on the output pixels with the order N, if N is not greater than the recent order notified by the local cache 22, it indicates that the output pixels with the order N have already completed the pre-transformation processing. In other words, the local cache 22 has already stored the target image data of the output pixels with the order N for target image processing.

[0065] Further, in this embodiment, when the local cache 22 does not store target image data corresponding to the target input pixel, it writes the target image data corresponding to the target input pixel stored in the storage entity into the local cache, so that the local cache stores the target image data corresponding to the target input pixel. Specifically, the local cache 22 can be used to determine whether target image data corresponding to the target input pixel is stored. If the determination is yes, otherwise, it writes the target image data corresponding to the target input pixel stored in the storage entity (e.g., DDR, etc.) to store the target image data corresponding to the target input pixel. If the determination is yes, no update is performed. For example, when the target input pixel 0 in the input image is determined according to the mapping relationship to correspond to the output pixel 0 (i.e., the output pixel with sequence 0), since the target output pixel 0 is the first output pixel in the output image to undergo pre-transformation processing, the local cache 22 will not store the target image data corresponding to the input pixel 0. At this time, the target image data corresponding to the target input pixel 0 stored in the storage entity will be written into the local cache 22, so that the local cache 22 stores the target image data corresponding to the target input pixel 0. For example, when determining the target input pixel 1 in the input image corresponding to output pixel 1 (i.e., the output pixel with sequence 1) based on the mapping relationship, since the local cache 22 stores the target image data corresponding to target input pixel 0, it is necessary to determine whether target input pixel 1 and target input pixel 0 correspond to the same target image data. If they do, no update is performed; that is, the target image data corresponding to target input pixel 1 stored in the storage entity will not be written. If not, then the target image data corresponding to target input pixel 1 stored in the storage entity will be written. Based on this, after processing an entire output image in a preset order, bandwidth resources can be greatly saved.

[0066] Therefore, for output pixels in sequence N, after pre-transformation, the local cache 22 stores at least the target image data for subsequent target image processing. In addition, the local cache 22 may also store the coordinates of its corresponding target input pixels, its corresponding sequence N, etc., for indexing its corresponding target image data. Based on this, the data type stored in the local cache 22 includes not only the candidate image data corresponding to the candidate input pixels, but also at least one of the following: the coordinates of the candidate input pixels, the sequence of the output pixels corresponding to the candidate input pixels, etc. In short, the data type stored in the local cache 22 includes not only candidate image data, but also at least one of the following: coordinates, sequence, etc., that have a corresponding relationship with the candidate image data. It should be understood that the candidate input pixels and candidate image data in this embodiment are intended for simplification. The candidate image data is intended to represent the image data stored in the local cache 22, specifically including target image data corresponding to different output pixels, and the candidate input pixels are intended to represent input pixels that have a corresponding relationship with the candidate image data.

[0067] Based on this, if the local cache 22 of this embodiment stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels, then the local cache 22 can determine whether target image data corresponding to the target input pixels is stored based on the received coordinates of the target input pixels. Specifically, it can determine whether target image data corresponding to the target input pixels is stored by comparing the coordinates of the target input pixels corresponding to the Nth output pixels with the already stored coordinates, thereby realizing data management of the local cache. Furthermore, if a match is found, it indicates that target image data corresponding to the target input pixels is stored in the local cache; if a match is not found, it indicates that target image data corresponding to the target input pixels is not stored in the local cache. It should be understood that under this data management strategy, the coordinates and corresponding candidate image data stored in the local cache will not be duplicated. That is, when the target input pixels corresponding to different output pixels are the same, the local cache will not repeatedly store the coordinates and target image data corresponding to the target input pixels corresponding to multiple output pixels.

[0068] For example, when the local cache 22 is implemented as a cache, when the cache receives the target input pixel determined in the pre-transformation process, it can determine whether the target image data corresponding to the target input pixel has been cached locally based on the local tag. If the determination result is miss, the target image data corresponding to the target input pixel stored in the storage entity is written locally. If the determination result is hit, the local cache is not updated. Here, tag represents the absolute coordinates of the target input pixel on the input image each time the cache is accessed. miss represents the state of the internal cache logic that the target image data corresponding to the target input pixel of the current accessed cache has not yet been cached locally. hit represents the state of the internal cache logic that the target image data corresponding to the target input pixel of the current accessed cache has been cached locally.

[0069] Accordingly, when the local cache 22 stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels, the image processing module 23 can specifically be used to first determine the target input pixel in the input image corresponding to the output pixel in sequence N according to the mapping relationship; and then read the candidate image data with the same coordinates as the target input pixel from the local cache 22 as the target image data. It should be understood that since the pre-transformation processing and image processing are two independent processing methods, the target input pixel corresponding to the output pixel in sequence N obtained by the pre-transformation processing cannot be processed by image processing. That is, under this data management strategy, the output pixel in sequence N needs to undergo coordinate transformation in both the pre-transformation processing and image processing, i.e., it needs to undergo two identical coordinate transformations. Furthermore, the first coordinate transformation aims to obtain the target input pixel for managing the local cache so that it stores the corresponding target image data, and the second coordinate transformation aims to obtain the target input pixel for reading the corresponding target image data from the local cache for target image processing.

[0070] If the local cache 22 of this embodiment stores candidate image data corresponding to candidate input pixels and the order of output pixels corresponding to candidate input pixels, then the local cache 22 can determine whether target image data corresponding to target input pixels is stored based on the received N. Specifically, it can determine whether target image data corresponding to target input pixels is stored by comparing the received N with the stored order, thereby realizing data management of the local cache. Further, if the comparison is successful, it indicates that target image data corresponding to target input pixels is stored in the local cache; if the comparison is unsuccessful, it indicates that target image data corresponding to target input pixels is not stored in the local cache. It should be understood that under this data management strategy, candidate image data corresponding to different orders stored in the local cache may be duplicated. That is, when the target input pixels corresponding to different output pixels are the same, the local cache will store the target image data corresponding to the target input pixels of the target input pixels corresponding to the multiple output pixels multiple times because the order of the different output pixels is different.

[0071] Accordingly, the image processing module 23 can specifically be used to read candidate image data in order N from the local cache 22 as target image data. Under this data management strategy, since the candidate image data in the local cache 22 can be indexed according to the order of the output pixels currently being processed, the output pixels in order N only need to undergo coordinate transformation in the pre-transformation process.

[0072] Based on a comprehensive consideration of both computational and storage resource requirements, this embodiment preferably adopts the former data management strategy. That is, the local cache 22 prioritizes storing the candidate image data corresponding to the candidate input pixel and the coordinates of that candidate input pixel, rather than the order of the candidate image data corresponding to the candidate input pixel and the output pixel corresponding to that candidate input pixel. Specifically, although the former data management strategy requires two identical coordinate transformations for the same output pixel, while the latter only requires one, considering the potential many-to-one correspondence between output and input pixels, the former data management strategy, compared to the latter, avoids repeated readings and writings of the same target image data when multiple output pixels correspond to the same target input pixel. This prevents waste of storage resources and an increase in overall bandwidth.

[0073] Furthermore, in this embodiment, the coordinate transformation of the output pixel can be implemented in multiple ways. Specifically, for output pixels of sequence N, determining the target input pixel in the input image corresponding to the output pixel of sequence N according to the mapping relationship can be implemented in at least the following three ways.

[0074] Method 1: Determine the coordinates of the target input pixel based on the mapping relationship and the coordinates of the output pixels in order N. Specifically, Method 1 can determine the coordinates of the target input pixel based on (u N, v N )=T{(x N, y N )} to achieve, where (x N, y N ) represents the coordinates of the output pixels in order N, and T represents the mapping relationship, (u N, v N The characterization represents the output pixel (x) obtained based on the mapping relationship T. N, y N The coordinates of the corresponding location in the input image can then be used to determine the distance coordinates (u). N, v N The most recent input pixel is used as the output pixel (x) N, y N The target input pixel corresponding to ) in this case is the distance from the coordinate position (u N, v N The nearest integer coordinate is the output pixel (x). N, y N The coordinates of the target input pixel can be obtained, and then the coordinates of several neighboring input pixels located in the vicinity of the target input pixel can be determined.

[0075] Method 2: Determine the coordinates of the target input pixels based on the relative positions of the output pixels in sequence N within the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image. Method 2 introduces output and input blocks with coordinate transformations that satisfy a mapping relationship, building upon Method 1. Specifically, the output image is divided into multiple output blocks, each containing multiple output pixels. The coordinates of the four vertices of each output block are known relative to the output image. Furthermore, the order of each output block can be obtained according to the preset left-to-right and top-to-bottom order described above. For example, output pixels in sequence N can be included in the target output block in sequence M. In this embodiment, the top-left vertex coordinates of the output block can be used to represent its own block coordinate position. Thus, the coordinates of the four vertices of the target output block can be determined based on the block coordinate positions of the four output blocks. Specifically, the top-left vertex coordinates of the target output block are its own block coordinate positions, the bottom-left vertex coordinates of the target output block are the block coordinate positions of the first output block located below and adjacent to it, the top-right vertex coordinates of the target output block are the block coordinate positions of the second output block located to its right and adjacent to it, and the bottom-right vertex coordinates of the target output block are the block coordinate positions of the third output block that is adjacent to both the first and second output blocks.

[0076] Furthermore, in Method 2, the vertex coordinates of the target input block can be determined based on the mapping relationship and the vertex coordinates of the target output block. Specifically, for the target output block with sequence M, it can be determined based on (u M, v M )=T{(b M, b M )} to achieve, where, (b M, b M ) represents the block coordinate position of the target output block in the output image with order M, and T represents the mapping relationship, (u M, v M The coordinates of the target output block obtained based on the mapping relationship T are represented in the input image. Therefore, by traversing each output block obtained from the output image, the coordinates of the four vertices of the target output block in the input image can be obtained. The block enclosed by these four vertices in the input image is the target input block corresponding to the target output block in the input image. Subsequently, based on the output pixel (x... N, y N The relative position of the pixel in the target output block and the coordinates of the four vertices of the target input block are used for interpolation and other operations to calculate the output pixel (x). N, y N The corresponding coordinate position (u) in the input image N, vN This allows us to determine the distance coordinates (u) N, v N The most recent input pixel is used as the output pixel (x) N, y N The target input pixel corresponding to ) in this case is the distance from the coordinate position (u N, v N The nearest integer coordinate is the output pixel (x). N, y N The coordinates of the target input pixel corresponding to the output pixel in the output image can be determined by traversing all the output blocks and obtaining their corresponding coordinates in the input image, or by traversing all the output blocks and obtaining their coordinates.

[0077] Method 3: Determine the coordinates of the target input pixels based on the relative positions of the N output pixels in the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image. Method 3, building upon Method 2, introduces an output grid map and an input grid map whose coordinate transformation relationship satisfies a mapping relationship. The output image corresponds to an output grid map containing multiple grid points, and the coordinate position of each grid point in the output grid map within the output image is known. Similarly, the input image corresponds to an input grid map containing multiple grid points, and the coordinate position of each grid point in the input grid map within the input image is known. Further, when the output block is a square block, the output image and the output grid map preferably satisfy the following relationship: h size / block size <=mesh nx and v size / block size <=mesh ny , where h size The magnitude of the row direction of the output image, v size Characterizes the size of the column orientation of the output image, block size The mesh represents the row and column size of the output block. nx The mesh represents the number of rows in the output mesh graph. ny This represents the number of columns in the output grid. It should be understood that the output grid preferably completely covers the output image, and there may be a positional offset between the output grid and the output image. Specifically, Figure 2 The correspondence between the output image, the output grid (i.e., Grid_uniform), the input image, and the input grid (i.e., Grid_warp) is shown.

[0078] Based on this, in addition to representing the position of the output block according to the output image, the output block can also be represented by mapping it onto the output mesh map. Specifically, it can be represented according to the mesh. _ x=init x +step x *block _ x and mesh _ y=init y +step y *block _ y is implemented, where block _ x, block _ y represents the relative position of the output block on the output image. For example, following the order from left to right and top to bottom, block(x,y) represents the output block located at the x-th position in the row direction and the y-th position in the column direction in the output image; mesh _ x, mesh _ y represents the row and column positions of the output block mapped onto the output mesh graph, respectively; that is, the position of the output block block(x,y) mapped onto the output mesh graph is mesh(x,y); step x step y These represent the row and column distances required to move from the original output block's row and column positions on the output grid to the current output block's row and column positions on the output grid when moving from one output block to another on the output image; init x init y These respectively characterize the offset of the output image relative to the output grid in the row and column directions. For example, when init... x init y When both are 0, it indicates that there is no offset between the top left vertex position of the output image and the output mesh image; that is, the output block (0,0) corresponds to the mesh point mesh (0,0) in the output mesh image. However, when init... x =1 and init y When the value is 2, the output block (0,0) corresponds to the grid point mesh (1,2) in the output mesh graph. It should be understood that `init`... x init y step x step y All of these may be non-integers. Therefore, the position mesh(x,y) that the output block(x,y) maps to in the output mesh graph may also be non-integer.

[0079] Furthermore, since the coordinate transformation relationship between the output mesh and the input mesh satisfies a mapping relationship, it should be understood that the output block (x,y) maps to the position mesh (x,y) in the output mesh, and also maps to the position mesh (x,y) in the input mesh. Therefore, the corresponding coordinate position of the position mesh (x,y) in the input mesh in the input image is the block coordinate position of the output block (x,y) in the input image corresponding to the input block. Since the position mesh (x,y) is usually a non-integer position, that is, the position of a non-mesh point, the coordinate position of the output block (x,y) corresponding to the input block in the input image can be determined based on several mesh points in the neighboring region of the nearest mesh point in the input mesh. Specifically, this can be obtained by interpolation and other operations based on the relative position of the position mesh (x,y) in the corresponding neighboring region and the coordinates of several mesh points in the corresponding neighboring region of the position mesh (x,y) in the input image. The description of the neighboring region can be found above and will not be repeated here.

[0080] Therefore, in Method 3, the vertex coordinates of the target input block can be determined based on the coordinates of several grid points in the vicinity of the target grid point corresponding to the vertex of the target input block, and the target grid point can be determined based on the position of the corresponding vertex in the target output block in the output grid graph. Specifically, for a target output block of sequence M, the target output block can be first... M (x, y) is mapped to the position mesh on the output mesh graph. M (x,y), thus allowing the distance from the input mesh to the location mesh. M The nearest grid point (x, y) is determined as the target grid point. Therefore, the mesh can be spun based on its location. M The relative position of (x, y) in the neighborhood of the target grid point and the coordinates of several grid points in the neighborhood of the target grid point in the input image are interpolated to obtain the position mesh in the input grid image. M The coordinates (x, y) in the input image correspond to the target output block. M The coordinates (x, y) in the input image correspond to the given coordinates. By traversing each output block of the output image, the coordinates of the four vertices of the target output block in the input image can be obtained. The block enclosed by these four vertices in the input image is the target input block corresponding to the target output block in the input image. Then, based on method two, the coordinates of the target input pixels can be determined according to the relative positions of the N output pixels in the corresponding target output blocks and the vertex coordinates of the target input blocks corresponding to the target output blocks in the input image.

[0081] In this embodiment, after determining the coordinates of the target input pixel, the coordinates of several neighboring input pixels located in the vicinity of the target input pixel can be determined. That is, the target image data corresponding to the target input pixel for subsequent target image processing can be determined. In this embodiment, target image processing may include, but is not limited to, pixel interpolation. For example, in this embodiment, if the data type stored in the local cache includes image data and its corresponding coordinates, the output pixel (x) can be determined first based on the mapping relationship during image processing. N, y N The corresponding coordinate position (u) in the input image N, v N ), and then in the input image, based on the coordinate position (u N, v N The target input pixel is determined and its coordinates are used to extract the target image data. Based on this, the coordinate position (u) in the input image can be used to extract the target image data. N, v N The extracted target image data is then interpolated to obtain the output pixel (x) in the output image. N, y N The corresponding pixel value. For example, if the data type stored in the local cache includes image data and its order, then in image processing, the target image data can be extracted based on the order N of the output pixels. Furthermore, the output pixels (x, y, z) in the output image can be directly processed based on the pixel values ​​corresponding to the target input pixels in the target image data. N, y N You can assign values ​​to the x-pixels in the output image, or perform custom calculations based on the target image data. N, y N The corresponding pixel value.

[0082] Furthermore, in this embodiment, the local cache 22 can manage data according to the PLRU (Pseudo-Least Recently Used) strategy or the LRU (Least Recently Used) strategy. Therefore, the local cache in this embodiment preferably adopts a replacement strategy of retaining the "most recently used" and replacing the "least recently used" to avoid repeated reading and writing of the same image data as much as possible, thereby reducing the overall bandwidth as much as possible.

[0083] Furthermore, to minimize the repeated reading and writing of the same image data, this embodiment can also obtain a local cache configuration scheme with the lowest overall bandwidth by simulating real-world shooting data for all scenes in advance. Here, "all scenes" refers to the actual scenarios that this embodiment can apply to, including but not limited to lens distortion correction, image registration, panoramic stitching, EIS image stabilization, SLAM, and other practical application scenarios. For example, when the local cache is implemented as a cache, the cache configuration scheme with the lowest overall bandwidth obtained based on simulation can include, but is not limited to, related configurations such as cache size, cacheline size and layout, set size and layout, and way size. Here, cache size represents the size of the cache, cacheline represents the basic data unit that the cache fills or replaces each time (e.g., 32*4), set represents a logical grouping unit and includes several cachelines (e.g., 32 cachelines), and way represents the number of parallel logical groups (e.g., the cache can include 32 ways).

[0084] Based on test data, this embodiment can achieve high performance of 4K 120fps for any angle between 0° and 360° of image rotation; for typical scenarios such as EIS image stabilization and image registration, this embodiment can achieve high performance of 4K 120fps; for special scenarios such as panoramic stitching and horizon correction, this embodiment can also achieve high performance of 4K 60fps.

[0085] This embodiment provides a chip that may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can implement the image transformation method provided in Embodiment 1 when executing the computer program.

[0086] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the image transformation method provided in Embodiment 1. The readable storage medium may include, but is not limited to, portable disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0087] In a possible implementation, the present invention can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps of implementing the image transformation method provided in Embodiment 1. The program code for executing the present invention can be written in any combination of one or more programming languages, and can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0088] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. An image transformation method, characterized in that, include: The output pixels in the output image are pre-transformed according to a preset order. The pre-transformation process includes: determining the target input pixel in the input image corresponding to the output pixel in the order of N according to a mapping relationship; managing a local cache according to the target input pixel; and writing the target image data corresponding to the target input pixel stored in the storage entity into the local cache when the target image data corresponding to the target input pixel is not stored in the local cache, so that the target image data corresponding to the target input pixel is stored in the local cache. Here, N is a natural number, the mapping relationship is used to characterize the coordinate transformation relationship between the output image and the input image, and the target image data corresponding to the target input pixel includes the data of several neighboring input pixels in the input image located in the vicinity of the target input pixel. The output pixels in the output image are processed according to the preset order, wherein the image processing includes: reading the target image data corresponding to the target input pixel from the local cache, and performing target image processing on the output pixels in order N according to the target image data corresponding to the target input pixel, wherein the progress of the pre-transformation processing is no later than the progress of the image processing; The image processing further includes: Determine whether N is not greater than the most recent order of the local cache notification, wherein the most recent order represents the order of the output pixels corresponding to the most recent input pixels used to manage the local cache; If so, then the step of reading the target image data corresponding to the target input pixel from the local cache is performed; The local cache stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels. Reading the target image data corresponding to the target input pixel from the local cache includes: The target input pixel in the input image corresponding to the output pixel in order N is determined according to the mapping relationship; The candidate image data whose corresponding coordinates are the same as the target input pixel is read from the local cache and used as the target image data.

2. The image transformation method as described in claim 1, characterized in that, The step of determining the target input pixel in the input image corresponding to the output pixel in order N according to the mapping relationship includes: The coordinates of the target input pixel are determined based on the mapping relationship and the coordinates of the output pixel points in order N.

3. The image transformation method as described in claim 1, characterized in that, The step of determining the target input pixel in the input image corresponding to the output pixel in order N according to the mapping relationship includes: The coordinates of the target input pixel are determined based on the relative positions of the output pixels (N) in the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image, wherein: The output image is divided into multiple output blocks, each of which includes multiple output pixels. The vertex coordinates of the target input block are determined based on the mapping relationship and the vertex coordinates of the target output block.

4. The image transformation method as described in claim 1, characterized in that, The step of determining the target input pixel in the input image corresponding to the output pixel in order N according to the mapping relationship includes: The coordinates of the target input pixel are determined based on the relative positions of the output pixels (N) in the corresponding target output block and the vertex coordinates of the target input block corresponding to the target output block in the input image, wherein: The output image is divided into multiple output blocks, each output block including multiple output pixels. The output image corresponds to an output grid map including multiple grid points, and the input image corresponds to an input grid map including multiple grid points. The output grid map and the input grid map satisfy the mapping relationship. The vertex coordinates of the target input block are determined based on the coordinates of several grid points in the vicinity of the target grid point corresponding to the vertex of the target input block. The target grid point is determined based on the position of the corresponding vertex in the target output block in the output grid map.

5. The image transformation method as described in claim 1, characterized in that, The start time of the pre-transformation process is earlier than the start time of the image processing, and / or the throughput of the pre-transformation process is higher than the throughput of the image processing.

6. The image transformation method as described in claim 1, characterized in that, The local cache stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels. Managing the local cache based on the target input pixel includes: Based on the coordinates of the target input pixel and the coordinates stored in the local cache, it is determined whether the target image data corresponding to the target input pixel is stored in the local cache; If not, the target image data corresponding to the target input pixel stored in the storage entity is written to the local cache.

7. The image transformation method as described in claim 1, characterized in that, The step of managing the local cache based on the target input pixel includes: The local cache is managed according to either the PLRU or LRU policy.

8. An image transformation system, characterized in that, include: The pre-transformation processing module is used to perform pre-transformation processing on the output pixels in the output image according to a preset order. Specifically, it is used to determine the target input pixels in the input image corresponding to the output pixels in the order of N according to the mapping relationship, where N is a natural number, and the mapping relationship is used to characterize the coordinate transformation relationship between the output image and the input image. A local cache is used for data management based on the target input pixel, so that when the target image data corresponding to the target input pixel is not stored in the storage entity, the target image data corresponding to the target input pixel is written to the storage entity, so that the target image data corresponding to the target input pixel is stored. The target image data corresponding to the target input pixel includes the data of several neighboring input pixels in the input image located in the vicinity of the target input pixel. The image processing module is used to perform image processing on the output pixels in the output image according to the preset order. Specifically, it is used to read the target image data corresponding to the target input pixel from the local cache, and perform target image processing on the output pixels in order N according to the target image data corresponding to the target input pixel. The progress of the pre-transformation processing module is no later than the progress of the image processing module. The image processing module is further configured to determine whether N is not greater than the most recent order of the local cache notification, wherein the most recent order represents the order of the output pixels corresponding to the most recent input pixels used to manage the local cache; if so, the target image data corresponding to the target input pixel is read from the local cache. The local cache stores candidate image data corresponding to candidate input pixels and the coordinates of the candidate input pixels. The image processing module is further specifically used to: determine the target input pixel in the input image corresponding to the output pixel in the order of N according to the mapping relationship; and read the candidate image data whose corresponding coordinates are the same as the target input pixel from the local cache as the target image data.

9. A chip comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the image transformation method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the image transformation method as described in any one of claims 1-7.

Images