Image deformation method, device and system

Abstract

The present disclosure proposes an image deformation method, device and system, which include: dividing the output image into blocks according to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates to obtain a plurality of corresponding sub-blocks; fitting a simple transformation relationship in each of the sub-blocks to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates; for each of the sub-blocks, fitting the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, compressing and storing the quantized fitting parameters and the sub-block ranges; and reordering the sub-block ranges and the corresponding fitting parameters according to the nearest neighbor first calculation principle.

Description

Technical Field

[0001] The present disclosure relates to image deformation processing, and in particular to an image deformation method, device and system. Background Art

[0002] Due to the improvement of camera resolution and the development of deep learning, we can extract more and more information from images with increasing accuracy. Therefore, image processing technology is currently widely used in various industries.

[0003] Due to the distortion of the camera lens and the interference of factors such as camera angle, object angle and rain, the collected original images are often distorted. Therefore, the image processing program usually uses image deformation operations to restore the distorted original image. In addition, image deformation is also widely used in deep learning training to enhance data. However, most of the current image deformation operations are relatively complex, including complex mathematical operations such as exponentials and logarithms, and are often floating-point operations, which will not only slow down the entire image processing program, but also waste a lot of computing resources. These two points are more obvious on edge devices that have certain requirements for real-time performance and have fewer storage resources and computing resources. Summary of the invention

[0004] In view of this, one aspect of the present disclosure provides an image deformation method, including: dividing the output image into blocks according to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates to obtain a corresponding plurality of sub-blocks; fitting a simple transformation relationship in each sub-block to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates; for each sub-block, fitting the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and sub-block ranges; and reordering the sub-block ranges and the corresponding fitting parameters based on the nearest neighbor first calculation principle.

[0005] Optionally, the above method also includes: dividing the output image into multiple sub-blocks that are the same as the input image; obtaining the range of each sub-block of the output image mapped to the input image based on a simple transformation relationship to obtain the corresponding sub-block range in the input image; reordering the sub-blocks of the output image according to the order in which the sub-block ranges are arranged; searching for fitting parameters quantized to integers in a parameter table for each sub-block of the output image; and performing image deformation operations based on the fitting parameters quantized to integers to obtain pixel positions corresponding to the converted coordinates.

[0006] Optionally, the fitting parameters are quantized into integers, and the quantized fitting parameters and sub-block ranges are compressed and stored, including: multiplying the fitting parameters originally floating-point numbers by 2 to the power of n to amplify them, and then rounding the amplified floating-point numbers to obtain fixed-point numbers, and compressing and storing the fixed-point numbers in the parameter table.

[0007] Optionally, the sub-block ranges and corresponding fitting parameters are reordered based on the nearest neighbor first calculation principle, including: the sorting strategy uses the smallest vertical coordinate value in the sub-block range mapped from the output image to the input image as the sorting key, and sorts the sub-block range of the input image from small to large, so that the sub-blocks with the closest minimum vertical coordinates are always adjacent.

[0008] Optionally, the simple transformation relationship only includes multiplication and addition; wherein each simple transformation relationship has a corresponding local area, and the simple transformation relationship can approximately replace the original transformation relationship in the corresponding local area to improve computational efficiency.

[0009] Optionally, an image deformation operation is performed according to the fitting parameters quantized as integers to obtain pixel positions corresponding to the converted coordinates, including: performing a graphic deformation operation according to the fitting parameters quantized as integers and outputting the calculation result, dividing the calculation result by 2 to the nth power, and restoring the result to a fixed-point number with n decimal places to obtain the converted coordinates; interpolating pixel values ​​adjacent to the converted coordinates to obtain approximate pixel values ​​of the converted coordinates, and storing the approximate pixel values ​​to the pixel positions corresponding to the output image coordinates.

[0010] Optionally, in the operation of fitting the input image according to a simple transformation relationship for each sub-block to obtain at least two sub-block ranges and corresponding fitting parameters, a greedy algorithm and a least squares method are used for fitting to obtain the sub-block ranges and corresponding fitting parameters.

[0011] Another aspect of the present disclosure provides an image deformation device, comprising:

[0012] An input image cache module is used to receive instructions from the fixed-point calculation module, read input image data from the memory according to the instructions and send it to the fixed-point calculation module, and receive the converted coordinates sent by the fixed-point calculation module, and send the approximate pixel values ​​corresponding to the converted coordinates in the input image to the output image cache module; wherein the input image data includes pixels at each point of the input image and their corresponding coordinates;

[0013] The output image buffer module is used to receive the instruction of the fixed-point calculation module, read the output image data from the memory according to the instruction and send it to the fixed-point calculation module; and receive the approximate pixel value corresponding to the converted coordinate sent by the input image buffer module, and write it into the memory; wherein the output image data includes the size of the output image, the pixels of each point of the output image and their corresponding coordinates;

[0014] An index cache module is used to read the first data and the second data in the parameter table from the memory and send them to the fixed-point calculation module; wherein the first data includes the original transformation relationship from the output image to the input image; the second data is calculated by the fixed-point calculation module according to the output image data, the input image data and the first data, and includes the sub-block range sorting and the fitting parameters quantized into integers;

[0015] The fixed-point calculation module is used to receive output image data, input image data and first data, and perform the following operations according to the output image data, the input image data and the first data: block the output image according to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates to obtain corresponding multiple sub-blocks; fit a simple transformation relationship in each sub-block to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates; for each sub-block, fit the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters; quantize the fitting parameters into integers, compress and store the quantized fitting parameters and sub-block ranges; and reorder the sub-block ranges and corresponding fitting parameters based on the nearest neighbor first calculation principle.

[0016] Optionally, the fixed-point calculation module is also used to receive output image data, input image data and second data, and perform the following operations according to the output image data, input image data and second data: obtain the range of each sub-block of the output image mapped to the input image according to a simple transformation relationship, and obtain the corresponding sub-block range in the input image; reorder the sub-blocks of the output image according to the order in which the sub-block ranges are arranged; search the parameter table for fitting parameters quantized as integers for each sub-block of the output image; and perform image deformation operations according to the fitting parameters quantized as integers to obtain the pixel positions corresponding to the converted coordinates.

[0017] Another aspect of the present disclosure provides an image deformation system, including: a memory, and an image deformation device, wherein the memory is used to store an input image, an output image and a parameter table, and the image deformation device is used to implement each step of the above method to obtain the pixel position corresponding to the converted coordinates.

[0018] The present disclosure provides an image deformation method, device and system, which have the following beneficial effects:

[0019] On the one hand, after obtaining the fitting parameters in the training stage, the present disclosure quantizes the fitting parameters into integer storage (i.e., replaces floating-point storage with fixed-point numbers), so that fixed-point operations can be performed in the inference stage, thereby improving the resource utilization of the fixed-point computing module and accelerating the operation speed.

[0020] On the other hand, the present disclosure also reorders the sub-block parameters in the lookup table according to the principle of nearest neighbor first calculation and calculates them according to the new sub-block order in the inference stage, so that the fixed-point calculation module can maximize the use of data in the cache for calculation during the inference stage. This can avoid the fixed-point calculation module directly reading data from the memory, which can greatly improve the operating efficiency of the entire image deformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The image before and after the image deformation in the prior art is shown; wherein, Figure 1 a is the image before mapping, Figure 1 b is the image after mapping;

[0022] Figure 2A A flowchart of the training phase of the image deformation method in the embodiment of the present disclosure is schematically shown;

[0023] Figure 2B A flowchart of the inference phase of the image deformation method in the embodiment of the present disclosure is schematically shown;

[0024] Figure 3 A schematic diagram of a process of inverse mapping an output image to an input image according to a simple transformation relationship in an embodiment of the present disclosure is shown;

[0025] Figure 4 A block diagram of an image deformation device in an embodiment of the present disclosure is schematically shown;

[0026] Figure 5 A block diagram of an image deformation system according to an embodiment of the present disclosure is schematically shown.

[0027] Description of Reference Numerals

[0028] 1, 2, 3-dividing the output image into sub-blocks;

[0029] 1', 2', 3' - inverse mapping of sub-blocks 1, 2 and 3 in the output image to the corresponding sub-blocks in the input image;

[0030] P 1 , P 2 , P 3 , P 4 - Output the coordinates of the points in the image;

[0031] P 1 '、P 2 '、P 3 '-From the coordinate point P in the output image 1 , P 2 , P 3 Inversely map to the corresponding coordinate points in the input image;

[0032] 400 - image deformation device; 410 - input image cache module; 420 - index cache module; 430 - output image cache module; 440 - fixed-point calculation module; 500 - image deformation system; 510 - memory. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific details and with reference to the accompanying drawings.

[0034] Figure 1 2 shows the images before and after the image deformation in the prior art. Figure 1 As shown, in the image deformation process, usually not every pixel in the input image is defined in the output image, that is, the two are not one-to-one coordinate transformations, so if the input image is transformed to the output image, there will often be a lot of noise, that is, image deformation occurs. At present, most image deformation methods use the output image coordinates as input, inversely transform the original transformation to infer the corresponding coordinates of the output image coordinates in the input image, and finally fill the output image. This method is also used in the present disclosure to avoid noise.

[0035] As introduced in the background technology, most of the image deformation operations used in the prior art to restore distorted images are relatively complex, including complex mathematical operations such as exponentials and logarithms, and are often floating-point operations, which not only slows down the entire image processing program, but also wastes a lot of computing resources. These two points are more obvious on edge devices that have certain requirements for real-time performance and have fewer storage resources and computing resources.

[0036] Therefore, if the image deformation module can be accelerated, on the one hand, the speed of image deformation operation can be improved to meet the needs of more applications with real-time requirements; on the other hand, computing resources can be saved and more and larger images can be processed when computing resources are limited.

[0037] Based on this, the present disclosure proposes an image deformation method, an image deformation device and a system to solve at least one of the above problems.

[0038] It is understandable that, since the interaction between the logic operation module and the memory is 100 times slower than the interaction between the logic operation module and the cache, the present disclosure adds a corresponding cache module when designing the image deformation device. After the logic operation module reads the data point from the memory, the data of the adjacent 32 bytes of the data point is read into the cache module for storage, which does not increase the extra reading time. When the logic operation module needs to use the data, the logic operation module can directly read the data from the cache module without having to read it from the memory.

[0039] Figure 2A and Figure 2B The flowcharts of the training phase and the reasoning phase of the image deformation method in the embodiment of the present disclosure are schematically shown respectively.

[0040] like Figure 2A As shown, the training phase of the image deformation method in the present disclosure includes operations S101 to S105.

[0041] In operation S101, the output image is divided into blocks according to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates to obtain a corresponding plurality of sub-blocks.

[0042] The output image is divided into a plurality of sub-blocks according to the known output image size and the transformation relationship f from the output image coordinates to the input image coordinates (hereinafter, the transformation relationship refers to the transformation relationship from the output image coordinates to the input image coordinates). Each sub-block may be, for example, a rectangle of size m*n, where m and n may be specifically defined according to actual conditions.

[0043] In operation S102, a simple transformation relationship is fitted in each sub-block to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates.

[0044] Specifically, in operation S102, a simple transformation relationship f' is fitted in each sub-block using the least square method to replace the original transformation relationship f, wherein the simple transformation relationship f' can be, for example, a simple binary polynomial. According to an embodiment of the present disclosure, the simple transformation relationship f' only includes multiplication and addition so that the operation logic unit (such as a fixed-point computing unit) can calculate quickly. Each simple transformation relationship f' has a corresponding local area, and the simple transformation relationship f' can approximately replace the transformation relationship in the corresponding local area, thereby improving the operation efficiency.

[0045] In operation S103, for each of the sub-blocks, the input image is fitted according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters.

[0046] Specifically, in this step, a greedy algorithm is used to search whether there are sub-blocks around a certain sub-block that can be merged. Specifically, the sub-block is merged with its adjacent sub-blocks, and the simple transformation relationship f' is refitted (for example, the simple transformation relationship f' can be represented by a binary polynomial). After the fitting parameters are obtained, the difference between the fitting parameter transformation and the original simple transformation relationship f' is calculated. If the difference is less than a preset parameter α (where α can be customized according to actual conditions), the two sub-blocks are merged. This process continues until all mergeable sub-blocks are searched, and the final sub-block range and the corresponding fitting parameters are obtained.

[0047] In operation S104, the fitting parameters are quantized into integers, and the quantized fitting parameters and the sub-block range are compressed and stored.

[0048] In the prior art, image deformation operations often use floating-point operations, which not only slows down the entire image processing program, but also wastes a lot of computing resources. In order to solve this problem, the present disclosure quantizes floating-point numbers into fixed-point numbers for storage. Specifically, according to an embodiment of the present disclosure, the fitting parameters are quantized into integers, and the quantized fitting parameters and sub-block ranges are compressed and stored, including: multiplying the fitting parameters that were originally floating-point numbers by 2 to the nth power to enlarge them by n times, and then rounding the enlarged floating-point numbers to obtain fixed-point numbers, and finally compressing and storing the fixed-point numbers in a parameter table. The method of replacing floating-point numbers with fixed-point numbers for compressed storage ensures that the subsequent reasoning stage can use fixed-point numbers for calculations, and uses right shift operations to dequantize the calculation results. Since the added right shift operation takes almost no time, it is equivalent to replacing a large number of floating-point calculations with fixed-point operations at no cost in the reasoning stage.

[0049] In operation S105 , the sub-block ranges and the corresponding fitting parameters are reordered based on the nearest neighbor first calculation principle.

[0050] According to an embodiment of the present disclosure, the sub-block ranges and the corresponding fitting parameters are reordered based on the principle of nearest neighbor calculation first, including: the sorting strategy uses the smallest vertical coordinate value in the sub-block range mapped from the output image to the input image as the sorting key, and sorts the sub-block range of the input image from small to large, so that the sub-blocks with the closest minimum vertical coordinates are always adjacent.

[0051] Specifically, in operation S105, the sub-block ranges and corresponding fitting parameters stored in the parameter table are reordered according to the principle of nearest neighbor first calculation. The purpose of this is mainly to maximize the utilization of the logical operation unit (eg Figure 5 The interaction between the fixed-point computing unit (shown in Figure 1) and the cache module.

[0052] It is understandable that the logic operation unit (eg Figure 5 When a logic operation unit (such as a fixed-point computing unit) fetches data from the memory, it will not only read the currently required data, but also continuously read 32-byte or 64-byte data and temporarily store it in the cache module. When a logic operation unit (such as a fixed-point computing unit) needs to fetch data, it will first check whether there is the required data in the cache module. If not, it will send a data request to the memory. Since the data in the cache module is time-sensitive, in order to improve the logic operation unit (such as Figure 5 The fixed-point computing unit shown in FIG. 100 reads the hit rate of the data, and in operation S105, the order of the sub-block range is reordered so that the sub-block currently being calculated is always closest to the sub-block last being calculated, which can greatly shorten the data transmission time.

[0053] Figure 3 The schematic diagram of the process of inverse mapping the output image to the input image according to the simple transformation relationship in the embodiment of the present disclosure is shown. Figure 3 The content shown in the figure describes the process of operation S105 in the embodiment of the present disclosure in detail. It should be noted that in the embodiment of the present disclosure, the number of sub-blocks of the output image and the input image is not limited to 3, and the number of sub-blocks can be defined according to actual conditions. Figure 3 The number of sub-blocks shown is only for facilitating the understanding of the content of the present disclosure and is not intended to limit the content protected by the present disclosure.

[0054] like Figure 3 As shown in Figure 2, in the image deformation operation, adjacent regular rectangles in the output image will become irregular shapes when inversely mapped to the input image, and their relative positions will change (for example, Figure 3 1 is inversely mapped from the output image to the input image). Under the premise that the image deformation operation process is to inversely map the output image to the input image, it can be known that if the sub-blocks are calculated according to the adjacent order of the output image, it is easy to cause discontinuity in the reading of the input image during the operation. If we calculate sub-block 1 and then calculate sub-block 2, since sub-block 1 and sub-block 2 are not continuous in rows, the logical operation unit (for example Figure 5 The fixed-point computing unit shown in FIG. 1 has to re-read the data of the row where sub-block 2 is located from the memory and clear the data of the row where sub-block 1 is located stored in the cache. Figure 3 It can be seen from the figure that several rows of sub-block 1 and sub-block 3 are located on the same row, which means that the logic operation unit (e.g. Figure 5 The fixed-point computing unit shown in FIG) is likely to have read the data to be used by sub-block 3 on these three rows into the cache (because the logic operation unit (e.g. Figure 5 The fixed-point computing unit shown in FIG. 1 reads data usually by row. Therefore, a reasonable solution is to first calculate sub-block 3 which has more overlaps with sub-block 1 in rows.

[0055] Please continue to see Figure 3 Each item in the parameter table consists of two parts: coordinate value and simple function parameter. The first part of the coordinate value includes the coordinate value of the upper left corner and the coordinate value of the lower right corner of each output image sub-block. For example, the coordinate value of output image sub-block 1 (1, x min ,y min ,,x max ,y max ) is the coordinate point P of the upper left corner in sub-block 1 1 'coordinate value (where P 1 The coordinates of ' are (x min ,y min) and the coordinate point P of the lower right corner 4 '(x max ,y max ). The second part is the parameters of the simple function obtained by fitting. In the sorting process, the range of sub-blocks of all output images mapped to the sub-blocks in the input image is first obtained by a simple function, and then the minimum vertical coordinate of each mapped sub-block in the input image is found, and the sub-blocks with the closest minimum vertical coordinates in the input image are arranged together, so that the sub-blocks with the closest minimum vertical coordinates are always adjacent. The re-ordered sub-blocks can be obtained through the above operations.

[0056] like Figure 2B As shown, the inference stage of the image deformation method in the present disclosure includes operations S106 to S110.

[0057] In operation S106, the output image is divided into a plurality of sub-blocks that are the same as the input image.

[0058] Specifically, in the inference phase, the output image is divided into multiple sub-blocks that are the same as the input image. Figure 3 ,like Figure 3 As shown, for example, the output image can be divided into three sub-blocks, including sub-block 1, sub-block 2 and sub-block 3. It should be noted that in the embodiment of the present disclosure, the number of sub-blocks of the output image and the input image is not limited to 3, and can also be defined according to actual conditions. Figure 3 The number of sub-blocks shown is only for facilitating the understanding of the content of the present disclosure and is not intended to limit the content protected by the present disclosure.

[0059] In operation S107, the range of each sub-block of the output image mapped to the input image is obtained according to a simple transformation relationship to obtain the corresponding sub-block range in the input image.

[0060] like Figure 3 As shown, according to the simple transformation relationship f', each sub-block of the output image (such as sub-block 1, sub-block 2 and sub-block 3) is inversely mapped to the range in the input image to obtain the corresponding sub-block range in the input image (such as sub-block 1', sub-block 2' and sub-block 3').

[0061] In operation S108, the sub-blocks of the output image are reordered in the order in which the sub-block ranges are arranged.

[0062] As mentioned above, during the image deformation operation, adjacent regular rectangles in the output image will become irregular shapes when mapped to the input image, and their relative positions will change. Under the premise that the image deformation operation is to reversely map the output image to the input image, it can be understood that if the sub-blocks are calculated according to the adjacent order of the output image, it is easy to cause discontinuity in reading the input image. Therefore, in operation S108, the sub-blocks of the output image are reordered according to the order of arrangement of the sub-block ranges in the parameter table obtained in the aforementioned training stage, so that the sub-blocks with the closest minimum ordinates are always adjacent. This is beneficial to the logic operation unit (for example Figure 5 The fixed-point calculation module shown in the figure can read data continuously during the subsequent calculation process, greatly compressing the time of data transmission.

[0063] In operation S109, a fitting parameter quantized into an integer is searched in a parameter table for each sub-block of the output image.

[0064] Among them, each sub-block of the output image searches for fitting parameters quantized as integers (ie, fixed-point parameters) in the parameter table, and then participates in the image deformation operation with fixed-point numbers.

[0065] In operation S110, an image deformation operation is performed according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the converted coordinates.

[0066] According to an embodiment of the present disclosure, in operation S110, specifically, the following is further included:

[0067] In operation S1101, an image deformation operation is performed according to the fitting parameters quantized as integers stored in the parameter table and the calculation result is output. The calculation result is divided by 2 to the power of n, and the result is restored to a fixed-point number with n decimal places to obtain the converted coordinates (i.e., the coordinates inversely mapped from the output image to the input image).

[0068] In operation S1102, pixel values ​​adjacent to the converted coordinates are obtained for interpolation to obtain approximate pixel values ​​of the converted coordinates, and the approximate pixel values ​​are stored in pixel positions corresponding to the output image coordinates.

[0069] The disclosed embodiment proposes an image deformation method. After obtaining the fitting parameters in the training phase, the fitting parameters are quantized into integer storage (i.e., fixed-point numbers are used instead of floating-point numbers for storage), so that fixed-point operations can be performed in the inference phase, thereby improving the logic operation unit (e.g. Figure 5 On the other hand, the method also reorders the sub-block parameters in the lookup table according to the principle of nearest neighbor first calculation and calculates them according to the new sub-block order in the inference stage, so that the logical operation unit (such as Figure 5The fixed-point calculation module shown in ) can make maximum use of the data in the cache for calculation, which can avoid reading data directly from the memory and greatly improve the operating efficiency of the entire image deformation operation.

[0070] Figure 4 The block diagram of the image deformation device 400 in the embodiment of the present disclosure is schematically shown. Figure 4 As shown, the image deformation device 400 includes: an input image buffer module 410 , an index buffer module 420 , an output image buffer module 430 and a fixed-point calculation module 440 .

[0071] The input image buffer module 410 is used to receive instructions from the fixed-point calculation module 440 and obtain the image from the memory (indicated at Figure 5 ) reads the input image data and sends it to the fixed-point calculation module 440, and receives the converted coordinates sent by the fixed-point calculation module 440, and sends the approximate pixel values ​​corresponding to the converted coordinates in the input image to the output image cache module 430; wherein the input image data includes the pixels of each point of the input image and their corresponding coordinates.

[0072] The output image buffer module 430 is used to receive the instruction of the fixed-point calculation module 440, and obtain the image from the memory (indicated at Figure 5 ) reads the output image data and sends it to the fixed-point calculation module 440; and receives the approximate pixel value corresponding to the converted coordinate sent by the input image buffer module 410, and writes it into the memory (indicated in Figure 5 wherein the output image data includes the size of the output image, pixels of each point of the output image and their corresponding coordinates.

[0073] The index cache module 420 is used to retrieve the index from the memory (indicated at Figure 5 ) reads the first data and the second data in the parameter table and sends them to the fixed-point calculation module 440; wherein the first data includes the original transformation relationship from the output image to the input image; the second data is calculated by the fixed-point calculation module 440 according to the output image data, the input image data and the first data, including the sub-block range sorting and the fitting parameters quantized into integers.

[0074] The fixed-point calculation module 440 is used to receive output image data, input image data and first data, and perform the following operations according to the output image data, the input image data and the first data: block the output image according to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates to obtain a corresponding plurality of sub-blocks; fit a simple transformation relationship in each sub-block to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates; for each sub-block, fit the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters; quantize the fitting parameters into integers, and compress and store the quantized fitting parameters and sub-block ranges; and reorder the sub-block ranges and the corresponding fitting parameters based on the nearest neighbor first calculation principle.

[0075] According to an embodiment of the present disclosure, the above-mentioned fixed-point calculation module 440 is also used to receive output image data, input image data and second data, and perform the following operations according to the output image data, input image data and second data: divide the output image into multiple sub-blocks that are the same as the input image; obtain the range of each sub-block of the output image mapped to the input image according to a simple transformation relationship, and obtain the corresponding sub-block range in the input image; reorder the sub-blocks of the output image according to the order in which the sub-block ranges are arranged; search the parameter table for fitting parameters quantized as integers for each sub-block of the output image; perform image deformation operations according to the fitting parameters quantized as integers to obtain the pixel positions corresponding to the converted coordinates.

[0076] The image deformation device proposed in the present disclosure can not only improve the image deformation operation speed and meet the needs of more applications with real-time requirements, but also save computing resources and can process more and larger images when computing resources are limited.

[0077] According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits, or at least part of the functions of any one of them can be implemented in one module. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be split into multiple modules for implementation. According to the embodiments of the present invention, any one or more of the modules, submodules, units, and subunits can be at least partially implemented as hardware circuits, such as field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems on chips, systems on substrates, systems on packages, application specific integrated circuits (ASICs), or can be implemented by hardware or firmware in any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, according to the embodiments of the present invention, one or more of the modules, submodules, units, and subunits can be at least partially implemented as computer program modules, and when the computer program modules are run, the corresponding functions can be performed.

[0078] For example, any multiple of the input image cache module 410, the index cache module 420, the output image cache module 430, and the fixed-point calculation module 440 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the input image cache module 410, the index cache module 420, the output image cache module 430, and the fixed-point calculation module 440 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or by any one of the three implementation methods of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the input image cache module 410, the index cache module 420, the output image cache module 430, and the fixed-point calculation module 440 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0079] Figure 5 A system block diagram 500 suitable for implementing the above-described image deformation method according to an embodiment of the present disclosure is schematically shown. Figure 5 The system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0080] like Figure 5 As shown, the system 500 includes: a memory 510 and an image deformation device 400, wherein the memory 510 is used to store an input image, an output image and a parameter table, and the image deformation device 400 is used to implement each step of the above method to obtain the pixel position corresponding to the converted coordinates.

[0081] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0082] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways without departing from the spirit and teachings of the present disclosure. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0083] Although the present disclosure has been shown and described with reference to specific exemplary embodiments of the present disclosure, it should be understood by those skilled in the art that various changes in form and details may be made to the present disclosure without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents. Therefore, the scope of the present disclosure should not be limited to the above-mentioned embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.

Claims

1. An image deformation method, characterized in that: include: According to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates, the output image is divided into blocks to obtain corresponding multiple sub-blocks; Fitting a simple transformation relationship in each of the sub-blocks to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates; the simple transformation relationship includes multiplication and addition; For each of the sub-blocks, fitting the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the sub-block range; Reordering the sub-block ranges and the corresponding fitting parameters based on the principle of nearest neighbor first calculation; Dividing the output image into a plurality of sub-blocks that are the same as the input image; Obtaining a range of each sub-block of the output image mapped to the input image according to the simple transformation relationship, and obtaining a corresponding sub-block range in the input image; Reordering the sub-blocks of the output image according to the order in which the sub-block ranges are arranged; For each of the sub-blocks of the output image, searching a parameter table for a fitting parameter quantized into an integer; An image deformation operation is performed according to the fitting parameters quantized as integers to obtain the pixel positions corresponding to the transformed coordinates.

2. The method according to claim 1, characterized in that: The step of quantizing the fitting parameters into integers and compressing and storing the quantized fitting parameters and the sub-block range comprises: The fitting parameter originally a floating point number is multiplied by 2 raised to the nth power to be amplified, and then the amplified floating point number is rounded off to obtain a fixed point number, and the fixed point number is compressed and stored in a parameter table.

3. The method according to claim 1, characterized in that The reordering of the sub-block ranges and the corresponding fitting parameters based on the principle of nearest neighbor first calculation includes: The sorting strategy uses the smallest vertical coordinate value in the sub-block range mapped from the output image to the input image as the sorting key, and sorts the sub-block range of the input image from small to large so that the sub-blocks with the closest minimum vertical coordinates are always adjacent.

4. The method according to claim 1, characterized in that: Each of the simple transformation relationships has a corresponding local area, and in the corresponding local area, the simple transformation relationship can approximately replace the original transformation relationship to improve the calculation efficiency.

5. The method according to claim 1, characterized in that The image deformation operation is performed according to the fitting parameters quantized as integers to obtain the pixel positions corresponding to the transformed coordinates, including: Performing a graphic deformation operation according to the fitting parameters quantized into integers and outputting a calculation result, dividing the calculation result by 2 raised to the nth power, and restoring the result to a fixed-point number with n decimal places to obtain the converted coordinates; Obtain pixel values ​​adjacent to the converted coordinates for interpolation to obtain approximate pixel values ​​of the converted coordinates, and store the approximate pixel values ​​at pixel positions corresponding to the output image coordinates.

6. The method according to claim 1, characterized in that In the operation of fitting the input image according to the simple transformation relationship for each sub-block to obtain at least two sub-block ranges and corresponding fitting parameters, a greedy algorithm and a least squares method are used for fitting to obtain the sub-block ranges and the corresponding fitting parameters.

7. An image deformation device, characterized in that: include: An input image cache module is used to receive instructions from a fixed-point calculation module, read input image data from a memory according to the instructions and send the data to the fixed-point calculation module, receive converted coordinates sent by the fixed-point calculation module, and send the approximate pixel values ​​corresponding to the converted coordinates in the input image to an output image cache module; wherein the input image data includes pixels at each point of the input image and their corresponding coordinates; The output image buffer module is used to receive the instruction of the fixed-point calculation module, read the output image data from the memory according to the instruction and send it to the fixed-point calculation module; and receive the approximate pixel value corresponding to the converted coordinate sent by the input image buffer module, and write it into the memory; wherein the output image data includes the size of the output image, the pixels of each point of the output image and their corresponding coordinates; An index cache module, used for reading first data and second data in the parameter table from the memory and sending them to the fixed-point calculation module; wherein the first data includes the original transformation relationship from the output image to the input image; the second data is calculated by the fixed-point calculation module according to the output image data, the input image data and the first data, and includes sub-block range sorting and fitting parameters quantized into integers; The fixed-point calculation module is used to receive the output image data, the input image data and the first data, and perform the following operations according to the output image data, the input image data and the first data: According to the size of the output image and the original transformation relationship from the output image coordinates to the input image coordinates, the output image is divided into blocks to obtain corresponding multiple sub-blocks; Fitting a simple transformation relationship in each of the sub-blocks to approximately replace the original transformation relationship from the sub-block coordinates to the input image coordinates; For each of the sub-blocks, fitting the input image according to the simple transformation relationship to obtain at least two sub-block ranges and corresponding fitting parameters; quantizing the fitting parameters into integers, and compressing and storing the quantized fitting parameters and the sub-block range; The sub-block ranges and the corresponding fitting parameters are reordered according to the principle of nearest neighbor first calculation. The fixed-point calculation module is also used to receive the output image data, the input image data and the second data, and perform the following operations according to the output image data, the input image data and the second data: Dividing the output image into a plurality of sub-blocks that are the same as the input image; Obtaining a range of each sub-block of the output image mapped to the input image according to the simple transformation relationship, and obtaining a corresponding sub-block range in the input image; Reordering the sub-blocks of the output image according to the order in which the sub-block ranges are arranged; For each of the sub-blocks of the output image, searching a parameter table for a fitting parameter quantized into an integer; An image deformation operation is performed according to the fitting parameters quantized into integers to obtain pixel positions corresponding to the transformed coordinates.

8. An image deformation system, characterized in that: include: A memory and an image deformation device, wherein the memory is used to store the input image, the output image and the parameter table, and the image deformation device is used to perform image deformation operations according to the method described in any one of claims 1 to 6 to obtain pixel positions corresponding to the converted coordinates.

Images