Image processing method and device, processor and equipment

Abstract

The invention discloses an image processing method and device, a processor and equipment, and relates to the field of computers. The method comprises the following steps: according to a vector calculation instruction, performing parallel distortion processing on a plurality of pixels in a to-be-processed image corresponding to elements in a second distortion field after pooling of a first distortion field of the to-be-processed image to obtain an intermediate processing image; and performing pixel arrangement on the plurality of pixels included in the pixels in the intermediate processing image according to a recombination rule to obtain a distorted image of the to-be-processed image. Wherein the second twisting field comprises elements pooling the first twisting field. The intermediate processing image comprises a distortion processing result of the recombined pixels after the pixels in the to-be-processed image are recombined according to the recombination rule. The recombined pixels comprise a plurality of pixels in the to-be-processed image. The recombination rule is used for indicating an arrangement rule of a plurality of pixels in the to-be-processed image in the recombination pixels in the recombination pixels. And through vectorization acceleration in the distortion processing process, the calculation performance of image distortion processing is improved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to an image processing method, apparatus, processor, and device. Background Art

[0002] Currently, image warping techniques are used to perform rendering operations such as interpolating frames or adding special image effects to videos to enhance the user's viewing experience. Typically, the processor performs image warping on the image being processed based on scalar computation instructions. Because scalar computation instructions specify processing of a single data point, the processor must execute these scalar computation instructions multiple times to obtain pixels in the image being processed and perform image warping on those pixels. Consequently, the computational performance of image warping is poor. Summary of the Invention

[0003] The present application provides an image processing method, apparatus, processor and device, thereby improving the computing performance of image distortion processing based on the vector computing capability of the processor.

[0004] In a first aspect, an image processing method is provided, comprising: performing warping processing in parallel on a plurality of pixels in an image to be processed corresponding to elements in a second warping field after pooling a first warping field of the image to be processed according to a vector calculation instruction, to obtain an intermediate processed image. Arranging the plurality of pixels contained in the pixels in the intermediate processed image according to a reorganization rule, to obtain a warped image of the image to be processed. The second warping field comprises elements after pooling the first warping field of the image to be processed. The intermediate processed image comprises a warping processing result of reorganized pixels after reorganizing the pixels in the image to be processed according to the reorganization rule. The reorganized pixels comprise a plurality of pixels in the image to be processed. The reorganization rule is used to indicate an arrangement rule of the plurality of pixels in the image to be processed in the reorganized pixels.

[0005] Compared with performing image distortion processing on the image to be processed according to scalar calculation instructions, since the processor needs to execute scalar calculation instructions multiple times to realize image distortion processing of the image to be processed, the computational performance of image distortion processing is poor. The image processing method provided by the present application, on the one hand, due to the characteristic of the consistency of the motion of adjacent pixels during image distortion motion, the first distortion field of the image to be processed is pooled to reduce the number of elements contained in the distortion field under the premise of ensuring that the loss of image distortion to the image is small; on the other hand, the pixels in the image to be processed are reorganized so that a reorganized pixel contains multiple pixels, and then, according to the vector calculation instruction, the distortion processing is performed on the image to be processed by the second distortion field after pooling the first distortion field through vector calculation, that is, executing the vector calculation instruction once, the distortion processing can be performed on multiple pixels in the image to be processed in parallel, completing the image distortion processing of the image to be processed, reducing the number of execution times of the instruction, and thus improving the computational performance of the image distortion processing through vectorization acceleration of the distortion processing process.

[0006] In a possible implementation, the reconstructed pixels indicated by the elements in the second warp field include consecutive storage addresses of a plurality of pixels in the image to be processed.

[0007] Due to the small storage capacity of the processor's on-chip cache, the processor cannot load the complete image to be processed. In addition, the storage addresses of multiple pixels in the image to be processed in the memory may be discontinuous. Therefore, the processor needs to frequently access the memory to load the data of the image to be processed stored in the memory into the processor's on-chip cache. The overhead of data transfer becomes a computing performance bottleneck for image distortion processing, wasting the processor's computing resources.

[0008] The image processing method provided in this application reorganizes the pixels in the image being processed so that the storage addresses of multiple pixels required for a single vector calculation are continuous. This reduces the number of memory accesses required by the processor, thereby reducing data movement overhead and improving the computational performance of image distortion processing. Furthermore, by fully utilizing the processor's vector computing capabilities, the utilization rate of the processor's computing resources is also improved.

[0009] In another possible implementation, warping processing is performed in parallel on multiple pixels in the image to be processed obtained based on elements in the second warping field according to a vector calculation instruction, including: step 1: obtaining a first recombined pixel indicated by a first element in the second warping field and multiple recombined pixels related to the interpolation calculation of the first recombined pixel, the first recombined pixel including multiple pixels in the image to be processed, and the first element being any element in the second warping field; step 2: performing interpolation calculation in parallel on the first recombined pixel and multiple recombined pixels related to the interpolation calculation of the first recombined pixel according to the vector calculation instruction; traversing the elements in the second warping field, repeating steps 1 and 2 to obtain an intermediate processed image.

[0010] In another possible implementation, obtaining the first reconstructed pixel indicated by the first element in the second warp field includes: obtaining the first reconstructed pixel indicated by the first element in the second warp field from the reconstructed image. The second warp field is used to indicate an offset rule for reconstructed pixels in the reconstructed image.

[0011] Before performing the warping process on the image to be processed, the pixels in the image to be processed may be reassembled. When the warping process is performed on the image to be processed, the first reassembled pixel indicated by the first element in the second warping field can be obtained from the reassembled image as quickly as possible, thereby improving the computational performance of the image warping process.

[0012] In another possible implementation, obtaining a first reconstructed pixel indicated by a first element in the second warp field and a plurality of reconstructed pixels related to an interpolation calculation of the first reconstructed pixel includes: forming the first reconstructed pixel from a plurality of pixels in the image to be processed determined based on a pixel in the image to be processed corresponding to the first element in the second warp field and a pooling area; and determining a plurality of reconstructed pixels related to an interpolation calculation of the first reconstructed pixel based on the first reconstructed pixel and a pooling step size.

[0013] If the processor has the ability to collect and reorganize data, when performing distortion processing on the image to be processed, the pixels in the image to be processed can be obtained in real time for reorganization, thereby saving the storage space occupied by storing the reorganized image.

[0014] In another possible implementation, before obtaining an intermediate processed image, warping processing is performed in parallel on multiple pixels in the image to be processed obtained based on the elements in the second warp field according to a vector calculation instruction, the method further includes: dividing the first warp field according to a pooling step size to obtain multiple pooling areas, each pooling area including multiple elements in the first warp field; and pooling the multiple elements contained in each of the multiple pooling areas to obtain a second warp field.

[0015] Due to the consistency of adjacent pixel motion during image distortion, the motion of adjacent pixels is similar when the image is warped. The image is divided into multiple pooling regions, each containing multiple pixels with similar motion. Pooling these regions reduces the number of pixels with similar motion, thereby minimizing the loss of the image warping process and reducing the number of elements in the warp field. This improves the computational performance of the image warping process.

[0016] In another possible implementation, before performing warping processing in parallel on multiple pixels in the image to be processed obtained based on the elements in the second warping field according to the vector calculation instruction to obtain an intermediate processed image, the method further includes: dividing the image to be processed according to the pooling step size to obtain multiple recombined regions, each recombined region including multiple pixels in the image to be processed; and recombining multiple pixels contained in each of the multiple recombined regions according to the recombination rule to obtain a recombined image.

[0017] By reorganizing the pixels in the image being processed so that the storage addresses of multiple pixels in the reorganized image are continuous, a single vector calculation can retrieve the required pixel data. This reduces the number of memory accesses required by the processor, thus reducing data movement overhead and improving the computational performance of image distortion processing.

[0018] In another possible implementation, the method further includes: determining a pooling step size according to a data length indicated by the vector calculation instruction and a data length of pixels in the image to be processed.

[0019] The pooling step size is determined appropriately based on the data length indicated by the vector calculation instruction and the data length of the pixels in the image being processed. This allows the data length of the reorganized pixels to be controlled when the pixels in the image being processed are reorganized according to the pooling step size. This prevents the reorganized pixels from having excessively long data lengths, which would prevent a single vector calculation from processing data exceeding the data length indicated by the vector calculation instruction and thus affect the computational performance of the image distortion process.

[0020] In another possible implementation, the method further includes: determining a pooling step size according to a data length indicated by the vector calculation instruction, a data length of pixels in the image to be processed, and a resolution of the image to be processed.

[0021] For images with higher resolution, the pooling step size is reasonably determined based on the data length indicated by the vector calculation instruction, the data length of the pixels in the image to be processed, and the resolution of the image to be processed to control the data length of the reorganized pixels.

[0022] In a second aspect, an image processing apparatus is provided, the image processing apparatus comprising modules for executing the image processing method in the first aspect or any possible design of the first aspect. For example, the image processing apparatus comprises a communication module and a distortion processing module.

[0023] The communication module is used to obtain the image to be processed and the second warp field obtained by pooling the first warp field of the image to be processed.

[0024] A distortion processing module is used to perform distortion processing in parallel on multiple pixels in the image to be processed obtained based on elements in the second distortion field according to vector calculation instructions to obtain an intermediate processed image, wherein the second distortion field includes elements after pooling the first distortion field of the image to be processed, the first distortion field is used to indicate the offset rule of the pixels in the image to be processed, and the intermediate processed image includes the distortion processing result of the recombined pixels after the pixels in the image to be processed are recombined according to the recombination rule, the recombined pixels include multiple pixels in the image to be processed, and the recombination rule is used to indicate the arrangement rule of the recombined pixels constituted by the multiple pixels in the image to be processed.

[0025] The distortion processing module is further used to arrange the pixels of the intermediate processed image according to the reorganization rule to obtain a distorted image of the image to be processed.

[0026] In a possible implementation, the reconstructed pixels indicated by the elements in the second warp field include consecutive storage addresses of a plurality of pixels in the image to be processed.

[0027] In another possible implementation, when the distortion processing module performs distortion processing in parallel on multiple pixels in the image to be processed obtained based on the elements in the second distortion field according to the vector calculation instruction, the module has the following steps: Step 1: Obtain a first recombined pixel indicated by a first element in the second distortion field and multiple recombined pixels related to the interpolation calculation of the first recombined pixel, where the first element is any element in the second distortion field; Step 2: Perform interpolation calculation in parallel on the first recombined pixel and multiple recombined pixels related to the interpolation calculation of the first recombined pixel according to the vector calculation instruction; traverse the elements in the second distortion field and repeat Steps 1 and 2 to obtain an intermediate processed image.

[0028] In another possible implementation, when the distortion processing module obtains the first reconstructed pixel indicated by the first element in the second distortion field, it is configured to: obtain the first reconstructed pixel indicated by the first element in the second distortion field from the reconstructed image, where the second distortion field is used to indicate an offset rule for the reconstructed pixel in the reconstructed image.

[0029] In another possible implementation, when the distortion processing module obtains the first reconstructed pixel indicated by the first element in the second distortion field and multiple reconstructed pixels related to the interpolation calculation of the first reconstructed pixel, it is configured to: form the first reconstructed pixel based on the pixel in the image to be processed corresponding to the first element in the second distortion field and multiple pixels in the image to be processed determined by the pooling area; and determine multiple reconstructed pixels related to the interpolation calculation of the first reconstructed pixel based on the first reconstructed pixel and the pooling step size.

[0030] In another possible implementation, the distortion processing module is also used to divide the first distortion field according to the pooling step size to obtain multiple pooling areas, each pooling area includes multiple elements in the first distortion field; pooling the multiple elements contained in each pooling area in the multiple pooling areas to obtain a second distortion field.

[0031] In another possible implementation, the distortion processing module is further used to divide the image to be processed according to the pooling step size to obtain multiple reconstructed regions, each reconstructed region includes multiple pixels in the image to be processed; and the multiple pixels contained in each reconstructed region in the multiple reconstructed regions are reconstructed according to the reconstructing rule to obtain a reconstructed image.

[0032] In another possible implementation, the distortion processing module is further configured to determine a pooling step size according to a data length indicated by the vector calculation instruction and a data length of pixels in the image to be processed.

[0033] In another possible implementation, the distortion processing module is further configured to determine a pooling step size according to a data length indicated by the vector calculation instruction, a data length of pixels in the image to be processed, and a resolution of the image to be processed.

[0034] In a third aspect, a processor is provided, which includes a logic circuit and a power supply circuit; wherein the power supply circuit is used to supply power to the logic circuit; and the logic circuit is used to execute the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0035] The processor has vector processing capabilities. For example, the processor includes at least one of a central processing unit (CPU) or a neural processing unit (NPU).

[0036] In a fourth aspect, a computer device is provided, which includes at least one processor and a memory, the memory being used to store a set of computer instructions; when the processor executes the set of computer instructions, the operating steps of the method described in the first aspect or any possible implementation of the first aspect are performed.

[0037] In a fifth aspect, a computer-readable storage medium is provided, comprising: computer software instructions; when the computer software instructions are executed in a computer device, the computer device executes the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0038] In a sixth aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the operating steps of the method described in the first aspect or any possible implementation of the first aspect.

[0039] The technical effects brought about by any design method in the second to sixth aspects can be referred to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here.

[0040] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of image distortion provided by this application;

[0042] Figure 2 A schematic diagram of the structure of a computer device provided in this application;

[0043] Figure 3 A flowchart of an image processing method provided in this application;

[0044] Figure 4 A schematic diagram of a pooling of a twisted field provided in this application;

[0045] Figure 5 A schematic diagram of image reconstruction provided by this application;

[0046] Figure 6 A flowchart of an image processing method provided in this application;

[0047] Figure 7 A schematic diagram of the process of an image processing method provided in this application;

[0048] Figure 8 A schematic diagram of the process of an image processing method provided in this application;

[0049] Figure 9 This is a structural diagram of an image processing device provided in this application. DETAILED DESCRIPTION

[0050] To facilitate understanding, the main terms involved in this application are first explained.

[0051] Image warping: A digital image processing process, a type of planar graphic transformation, involves resampling the original image according to a rule to transform its geometry. Image warping is used in scenarios such as image effects, image distortion correction, and optical flow interpolation.

[0052] For example, Figure 1This is a schematic diagram of an image distortion method provided by the present application. The positions of pixels in the original image are changed based on a distortion field, and interpolation calculations are performed on the pixels in the original image indicated by the elements in the distortion field to achieve the distortion effect of the original image.

[0053] The warp field is used to indicate the positional correspondence between pixels in the original image and the warped image. For example, the position of an element in the warp field corresponds to the position of a pixel in the warped image, and the element in the warp field contains the coordinates of a pixel in the original image. This means that the pixel in the original image indicated by the element in the warp field is offset to the position of the element. For another example, the position of an element in the warp field corresponds to the position of a pixel in the original image, and the element in the warp field contains an offset value. This means that the pixel in the original image corresponding to the position of the element in the warp field is offset according to the offset value.

[0054] Pooling: A type of downsampling used to reduce the dimension of data, reduce the amount of data, and avoid overfitting. Pooling includes max pooling and average pooling.

[0055] Vector Processor: A vector processor is used to perform operations on one-dimensional arrays (e.g., vectors) according to instructions. Vector processors are also called array processors. Common instructions include the single instruction multiple data (SIMD) instruction set.

[0056] Single instruction multiple data (SIMD) is a technology that achieves spatial parallelism by performing the same operation on each piece of data in a set of data (also known as a "data vector"). SIMD is an extension of the basic instruction set of the central processing unit (CPU).

[0057] Scalable Vector Extension (SVE): A SIMD instruction set for the ARM AArch64 architecture, designed to accelerate high-performance computing. It supports variable vector lengths, per-lane prediction, gather-load and scatter-store, and horizontal operations.

[0058] With the development of chip computing power, processors have acquired vector computing capabilities. It is hoped that using this processor's vector processing capabilities to perform image warping processing on multiple pixels in the image to be processed in parallel will improve the computational performance of image warping. However, the warping field reflects the motion of pixels within the image, and pixels within the image have displacements in both the x- and y-directions. The storage addresses of the multiple pixels required for a single vector calculation may be discontinuous. When the image motion is large, the processor's on-chip storage medium (such as memory or cache) may not contain the pixels required for vector calculation, and the required pixels must be collected from storage media outside the processor. Understandably, due to the limited storage capacity of the processor's on-chip cache, the processor cannot load the complete image to be processed and the warping field. Furthermore, since the storage addresses in memory used to search for pixels in the image to be processed based on the warping field may be discontinuous, the processor needs to frequently access memory to load the pixels of the image to be processed stored in memory into the processor's on-chip cache. This data transfer overhead becomes a bottleneck in the computational performance of image warping, wasting the processor's computing resources.

[0059] In addition, when using the CPU to perform image warping, due to the poor parallel computing capability of the CPU, the image warping cannot be performed on images with larger resolutions, and the computing performance is poor. In addition, the problem of finding the non-continuous storage addresses of pixels in the image to be processed based on the warping field during the warping process still exists.

[0060] In order to solve the problem of poor computational performance of image distortion processing. The present application provides an image processing method, that is, based on the consistency of the motion of adjacent pixels during image distortion motion, a first distortion field of the image to be processed is pooled to obtain a second distortion field, and then the pixels in the image to be processed are reorganized so that the storage addresses of the reorganized pixels indicated by continuous elements in the second distortion field are continuous. When performing image distortion processing on the image to be processed, distortion processing is performed in parallel on the reorganized pixels corresponding to the elements in the second distortion field according to vector calculation instructions to obtain an intermediate processed image. According to the reorganization rule, the pixels contained in the pixels in the intermediate processed image are arranged to obtain a distorted image of the image to be processed. The second distortion field includes the elements after pooling the first distortion field. The intermediate processed image includes the distortion processing results of the reorganized pixels after the pixels in the image to be processed are reorganized according to the reorganization rule. The reorganized pixels include multiple pixels in the image to be processed. The reorganization rule is used to indicate the arrangement rule of multiple pixels in the image to be processed in the reorganized pixels.

[0061] While ensuring minimal image distortion, the number of elements contained in the distortion field is reduced. Furthermore, based on vector calculation instructions, the image being processed is warped using a second distortion field pooled from the first distortion field. This means that executing a single vector calculation instruction can perform distortion processing on multiple pixels in the image being processed in parallel, completing the image distortion processing for the image being processed. This reduces the number of instruction executions, thereby improving the computational performance of the image distortion processing through vectorized acceleration of the distortion processing process.

[0062] The image processing method provided in this application is applied to digital image processing scenarios, specifically image distortion processing. For example, application scenarios include interpolation technology based on optical flow method, distortion effects of video images, and pre-distortion of projection display systems. The execution subject of the image processing method provided in this application mainly includes a processor with vector processing capabilities, and the processor with vector processing capabilities includes vector processing instructions such as SVE instructions and SIMD instructions. For example, CPU, NPU, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc.

[0063] The following describes in detail the implementation of the image processing method provided by this application with reference to the accompanying drawings.

[0064] Figure 2 This is a schematic diagram of the structure of a computer device provided in this application. Figure 2 As shown, computer device 200 includes processor 210, memory 220, communication interface 230, storage 240 and bus 250. Processor 210, memory 220, communication interface 230 and storage 240 are connected via bus 250.

[0065] The processor 210 is the control center of the computer device 200. The processor 210 may be a central processing unit (CPU). The processor 210 includes one or more processor cores. For example, Figure 2 The processor 210 shown in FIG. 1 includes N processor cores.

[0066] In some embodiments, processor 210 includes registers and cache memory.

[0067] The cache memory is used to store instructions or data that may be accessed multiple times by the processor core in the processor, thereby increasing the speed at which the processor processes data and preventing the processor from frequently accessing the memory 220 .

[0068] In physical form, cache memory can be random access memory (RAM), static random-access memory (SRAM), dynamic RAM (DRAM), or other types of storage devices that can store information and instructions.

[0069] In terms of logical form, a cache memory can be a first-level cache (L1 cache), a second-level cache (L2 cache), a third-level cache (L3 cache), or a cache device at any level.

[0070] Registers are used to store instructions or data that may be accessed multiple times by the processor core of processor 210. Since register access speed is higher than cache memory access speed, instructions or data that may be accessed multiple times by the processor core can be stored in registers first, which can further improve the speed at which the processor processes data.

[0071] Alternatively, the processor 210 may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0072] Memory 220 (also referred to as main memory) can be a volatile memory pool or a non-volatile memory pool, or can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0073] The bus 250 may include a path for transmitting information between the above-mentioned components (such as the processor 210, the memory 220, the communication interface 230, and the storage 240). In addition to the data bus, the bus 250 may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus 250 in the figure. The bus 250 may be a Peripheral Component Interconnect Express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus 250 can be divided into an address bus, a data bus, a control bus, etc.

[0074] The memory 240 is used to store program instructions for executing the image processing method provided in this application.

[0075] The communication interface 230 is used to obtain the image to be processed and the distortion field of the image to be processed.

[0076] In this application, if Figure 2 As shown in (a) of FIG. , processor 210 is configured to perform pooling, reorganization, and image warping. For example, processor 210 is configured to obtain a to-be-processed image and a first warped field of the to-be-processed image from memory 220, pool the first warped field of the to-be-processed image to obtain a second warped field, and reorganize pixels in the to-be-processed image to obtain a reorganized image, such that the reorganized pixels in the reorganized image indicated by the elements in the second warped field have consecutive storage addresses for multiple pixels in the to-be-processed image. Processor 220 may store the second warped field and the reorganized image in memory 220. When performing image warping on the to-be-processed image, processor 220 is configured to obtain the second warped field from memory 220 and store the second warped field in a register or cache memory of processor 220. The processor 220 is configured to load the reconstructed pixels indicated by the elements in the second warp field from the memory 220 into a register or a cache memory of the processor 220, perform warping processing in parallel on a plurality of pixels in the image to be processed that are included in the reconstructed pixels corresponding to the elements in the second warp field according to a vector calculation instruction to obtain an intermediate processed image, and arrange the plurality of pixels included in the pixels in the intermediate processed image according to a reconstructing rule to obtain a warped image of the image to be processed.

[0077] In other embodiments, the computer device 200 may further include other dedicated processors. For example, the computer device 200 includes a processor 260, which may be a computing unit with computing capabilities such as a neural processing unit (NPU).

[0078] The image processing method provided in the present application may also be executed by the processor 260 , that is, the processor 260 is used to perform pooling, reorganization, and image warping processing.

[0079] In other embodiments, the pooling, reorganization, and image warping processes may be performed by different processors. Figure 2 As shown in (b) of FIG. 5 , the processor 220 is used to perform pooling and reorganization. The processor 260 is used to perform image warping processing.

[0080] It is worth mentioning that Figure 2 In the example, a computer device including two processors is used. In a specific embodiment, the number of devices or equipment can be determined according to business requirements. Figure 2The device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the computer device may also include a graphics processing unit (GPU) and a data processing unit (DPU).

[0081] Next, the image processing method provided by this application is described in detail with reference to the accompanying drawings.

[0082] For ease of explanation, we take a processor that performs pooling, reorganization, and image warping as an example. Figure 3 , Figure 3 This is a flow chart of an image processing method provided by this application. Here we first explain the pooling and reorganization included in the image preprocessing stage. Figure 3 As shown, the method includes the following steps.

[0083] Step 310: Pool the first warped field of the image to be processed to obtain a second warped field.

[0084] When an image is warped, adjacent pixels exhibit motion consistency. This means that when the image is warped, adjacent elements in the first warp field move in similar directions. Pooling these motion-consistent adjacent pixels reduces the number of elements in the warp field, improving computational performance.

[0085] In some embodiments, the first warp field is divided according to the pooling step size to obtain multiple pooling areas, and multiple elements contained in each of the multiple pooling areas are pooled to obtain the second warp field.

[0086] The larger the pooling step size, the more elements the pooling area contains, the greater the potential loss in the image warping process, and the more pixels in the image are used in a single image warping process based on vector calculation instructions. Conversely, the smaller the pooling step size, the fewer elements the pooling area contains, and the smaller the potential loss in the image warping process. Therefore, while ensuring that the image warping causes minimal loss to the image, pooling the first warping field of the image to be processed reduces the number of elements contained in the warping field, thereby improving the computational performance of the image warping process.

[0087] In some embodiments, the pooling step size is determined based on the data length indicated by the vector calculation instruction and the data length of the pixels in the image to be processed.

[0088] In other embodiments, the greater the image resolution and the more pixels the image contains, the larger the pooling step size. The pooling step size is determined based on the data length indicated by the vector calculation instruction, the data length of the pixels in the image to be processed, and the resolution of the image to be processed.

[0089] For example, the pooling step size can be 2, 4, or 8.

[0090] This application does not limit the shape of the pooling area. For example, the shape of the pooling area includes a square. For example, assuming that the pooling step size is S and the pooling area is an S×S area, the number of elements included in the second warp field becomes 1 / S of the number of elements included in the first warp field. 2 For another example, the shape of the pooling region includes a rectangle. Assuming that the pooling steps include S1 and S2, and S1 and S2 are not equal, the pooling region can be an area of ​​S1×S2.

[0091] For example, Figure 4 As shown in (a) of the figure, assume that the first warp field contains 64 elements, the pooling stride is 2, and the pooling area is a 2×2 area. The first pooling area contains 4 elements from the first warp field. A warp field can be divided into 16 pooling areas. After pooling the first warp field, the second warp field contains 16 elements. For example, after pooling area 1, the second warp field is obtained by element 1. After pooling area 2, the second warp field is obtained by element 2.

[0092] Pooling methods include but are not limited to maximum value, minimum value, element averaging, random point selection, or fixed point selection within the pooling area.

[0093] For example, Figure 4 As shown in (b) of the figure, the pooled region includes the coordinates of element 1 (x1, y1), element 2 (x2, y2), element 3 (x3, y3), and element 4 (x4, y4). Assuming that element 4 has the largest coordinates (x4, y4), element 4 is used as the pooled element. Alternatively, any one of the four elements can be used as the pooled element. Alternatively, the x and y coordinates of element 1, element 2, element 3, and element 4 can be averaged, and the average value is used as the pooled element.

[0094] Optionally, the first warped field is pooled using pooling regions of various sizes and shapes. That is, the shapes and sizes of the pooling regions used in pooling the first warped field can be the same or different. In addition, the pooling methods used in pooling the first warped field can also be different.

[0095] The elements in the first warping field include the coordinates of pixels in the image to be processed or the offset values ​​of the coordinates of pixels in the image to be processed.

[0096] In the first method, the elements in the first warp field contain the offset of the coordinates of the pixels in the image to be processed. The offset is divided by the pooling step size to convert it into the pooled offset, which is used as the element in the second warp field.

[0097] In the second method, the elements in the first warp field contain the coordinates of the pixels in the image to be processed, the coordinates are converted to offsets, and the offsets are divided by the pooling step size to convert to pooled coordinates as elements in the second warp field.

[0098] In a third approach, elements in the first warped field are matched to reconstructed pixels of the reconstructed image.

[0099] Step 320: recombining the pixels in the image to be processed to obtain a recombined image.

[0100] The image to be processed is divided into multiple reconstructed regions according to the pooling step size, each of which includes multiple pixels in the image to be processed. The multiple pixels contained in each of the multiple reconstructed regions are reconstructed according to the reconstructing rule to obtain a reconstructed image. Each reconstructed pixel in the reconstructed image contains multiple pixels in the image to be processed. Assuming that the pooling step size is S, the number of reconstructed pixels contained in the reconstructed image becomes 1 / S of the number of pixels contained in the image to be processed. 2 .

[0101] The image to be processed can be divided according to the pooling step size in the same manner as the first warp field, and the size of the resulting reconstructed region can be the same as the size of the pooling region. For example, assuming the first warp field contains 64 elements, the pooling step size is 2, and the pooling region is a 2×2 region. The first pooling region contains 4 elements from the first warp field. A warp field can be divided into 16 pooling regions, and the second warp field obtained after pooling the first warp field contains 16 elements. The image to be processed contains 64 pixels. By dividing the image to be processed according to the pooling step size, the image to be processed can be divided into 16 reconstructed regions, each reconstructed region contains 4 pixels, and the reconstructed image obtained after reconstructing the image to be processed contains 16 pixels.

[0102] It should be noted that, since the pixels in the processed image indicated by multiple elements in the pooling region may not belong to the same reconstructed region, the reconstructed pixel may include at least one pixel indicated by multiple elements in the pooling region. In other words, the reconstructed pixel may include all or part of the pixels indicated by multiple elements in the pooling region.

[0103] The reorganization rule is used to indicate the arrangement rule of multiple pixels in the image to be processed in the reorganized pixel. The reorganization rule includes but is not limited to (RGB, RGB...) or (RR...GG...BB...). This application does not limit the reorganization rule.

[0104] For example, Figure 5 As shown, assume that pooling region 1 contains four elements, each representing four pixels in the image to be processed. For example, the coordinates (x1, y1) of element 1 indicate pixel 1 (R1G1B1), the coordinates (x2, y2) of element 2 indicate pixel 2 (R2G2B2), the coordinates (x3, y3) of element 3 indicate pixel 3 (R3G3B3), and the coordinates (x4, y4) of element 4 indicate pixel 4 (R4G4B4). The reconstructed pixel rule is (pixel 1, pixel 2, pixel 3, pixel 4), and the reconstructed pixel consists of (R1G1B1, R2G2B2, R3G3B3, R4G4B4). The arrangement of the four pixels in the reconstructed pixel rule is (R1G1B1, R2G2B2, R3G3B3, R4G4B4). After pooling the four elements contained in pooling region 1, element 1 in the second warp field is obtained. Element 1 can be used to indicate the reconstructed pixel 1 in the reconstructed image.

[0105] In some embodiments, the dimensions and pixel order of the image to be processed are changed according to the pooling step size S, and the pixels in the image to be processed are reorganized to obtain a reorganized image, which includes a plurality of reorganized pixels. For example, the dimensions of the image to be processed include height (high) and width (wide). Assuming that the height of the image to be processed is H, the width of the image to be processed is W, and the number of channels of the pixel is C, the height of the reorganized image is H / S, the width of the reorganized pixel is W / S, and the number of channels of the reorganized pixel is C*S 2 .

[0106] It should be noted that the reconstructed pixels indicated by the elements of the second warp field contain multiple pixels with consecutive storage addresses. That is, after adjacent pixels in the processed image are reconstructed, the storage addresses of the multiple pixels within a reconstructed pixel are consecutive. The first warp field is used to indicate the offset rule for pixels in the processed image. The second warp field is used to indicate the offset rule for reconstructed pixels in the reconstructed image.

[0107] The storage addresses of the reconstructed pixels indicated by consecutive elements in the second warped field may or may not be consecutive.

[0108] After pooling the first warp field of the image to be processed and recombining the image to be processed, image warping can be performed based on the second warp field and the recombined image. Figure 6 , Figure 6 This is a flow chart of an image processing method provided by this application. Figure 6 As shown, the method includes the following steps.

[0109] Step 610: Perform warping processing in parallel on a plurality of pixels in the image to be processed obtained according to the elements in the second warping field according to the vector calculation instruction to obtain an intermediate processed image.

[0110] Step 1: Obtain a first reconstructed pixel indicated by a first element in the second warped field and a plurality of reconstructed pixels related to the interpolation calculation of the first reconstructed pixel. The first element is any element in the second warped field.

[0111] In one possible implementation, since step 420 has been pre-executed, a reconstructed image is obtained. A first reconstructed pixel indicated by a first element in the second warp field is obtained from the reconstructed image. For example, the first element may include the coordinates of the first reconstructed pixel in the reconstructed image, and the first reconstructed pixel is obtained based on the coordinates of the first reconstructed pixel. Furthermore, the multiple reconstructed pixels associated with the interpolation calculation of the first reconstructed pixel may include pixels adjacent to the first reconstructed pixel, for example, reconstructed pixels surrounding the first reconstructed pixel.

[0112] In another possible implementation, the processor may not execute step 420, and acquire reconstructed pixels in real time when performing image distortion processing. For example, the first reconstructed pixel is composed of the pixel in the image to be processed corresponding to the first element in the second distortion field and the multiple pixels in the image to be processed included in the pooling area. That is, the pixel in the image to be processed indicated by the element in the pooling area of ​​the first distortion field and the pixel in the image to be processed determined by the pooling area of ​​the first distortion field constitute the first reconstructed pixel. It is understandable that the first element in the second distortion field can be any element in the pooling area used to generate the first element. For example, the maximum value, minimum value, average value or a random element in the pooling area. After first acquiring the element in the pooling area, determine the pixel indicated by the element in the image to be processed, and form the first reconstructed pixel with multiple pixels within the pooling area centered on the pixel indicated by the element.

[0113] Step 2: Interpolation calculations are performed in parallel on the first reconstructed pixel and multiple reconstructed pixels associated with the interpolation calculation of the first reconstructed pixel according to the vector calculation instruction. The multiple reconstructed pixels associated with the interpolation calculation of the first reconstructed pixel include reconstructed pixels surrounding the first reconstructed pixel. The multiple pixels in the image to be processed included in the first reconstructed pixel can be a vector containing multiple pixels in the image to be processed. Interpolation calculations are performed in parallel on the multiple pixels in the image to be processed and the multiple pixels in the multiple reconstructed pixels. This application does not limit the specific method of interpolation calculation. Examples include bilinear interpolation, cubic interpolation, and interpolation.

[0114] It should be noted that if the amount of data to be calculated is less than or equal to the amount of data that can be processed by the vector calculation instruction, the vector calculation instruction can be executed once to complete the calculation. For example, a pixel in the image to be processed contains 3 channels (such as RGB), a pooling area contains 16 elements, and the number of channels of the reconstructed pixel indicated by the elements after pooling in the pooling area is 48 channels. If the vector calculation instruction can process 48 channels of data, the vector calculation instruction can be executed once to complete the calculation. If the vector calculation instruction can process 32 channels of data, the vector calculation instruction can be executed twice to complete the calculation.

[0115] Traverse the elements in the second warped field and repeat steps 1 and 2 to obtain an intermediate processed image, wherein the intermediate processed image includes the warping result of the recombined pixels after the pixels in the image to be processed are recombined according to the recombining rule.

[0116] Step 620: Arrange the data contained in the pixels of the intermediate processed image according to the reorganization rule to obtain a distorted image of the image to be processed.

[0117] The reconstructed pixels in the reconstructed image are distorted according to the second warping field, for example, by shifting the reconstructed pixels in the reconstructed image, to obtain an intermediate processed image. A pixel in the intermediate processed image contains multiple pixels, and the pixel in the intermediate processed image needs to be restored according to the reconstruction rule. Specifically, the storage addresses of the multiple pixels in the memory of the pixel in the intermediate processed image are rearranged according to the reconstruction rule.

[0118] The image processing method provided by the present application divides the distortion field into small pooling areas and performs fast downsampling based on the characteristics of the consistency of motion of adjacent pixels during image distortion motion. Therefore, the number of elements of the distortion field used for image distortion processing is reduced, thereby reducing the number of calls to vector instructions. By rearranging the memory of the pixels in the image to be processed, the storage addresses of multiple pixels in the reconstructed pixels corresponding to the elements in the pooled distortion field are made continuous, so that the storage addresses of multiple pixels required for a vector calculation are made continuous. When the image is distorted, the number of accesses to the storage medium outside the processor is reduced, thereby improving the computing performance of the image distortion processing through vectorized acceleration of the distortion processing process.

[0119] The image processing method provided in this application is illustrated below with examples.

[0120] Figure 7A schematic diagram of an image processing process provided in this application. Assume that the first warped field contains 64 elements, the pooling step is 2, the pooling area is a 2×2 area, and the second warped field obtained after pooling the first warped field contains 16 elements. The image to be processed contains 64 pixels, and the reconstructed image contains 16 reconstructed pixels. The fact that the second warped field contains 16 elements indicates that the reconstructed image contains 16 reconstructed pixels. For example, the second warped field containing 16 elements can include the coordinates of 16 reconstructed pixels.

[0121] like Figure 7 As shown, the reconstructed pixel in the 3rd row and 2nd column indicated by the element A in the second warped field and multiple reconstructed pixels related to the interpolation calculation of the reconstructed pixel perform interpolation calculation in parallel, and the reconstructed pixel after the interpolation calculation can be migrated to the 2nd row and 2nd column.

[0122] The intermediate processed image contains 16 reconstructed pixels after interpolation calculations. The data contained in the pixels in the intermediate processed image are arranged according to the reconstruction rules to obtain a distorted image of the image to be processed. The distorted image of the image to be processed contains 64 pixels.

[0123] Figure 8 This is a schematic diagram of an image processing process provided by this application. Figure 7 The difference is that the processor may not execute step 420 and obtain the reconstructed pixels in real time when performing the image warping process. Assume that element 1 in the second warping field corresponds to pixel 1 in the image to be processed, and pixels 2, 3, and 4 determined according to pixel 1 and the pooling area constitute the first reconstructed pixel. Collect S pixels at intervals of pooling step S in the image to be processed. 2 pixels as a plurality of reconstructed pixels associated with the interpolation calculation of the first reconstructed pixel.

[0124] It is understood that in order to implement the functions in the above embodiments, the device includes hardware structures and / or software modules that perform the corresponding functions. Those skilled in the art should easily appreciate that, in combination with the units and method steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in hardware or in a computer software-driven hardware manner depends on the specific application scenario and design constraints of the technical solution.

[0125] Combined with the above Figures 1 to 8 , describes in detail the image processing method provided by this application, and will be combined with Figure 9 , describes the image processing device provided by this application.

[0126] Figure 9This is a schematic diagram of the structure of a possible image processing device provided by this application. These image processing devices can be used to implement the functions of the processor in the above method embodiment, and thus can also achieve the beneficial effects possessed by the above method embodiment. In this embodiment, the image processing device can be as follows Figure 2 Processor shown.

[0127] like Figure 9 As shown, the image processing device 900 includes a communication module 910, a distortion processing module 920 and a storage module 930. The image processing device 900 is used to implement the above Figure 4 and Figure 6 The functions of the processor in the method embodiments are shown.

[0128] The communication module 910 is used to obtain the image to be processed and the first distortion field of the image to be processed. For example, the communication module 910 is used to perform Figure 5 Step 510.

[0129] The distortion processing module 920 is configured to divide the first distortion field according to the pooling step size to obtain a plurality of pooling regions, each of which includes a plurality of elements in the first distortion field; and to pool the plurality of elements contained in each of the plurality of pooling regions to obtain a second distortion field. For example, the distortion processing module 920 is configured to perform Figure 4 Step 410.

[0130] The distortion processing module 920 is used to divide the image to be processed according to the pooling step size to obtain multiple reconstructed regions, each of which includes multiple pixels in the image to be processed; and reconstruct the multiple pixels contained in each of the multiple reconstructed regions according to the reconstructing rule to obtain a reconstructed image. For example, the distortion processing module 920 is used to perform Figure 4 Step 420.

[0131] The distortion processing module 920 is used to perform distortion processing on multiple pixels in the image to be processed obtained according to the elements in the second distortion field in parallel according to the vector calculation instruction to obtain an intermediate processed image. For example, the distortion processing module 920 is used to perform Figure 6 Step 610.

[0132] The distortion processing module 920 is used to arrange the pixels contained in the intermediate processing image according to the reorganization rule to obtain the distorted image of the image to be processed. Figure 6 Step 620.

[0133] The storage module 930 is used to store the warping field, image, etc., so as to facilitate image warping processing.

[0134] It should be understood that the image processing device 900 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. It can also be implemented by software. Figure 4 or Figure 6 The image processing method shown in the figure and its respective modules may also be software modules, and the image processing apparatus 900 and its respective modules may also be software modules.

[0135] The image processing apparatus 900 according to the embodiment of the present application may correspond to executing the method described in the embodiment of the present application, and the above and other operations and / or functions of each unit in the image processing apparatus 900 are respectively to realize Figure 4 or Figure 6 For the sake of brevity, the corresponding processes of each method in are not repeated here.

[0136] The present application also provides a processor, which includes a logic circuit and a power supply circuit; wherein the power supply circuit is used to supply power to the logic circuit; and the logic circuit is used to execute the operating steps of the image processing method described in the above method embodiment.

[0137] The method steps in this embodiment can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. In addition, the ASIC can be located in a computing device. Of course, the processor and storage medium can also exist as discrete components in a computing device.

[0138] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the process or function described in the embodiments of the present application is performed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device or other programmable device. The computer program or instruction can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired or wireless means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, a hard disk, or a tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (SSD). The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: Performing a warping process in parallel on a plurality of pixels in the image to be processed obtained based on elements in the second warping field according to the vector calculation instruction to obtain an intermediate processed image, wherein the second warping field includes elements obtained by pooling the first warping field of the image to be processed, and the intermediate processed image includes a warping process result of recombined pixels after recombining the pixels in the image to be processed according to a recombination rule, the recombined pixels including the plurality of pixels in the image to be processed, and the recombination rule is used to indicate an arrangement rule of the plurality of pixels in the image to be processed constituting the recombined pixels; Arrange the pixels of the intermediate processed image according to the reorganization rule to obtain a distorted image of the image to be processed.

2. The method according to claim 1, characterized in that The reconstructed pixels indicated by the elements in the second warping field include a plurality of pixels in the image to be processed whose storage addresses are continuous.

3. The method according to claim 1 or 2, characterized in that Performing warping processing in parallel on a plurality of pixels in the image to be processed obtained according to the elements in the second warping field according to the vector calculation instruction, including: Step 1: obtaining a first reconstructed pixel indicated by a first element in the second warped field and a plurality of reconstructed pixels related to an interpolation calculation of the first reconstructed pixel, wherein the first element is any element in the second warped field; Step 2: performing interpolation calculations on the first reconstructed pixel and a plurality of reconstructed pixels related to the interpolation calculation of the first reconstructed pixel in parallel according to the vector calculation instruction; Traverse the elements in the second warp field and repeatedly perform steps 1 and 2 to obtain the intermediate processed image.

4. The method according to claim 3, characterized in that Obtaining a first reconstructed pixel indicated by a first element in the second warped field includes: A first reconstructed pixel indicated by a first element in the second warp field is obtained from the reconstructed image, where the second warp field is used to indicate an offset rule of the reconstructed pixels in the reconstructed image.

5. The method according to claim 3, characterized in that Acquiring a first reconstructed pixel indicated by a first element in the second warped field and a plurality of reconstructed pixels related to an interpolation calculation of the first reconstructed pixel, comprising: A plurality of pixels in the image to be processed determined according to the pixel in the image to be processed corresponding to the first element in the second warping field and the pooling area constitute the first reconstructed pixel; A plurality of reconstructed pixels related to an interpolation calculation of the first reconstructed pixel are determined according to the first reconstructed pixel and a pooling step size.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining an intermediate processed image by performing warping processing in parallel on a plurality of pixels in the to-be-processed image obtained according to the elements in the second warping field according to the vector calculation instruction, the method further includes: Dividing the first warped field according to a pooling step size to obtain a plurality of pooled regions, each of the pooled regions including a plurality of elements in the first warped field; Pooling is performed on a plurality of elements contained in each of the plurality of pooling areas to obtain the second warped field.

7. The method according to claim 6, characterized in that Before obtaining an intermediate processed image by performing warping processing in parallel on a plurality of pixels in the to-be-processed image obtained according to the elements in the second warping field according to the vector calculation instruction, the method further includes: Dividing the image to be processed according to the pooling step size to obtain a plurality of recombined regions, each of the recombined regions including a plurality of pixels in the image to be processed; A plurality of pixels contained in each of the plurality of recombined regions are recombined according to the recombined rule to obtain a recombined image.

8. The method according to any one of claims 5 to 7, characterized in that The method further comprises: The pooling step size is determined according to the data length indicated by the vector calculation instruction and the data length of the pixels in the image to be processed.

9. The method according to any one of claims 5 to 7, characterized in that The method further comprises: The pooling step size is determined according to the data length indicated by the vector calculation instruction, the data length of the pixels in the image to be processed, and the resolution of the image to be processed.

10. An image processing device, characterized in that: include: A communication module, configured to obtain an image to be processed and a second warped field obtained by pooling the first warped field of the image to be processed; a warping processing module, configured to perform warping processing in parallel on a plurality of pixels in the image to be processed obtained based on elements in the second warping field according to a vector calculation instruction, to obtain an intermediate processed image, wherein the second warping field includes elements obtained by pooling the first warping field of the image to be processed, and the intermediate processed image includes a warping processing result of recombined pixels after recombining the pixels in the image to be processed according to a recombination rule, the recombined pixels including the plurality of pixels in the image to be processed, and the recombination rule is used to indicate an arrangement rule of the plurality of pixels in the image to be processed constituting the recombined pixels; The distortion processing module is further configured to arrange the pixels of the plurality of pixels included in the pixels of the intermediate processed image according to the reorganization rule to obtain a distorted image of the image to be processed.

11. A processor, characterized in that: The processor includes a logic circuit and a power supply circuit; wherein the power supply circuit is used to supply power to the logic circuit; and the logic circuit is used to execute the operation steps of the method according to any one of claims 1 to 9.

12. The processor according to claim 11, wherein: The processor includes at least one of a central processing unit or a neural processing unit.

13. A computer device, characterized in that: The computer device includes at least one processor and a memory, wherein the memory is used to store a set of computer instructions; when the processor executes the set of computer instructions, the operation steps of any one of the methods described in claims 1-9 are performed.

Images