Distortion correction device and method

Abstract

A distortion correction method integrated with image operations is provided. The method includes modifying a warpmap by one or more image operations. These image operations include cropping, geometric transformation, stitching, or a combination thereof. The method further includes converting an input image to an output image using the modified deformation map.

Description

TECHNICAL FIELD

[0001] The present invention relates to image processing, and in particular to an apparatus and method for distortion correction. BACKGROUND

[0002] Distortion is a phenomenon that arises due to imperfections in a photographic lens. The optical elements in the lens can not perfectly refract light rays, resulting in a bending of the light rays and a subsequent distortion of the scene acquired, which can manifest as barrel distortion and pincushion distortion. Barrel distortion causes straight lines to appear to curve outward, while pincushion distortion causes straight lines to appear to curve inward. These distortions are particularly evident at the edges of the image, and can be more pronounced in wide-angle lenses.

[0003] Distortion can be corrected by warping the image using inverse distortion. This process involves determining a correspondence between each distorted pixel in the uncorrected image and its undistorted counterpart in the corrected image. Due to the non-linearity of the distortion, it is impractical to use a simple function or formula to map the pixels of the uncorrected and corrected images. Therefore, a one-to-one mapping table is typically used to record the mapping relationship between each distorted pixel in the uncorrected image and its corresponding undistorted pixel, in order to convert the uncorrected image to the corrected image. This mapping table is commonly referred to as a "warp map".

[0004] Figure 1 illustrates a conventional image processing pipeline P10. As shown in Figure 1, an image acquired by a photographic device 11 undergoes a series of steps S12-S17 before being output for display or storage. In step S12, the image acquired by the photographic device 11 undergoes image signal processing, including for example demosaicing, noise reduction, image sharpening, and gamma correction. Subsequently, in step S13, the aforementioned warp map is applied for distortion correction. After distortion correction, various image operations are applied to the corrected image in steps S14-S17, collectively referred to as image operations 150. In the example shown in Figure 1, these image operations 150 include cropping in step S14, stitching in step S17, and further include geometric transformations 160 of rotation in step S15 and scaling in step S16. It should be understood that the description of steps S14-S17 in Figure 1 is illustrative, and in practice the image operations can involve more or fewer operations than shown. The geometric transformations 160 can further include operations of clipping, reflection, translation, orthogonal projection, etc. Furthermore, the order of execution of these image operations 150 can vary, although cropping is typically performed first and stitching is typically performed last.

[0005] As shown in FIG. 1, the image processing pipeline P10 involves many steps, especially in the case of the image operation 150. Moreover, each step in FIG. 1 inevitably introduces some latency. Although the image operation 150 is usually implemented using efficient graphics processing units (GPUs) or specialized circuits, the cumulative latency from the numerous steps can still become quite large. For example, assuming a requirement of processing 60 frames per second, this implies a latency of 1 / 60 second per step, and with the distortion correction together with the image operation 150 spanning five steps S13-S17, this results in a total latency of 5 / 60 seconds. This cumulative latency can impact the user experience for certain applications that have high performance requirements, such as image streaming, autonomous driving, or gaming.

[0006] In view of the above challenges, a distortion correction solution is proposed that can reduce system load and latency by handling distortion correction and various image operations comprehensively in a single step. SUMMARY

[0007] One embodiment of the present disclosure provides a distortion correction method for a distortion correction apparatus. The method includes a first operation and a second operation. The first operation involves modifying a warped map by one or more image operations. The image operations include cropping, geometric transformation, stitching, or a combination thereof. The second operation involves converting an input image to an output image using the modified warped map.

[0008] In one embodiment, the geometric transformation is rotation, scaling, or a combination thereof.

[0009] In one embodiment, the input image is not processed by any of the one or more image operations before the second operation.

[0010] In one embodiment, the first operation pre-determines the modified warped map, and the second operation further involves converting each frame of an input image sequence to an output image frame using the modified warped map pre-determined by the first operation.

[0011] In one embodiment, the first operation further involves cropping the warped map to obtain a plurality of cropped portions, geometrically transforming the plurality of cropped portions to obtain a plurality of transformed portions, and stitching the plurality of transformed portions into the modified warped map.

[0012] In one embodiment, the first operation further involves performing object detection on the input image to obtain one or more bounding boxes, cropping the warped map according to the one or more bounding boxes to obtain the plurality of cropped portions, and geometrically transforming the plurality of cropped portions according to the one or more bounding boxes to obtain the plurality of transformed portions.

[0013] In one embodiment, the first operation is modifying a warp map for a second frame of the sequence of input images after the first frame is warped by the second operation.

[0014] In one embodiment, the second operation further involves up-sampling the modified warp map to match a specified resolution of the output image before warping the input image.

[0015] Embodiments of the present application also provide a distortion correction device. The device includes an integrated circuit. The integrated circuit is configured to warp an input image into an output image using a modified warp map. The modified warp map is predetermined by modifying a warp map using one or more image operations. The image operations include cropping, geometric transformation, stitching, or a combination thereof.

[0016] In one embodiment, the integrated circuit does not perform any of the one or more image operations on the input image sequence.

[0017] In one embodiment, the integrated circuit is further configured to up-sample the modified warp map to match a specified resolution of the output image before warping the input image.

[0018] Embodiments of the present application also provide another distortion correction device. The device includes a storage unit, a processing unit, and a distortion correction unit. The storage unit is configured to store at least an input image and a warp map. The processing unit is coupled to the storage unit, configured to read the warp map from the storage unit, modify the warp map using one or more image operations to generate a modified warp map, and store the modified warp map into the storage unit. The distortion correction unit is coupled to the storage unit and the processing unit, configured to receive a control signal from the processing unit, read the input image and the modified warp map from the storage unit according to the control signal, and warp the input image into an output image using the modified warp map. The image operations include cropping, geometric transformation, stitching, or a combination thereof.

[0019] In one embodiment, the processing unit does not perform any image operations on the input image.

[0020] In one embodiment, the processing unit predetermines the modified warp map by modifying the warp map using one or more image operations, and the distortion correction unit warps a sequence of input images into a sequence of output images using the predetermined modified warp map.

[0021] In one embodiment, the processing unit predetermines the modified warp map by cropping the warp map to obtain a plurality of cropped portions from the warp map, geometrically transforming the plurality of cropped portions to obtain a plurality of transformed portions, and stitching the plurality of transformed portions into the modified warp map.

[0022] In one embodiment, the processing unit further performs object detection on the input image to obtain one or more bounding boxes, crops the warped map according to the one or more bounding boxes to obtain a plurality of cropped parts, and geometrically transforms the plurality of cropped parts according to the bounding boxes to obtain the plurality of transformed parts.

[0023] In one embodiment, the distortion correction unit is modifying the warped map for transforming a second frame of the input image sequence when transforming a first frame of the input image sequence.

[0024] In one embodiment, the distortion correction unit further upsamples the modified warped map to match a specified resolution of the output image before transforming the input image. BRIEF DESCRIPTION OF DRAWINGS

[0025] FIG. 1 depicts a conventional image processing flow;

[0026] Figure 2 A novel image processing flow according to an embodiment of the present disclosure is described;

[0027] Figure 3 is a schematic diagram of a distortion correction method that integrates image operations according to an embodiment of the present disclosure; Figure 4A is an example implementation flow diagram of modifying a warped map according to an embodiment of the present disclosure;

[0028] Figure 4B Example modified warped maps that are processed by a first operation according to an embodiment of the present disclosure are described;

[0029] Figure 5 is a block diagram of a distortion correction apparatus that integrates image operations according to an embodiment of the present disclosure;

[0030] Figure 6 is a block diagram of a distortion correction apparatus that integrates image operations according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of the application is intended to explain the general principles of the application and is not to be limited in scope by the specific embodiments described. The application is best understood from the attached drawings and description.

[0032] In each of the embodiments below, like reference numerals refer to like or similar elements or components.

[0033] It must be understood that the terms "comprise" and "include" in the specification are used to indicate the presence of a specific technical feature, value, method step, process operation, element, and / or component, but not to exclude additional technical features, values, method steps, process operations, elements, components, or any combination of the above.

[0034] The ordinal numbers used in the claims, such as "first", "second", "third", etc., are used only for the convenience of explanation and do not imply any priority among them.

[0035] As mentioned before, the warp map is a one-to-one mapping table recording the mapping relationship between each distorted pixel in the uncorrected image and the corresponding non-distorted pixel in the corrected image. More specifically, each pixel of the warp map indicates which location in the uncorrected image to retrieve pixel value from for the corrected / warped image. Therefore, any operation performed on the warp map, such as cropping, geometric transformation and / or stitching, will eventually be reflected in the warped image. This basic principle is exploited in the embodiments of the present disclosure.

[0036] Figure 2 A novel image processing flow P20 according to an embodiment of the present disclosure is shown. As Figure 2 indicated, the image acquired by the photographic device 21 only goes through two steps S22 and S23 before being output for display or storage. In step S22, similar to step S12 in Fig. 1, the acquired image goes through image signal processing. The main difference between the image processing flow P20 and P10 is in step S23, the distortion correction and image operation are combined. Since step 23 has already generated the output image required for display or storage, and there is no need for an image operation step similar to 150 in Fig. 1, for example, there is no need for steps S14-S17 shown in Fig. 1.

[0037] Compared with step S13 in Fig. 1, step S23 of the present invention actually has the advantage of no additional time delay. This is because step S23 and step S13, although both use the warp map to convert the input image to the output image for the distortion correction operation, the difference is that the modified warp map after image operation is used for distortion correction in step S23. That is, Figure 2Although step 23 also involves image manipulation, the image manipulation is performed on a warp map, rather than on the input image as in the embodiment of FIG. 1. That is, the input image does not have to go through image manipulation as in 150 of FIG. 1. Since the warp map is a mapping table, it records the mapping relationship of a portion of the pixels in the input image, rather than the actual pixel coordinates, and contains horizontal and vertical mapping components. Since in some embodiments, the warp map does not record the mapping relationship of all the pixels in the input image, the time and resources consumed for image manipulation on the warp map is much less than that for image manipulation on the input image. The distortion correction and image manipulation involved in step 23 can be performed in parallel, that is, while the modified warp map for the next frame in the input image sequence is being computed, the modified warp map obtained previously can be used to transform the current frame in the input image sequence into the current frame in the output image sequence, thus reducing the latency and improving the overall processing speed.

[0038] Again assuming the requirement of processing 60 frames per second, which means a latency of 1 / 60 second per step, the distortion correction and image manipulation are performed in a single step S23, resulting in a latency of only 1 / 60 second. This is equivalent to 20% of the latency produced by the image processing flow P10.

[0039] Figure 3 FIG. 30 is a schematic diagram of a distortion correction method 30 that integrates image manipulation, according to one embodiment of the present disclosure. As shown in FIG. 30, the method M30 includes a first operation O31 and a second operation O32. Figure 3 The first operation O31 is the same as the first operation O11 of the method M10 of FIG. 10, and the second operation O32 is the same as the second operation O12 of the method M10 of FIG. 10.

[0040] The first operation O31 involves modifying the warped map 301 by one or more image operations to obtain a modified warped map 302. As mentioned previously, the warped map 301 is used for distortion correction, and any operation performed on the warped map 301 in addition to the effect of distortion correction will also be reflected in the output image 304. If various image operations are to be performed on the input image 303 or the rectified image (i.e. the result of applying the warped map 301 to the input image 303 for distortion correction), it would be time and resource consuming. In the present solution, the various image operations are instead performed on the warped map. The generated output image 304 will still exhibit the effects of distortion correction as well as the image operations. Furthermore, it is worth noting that the computational effort required to perform image operations on the warped map 301 is generally less than the computational effort required to perform image operations on the input image 303. This is because the input image 303 generally contains multiple channels. For example, an input image 303 defined in the RGB color space has three channels - R (red), G (green) and B (blue). Direct image operations on the input image in the prior art require computation for each of these color channels. In contrast, the warped map 301 only requires one channel to record the mapping relationship, and thus the computational effort for performing image operations on it is less.

[0041] In the present embodiment, the image operations include cropping, geometric transformation, stitching or any combination thereof. To elaborate further, cropping involves removing unwanted areas and retaining only the desired parts. Geometric transformation involves changing the spatial arrangement or shape of an image by operations such as rotation, scaling, shearing, translation, reflection, orthogonal projection or any combination thereof. Stitching involves combining multiple images or image segments along common edges to create a new composite image.

[0042] The second operation O32 involves converting the input image 303 to the output image 304 using the modified warped map 302. Referring to the image processing flow P20 shown in Figure 2 , the input image 303 is an image that has not undergone distortion correction and image operations, but can have already undergone image signal processing in step S22, such as demosaicing, noise reduction, image sharpening and gamma correction, although this is not essential. The output image 304 is an image that has undergone distortion correction and image operations in step S23, making it ready for display and storage.

[0043] It should be understood that since the image operations originally intended to be performed on the input image 303 or the rectified image have already been performed on the warped map 301 by the first operation, the input image 303 only needs to be transformed using the modified warped map 302 that incorporates the image operations, without having to perform these image operations again. Therefore, in one embodiment, the input image 303 is not subjected to any of the above image operation processes prior to the second operation O32.

[0044] In an embodiment, the resolution of the warped image 301 and the modified warped image 302 can be smaller than the specified resolution of the output image 304. For example, while the output image 304 can have a high definition resolution of 1920*1080, the resolution of the warped image 301 and the modified warped image 302 can only be 100*100, which cannot match the specified resolution of the output image 304. Therefore, the second operation O32 can involve up-sampling the modified warped image to match the resolution of the output image before transforming the input image 303. Such up-sampling process can be implemented using methods such as bilinear interpolation, bicubic interpolation, or deep learning techniques, but the present disclosure is not limited thereto.

[0045] In an embodiment, the geometric transformation is rotation, scaling, or a combination thereof. This results in three cases: distortion correction integrating only rotation, distortion correction integrating only scaling, and distortion correction integrating both rotation and scaling.

[0046] Taking the geometric transformation in the image operation as rotation as an example, the specific calculation method of the first operation O31 is given below. For a rotation with an angle of θ in the counterclockwise direction (positive direction), its functional form is:

[0047]

[0048] where x and y are the mapping components of the warped image 301 corresponding to the pixel points of the input image in the horizontal and vertical directions, respectively, and (x, y) is the coordinate of the original point to be rotated, while x' and y' are the mapping components of the warped image 301 after rotation in the horizontal and vertical directions, respectively. In matrix form, it becomes:

[0049]

[0050] which is called a rotation matrix.

[0051] Taking the geometric transformation in the image operation as scaling as an example, the specific calculation method of the first operation O31 is given below. For scaling h times along the x-axis and k times along the y-axis, assuming that h and k are independent, its functional form is:

[0052]

[0053] where x and y are the mapping components of the warped image 301 corresponding to the pixel points of the input image in the horizontal and vertical directions, respectively (or the mapping components of the warped image 301 after the above rotation operation in the horizontal and vertical directions), and (x, y) is the coordinate of the original point to be scaled, while x' and y' are the mapping components of the warped image 301 after scaling in the horizontal and vertical directions, respectively. In matrix form, it becomes:

[0054]

[0055] wherein is called a scaling matrix.

[0056] In the case where the geometric transformation is a combination of rotation and scaling, indicating that the distortion correction integrates rotation and scaling, the mapping components of the warped image 301 in horizontal and vertical directions after the geometric transformation can be calculated by matrix multiplication of a rotation matrix and a scaling matrix However, it should be noted that matrix multiplication does not necessarily follow the commutative law, meaning that the order of execution of rotation and scaling is important. Detailed mathematical formulas are not provided here.

[0057] In one embodiment, the modified warped image 302 is predetermined by the first operation O31. Thus, the second operation O32 can use the modified warped image predetermined by the first operation O31 to convert each frame of the input image sequence into an output image frame. In further detail, in practical application scenarios of the image processing pipeline, such as the image processing pipeline P20 shown in Figure 2 , there is no need to repeatedly perform the operation O31 to obtain the modified warped image 302. This is because the image operation parameters required for each frame of the image sequence are the same, allowing the modified warped image 302 to be applied to each frame of the input image sequence. This embodiment is particularly suitable for applications where the shooting scene is relatively static, the transformation requirements of the input image are fixed and do not change with the environment, such as in a monitoring system. For example, a store, a bank or a school can deploy cameras at specific locations, each camera monitoring a specific area at a specific angle. Therefore, the parameters of the image operation, such as cropping, geometric transformation and stitching, can be fixed to ensure that multiple areas presented on the screen of the monitoring center are clearly visible at the same time. By pre-applying the image operation to the warped image 301, the image processing pipeline is free from time-consuming image operations, thereby improving overall efficiency.

[0058] In one embodiment, each frame of the input image sequence corresponds to a variable warped image. In addition, the first operation O31 and the second operation O32 are executed in parallel. Specifically, when the second operation O32 is converting the current frame of the input image sequence, the first operation O31 is modifying the warped image for converting the subsequent frame. This embodiment is particularly suitable for real-time applications where the required image operation changes with the environment, such as in a video conference. For example, the modified warped image 302 can change as the number and / or location of participants change.

[0059] Figure 4A is the execution flowchart of the first operation O31 according to an embodiment of the present disclosure, in which cropping, geometric transformation and stitching are executed in sequence. As Figure 4A shown, the first operation O31 can include steps ST41-43. Accordingly,Figure 4B An exemplary modified warping map RW40 is shown, which is obtained by processing the warping map 40 by the first operation O31 according to this embodiment. Please refer to Figure 4A and Figure 4B for a better understanding of this embodiment.

[0060] In step ST41, the warping map W40 is cropped to obtain a plurality of cropped portions from the warping map. The cropped portions are desired portions that are retained after unwanted areas are removed. As shown in the example provided in Figure 4B , the warping map 40 corresponds to the input image I40. In step ST41, nine cropped portions are obtained from the warping map 40, including a first cropped portion C41, a second cropped portion C42, a third cropped portion C43, and six other cropped portions. In addition to their different positions, the sizes of these cropped portions can also be different, including differences in length and / or width.

[0061] In step ST42, the cropped portions are geometrically transformed to obtain a plurality of transformed portions. As shown in the example provided in Figure 4B , the first cropped portion C41, the second cropped portion C42, and the third cropped portion C43 are respectively transformed into a first transformed portion T41, a second transformed portion T42, and a third transformed portion T43. The first transformed portion T41 is rotated 90 degrees counterclockwise with respect to the first cropped portion C41. The second transformed portion T42 is rotated 270 degrees counterclockwise and scaled down in size with respect to the second cropped portion C42. The third transformed portion T43 is rotated 180 degrees counterclockwise and scaled up in size with respect to the third cropped portion C43. Similarly, the other six cropped portions are transformed into their respective transformed portions.

[0062] In step ST43, the transformed portions are stitched into a modified warping map RW40. As shown in the example illustrated in Figure 4B , the resulting modified warping map RW40 is composed of the first transformed portion T41, the second transformed portion T42, the third transformed portion T43, and the other six transformed portions. It is worth noting that although the modified warping map RW40 is depicted as a nine-square grid in Figure 4B , with each square representing a transformed portion such as the first transformed portion T41, the second transformed portion T42, and the third transformed portion T43, which are squares of the same size, this depiction is merely an example and not a limitation. In various implementations, the modified warping map RW40 is not necessarily square, and the transformed portions are not necessarily of the same size. The main goal is to ensure that the input image 303 is mapped to the output image 304 according to the specifications of the display output.

[0063] Subsequently, the modified warping map RW40 can be used by a second operation O32 to transform and map the regions on the input image I40 corresponding to the nine cropped portions to the output image. For example, the output image consists of a nine-square grid composed of nine square image regions stitched together, like the modified warping map RW40. The region on the input image I40 corresponding to the first cropped portion C41 undergoes the same operations as the step ST42 performed on the first cropped portion C41, i.e., a 90-degree counter-clockwise rotation and adjustment to the proper size, and then is mapped to the top-left region of the output image. Similarly, the region on the input image I40 corresponding to the second cropped portion C42 undergoes the same operations as the step ST42 performed on the second cropped portion C42, i.e., a 270-degree counter-clockwise rotation and adjustment to the proper size, and then is mapped to the bottom-left region of the output image. The region on the input image I40 corresponding to the third cropped portion C43 undergoes the same operations as the step ST42 performed on the third cropped portion C43, i.e., a 180-degree counter-clockwise rotation and adjustment to the proper size, and then is mapped to the middle-bottom region of the output image. The other six regions follow the same procedure. It is worth mentioning again that the present disclosure does not require all cropped portions to be adjusted to the same size. The main goal is to ensure that the resized portions can be seamlessly stitched together to meet the display output specification.

[0064] In one embodiment, the first operation O31 further comprises performing object detection on the input image to obtain one or more bounding boxes. Furthermore, the cropping of the warped image in step ST41 and the geometric transformation performed on the cropped portion in step ST42 are both based on the bounding boxes. In further detail, a bounding box is a rectangle that encloses the region of the detected object (e.g. face) in the input image 303. This region can be defined by the coordinates of the top-left corner and the bottom-right corner of the rectangle. The warped image is then cropped to retain only the pixels within the bounding box, ensuring that the subsequent distortion correction and image manipulation specifically focus on the detected face region. Furthermore, each detected object in the cropped portion can be resized according to the size of the bounding box. For example, assume that the ideal size of the object to be displayed is M x N pixels. In the case where the size of the bounding box is M / 2 x N / 3 pixels, the scaling factors h and k will be set to 2 and 3 respectively to ensure that the image region of the detected object is resized to the appropriate size. This targeted approach improves the efficiency and accuracy of the face feature distortion correction process. Furthermore, since the modified warped image is composed of multiple resized transformed portions, it ensures that the detected face in the input image 303 appears to be appropriately sized after transformation. This is particularly beneficial for applications such as video conferencing and surveillance. For example, in video conferencing, when multiple participants are in the same camera frame, despite their varying distances from the camera, this approach ensures that no participant's head appears too small on the screen, reinforcing each participant's presence and engagement in the meeting. However, the head size of each participant does not need to be uniformly displayed. For example, the head of the meeting host or the person speaking can be enlarged and placed in the center of the screen. In surveillance, for example, by using camera equipment to monitor the concentration of students in a classroom simultaneously, this approach enables fair monitoring of students at different seating positions.

[0065] Various embodiments of the distortion correction method M30 are applied in distortion correction apparatuses, which can be either specially designed hardware devices or general-purpose computer systems driven by software programs. The structures of these distortion correction apparatus aspects will be described with reference to Figure 5 and Figure 6 .

[0066] Figure 5is a block diagram of a distortion correction apparatus 50 that integrates the image operations according to embodiments of the present disclosure. The distortion correction apparatus 50 includes an integrated circuit 501, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or a system on chip (SoC), configured to perform various embodiments of the second operation O32 described above. In addition, the distortion correction apparatus 50 can also include a non-volatile memory 502 for storing the modified warping map 302. The non-volatile memory 502 can include a read only memory (ROM), an electrically erasable programmable read only memory (EEPROM), a flash memory, a non-volatile random access memory (NVRAM), or any combination thereof.

[0067] Figure 6is a block diagram of a distortion correction device 60 that integrates image operations according to another embodiment of the present disclosure. In this embodiment, the distortion correction device 60 is a computer system capable of performing computational tasks, such as a personal computer (including desktop computers, laptops, tablets, etc.), or a server computer. The distortion correction device 60 includes a processing unit 601, which can include one or more general-purpose processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, or a microcontroller. The processing unit 601 can also include random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), and / or combinations thereof, although the present disclosure is not limited thereto. The distortion correction device 60 also includes a storage unit 602, which can be any storage device having non-volatile memory, such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or non-volatile random access memory (NVRAM). Examples of these devices include a hard disk drive (HDD) array, a solid-state drive (SSD), or an optical disc, although the present disclosure is not limited thereto. The processing unit 601 can communicate with the storage unit 602 through various wired or wireless communication interfaces, such as a system bus, a serial advanced technology attachment (SATA), a serial attached SCSI (SAS), NVM Express (NVMe), PCI Express (PCIe), a universal serial bus (USB) interface, a USB Type-C interface, a Thunderbolt interface, a fifth generation (5G) wireless system, Wi-Fi, and / or any combinations thereof, although the present disclosure is not limited thereto. According to embodiments of the present disclosure, the storage unit 602 is coupled to the processing unit 601 for storing at least an input image and a warp map. The processing unit 601 is configured to read the warp map from the storage unit 601 and perform various embodiments of the first operation O31. Specifically, the processing unit 601 modifies the warp map using one or more image operations to obtain a modified warp map and stores it into the storage unit 601. The distortion correction device 60 also includes a distortion correction unit 603. The distortion correction unit 603 is coupled to the processing unit 601 and the storage unit 602. The distortion correction unit 603 receives a control signal sent from the processing unit 601, reads the input image and the modified warp map from the storage unit 601 according to the received control signal, and performs various embodiments of the second operation O32. Specifically, the distortion correction unit 603 converts the input image into an output image using the modified warp map and stores the output image into the storage unit 601 for subsequent display by a display device, or directly displays the output image.

[0068] The above distortion correction unit 603 can be implemented using a central processing unit (CPU), a graphics processing unit (GPU), or the like, or can be implemented using a dedicated hardware unit. The distortion correction device 60 can be implemented in a manner in which a general-purpose CPU or GPU is used to implement the processing unit and a dedicated hardware unit is used to implement the distortion correction unit. The CPU and the GPU have strong computing power and can handle real-time dynamic situations, and thus various embodiments in which the processing unit is implemented using a CPU or a GPU to perform the first operation O31 can achieve the purpose of updating the warping map in real time according to requirements. When various embodiments in which the distortion correction unit is implemented using a dedicated hardware unit to perform the second operation O32, the dedicated hardware unit is more efficient than a general-purpose CPU or GPU, and can reduce the burden on the CPU or the GPU, and can improve the performance and response speed of the overall system. Since the distortion correction unit is implemented using a dedicated hardware unit, its function and the required operation of the distortion correction task are single, which makes it more stable and reliable and reduces the probability of errors. In summary, the use of a dedicated distortion correction unit in combination with the use of a general-purpose CPU or GPU can improve the processing efficiency of specific tasks and the overall performance of the system while maintaining the flexibility and strong computing power of the system.

[0069] In one embodiment, the one or more image operations include cropping, geometric transformation, stitching, or a combination thereof. The geometric transformation is rotation, scaling, or a combination thereof.

[0070] In one embodiment, the processing unit 601 does not perform any of the one or more image operations on the input image.

[0071] In one embodiment, the processing unit 601 pre-determines a modified warping map by modifying the warping map using the one or more image operations, and the distortion correction unit 603 uses the pre-determined modified warping map to convert the input image sequence into the output image sequence.

[0072] In one embodiment, the processing unit 601 obtains a plurality of cropped portions from the warping map by cropping the warping map, obtains a plurality of transformed portions by performing the geometric transformation on the cropped portions, and stitches the plurality of transformed portions into the modified warping map.

[0073] In one embodiment, the processing unit 601 further performs object detection on the input image to obtain one or more bounding boxes, crops the warping map according to the one or more bounding boxes to obtain a plurality of cropped portions, and performs geometric transformation on the plurality of cropped portions according to the one or more bounding boxes to obtain the plurality of transformed portions.

[0074] In one embodiment, the distortion correction unit 603 is modifying the warping map for converting a second frame of the input image sequence after converting a first frame of the input image sequence.

[0075] In one embodiment, the distortion correction unit 603 further upsamples the modified warping map to match a specified resolution of the output image prior to converting the input image.

[0076] The above paragraphs are described in multiple aspects. It is clear that according to the teachings of the specification, it can be performed in many ways. Any particular structure or function disclosed in the examples is only a representative case. According to the teachings of the specification, it should be noted that those skilled in the art can perform any disclosed aspect alone or combine two or more aspects.

[0077] Although the present application has been described in terms of the examples and preferred embodiments, it is to be understood that the application is not limited to the disclosed embodiments. Rather, the application is intended to cover various modifications and equivalent arrangements (to those skilled in the art). Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and equivalent arrangements.

Claims

1. A distortion correction method for a distortion correction device, the method comprising: The first operation includes modifying the deformed image using one or more image operations to obtain a modified deformed image, wherein the one or more image operations include cropping, geometric transformation, stitching, or a combination thereof; as well as The second operation includes converting the input image into an output image using the modified deformed image.

2. The distortion correction method as described in claim 1, characterized in that, The geometric transformation is rotation, scaling, or a combination thereof.

3. The distortion correction method as described in claim 1, characterized in that, The input image is not processed by any of the one or more image operations prior to the second operation.

4. The distortion correction method as described in claim 1, characterized in that, The first operation predetermines the modified deformation map, and the second operation further includes using the modified deformation map predetermined by the first operation to convert each frame of the input image sequence into an output image frame.

5. The distortion correction method as described in claim 1, characterized in that, The first operation further includes: The deformation image is cropped according to the bounding box to obtain multiple cropped portions from the deformation image; The geometric transformation is performed on the plurality of cut portions to obtain a plurality of transformed portions; and The multiple transformed parts are stitched together to form the modified deformed diagram.

6. The distortion correction method as described in claim 5, characterized in that, The first operation further includes: Perform object detection on the input image to obtain one or more bounding boxes; The deformation diagram is trimmed according to one or more bounding boxes to obtain the plurality of trimmed portions; The geometric transformations of the plurality of cropped portions are performed based on the bounding box to obtain the plurality of transformed portions.

7. The distortion correction method as described in claim 1, characterized in that, While the second operation is transforming the first frame of the input image sequence, the first operation is modifying the deformed image of the second frame following the first frame of the input image sequence.

8. The distortion correction method as described in claim 1, characterized in that, The second operation further includes upsampling the modified deformable image to match a specified resolution of the output image before transforming the input image.

9. A distortion correction device, comprising: An integrated circuit configured to convert each frame of an input image sequence into an output image frame using a modified deformable map, wherein the modified deformable map is predetermined by modifying the deformable map using one or more image operations, and wherein the one or more image operations include cropping, geometric transformation, stitching, or a combination thereof.

10. The distortion correction device as described in claim 9, characterized in that, The geometric transformation is rotation, scaling, or a combination thereof.

11. The distortion correction device as described in claim 9, characterized in that, The integrated circuit does not perform any of the one or more image operations on the input image sequence.

12. The distortion correction device as described in claim 9, characterized in that, The integrated circuit is further configured to upsample the modified deformable image to match the resolution of the output image before converting the input image.

13. A distortion correction device, comprising: A storage unit for storing at least the input image and the deformed image; A processing unit, coupled to the storage unit, reads the deformation image from the storage unit, modifies the deformation image through one or more image operations to generate a modified deformation image, and stores the modified deformation image into the storage unit. as well as A distortion correction unit, coupled to the storage unit and the processing unit, is used to receive a control signal from the processing unit, read the input image and the modified deformed image from the storage unit according to the control signal, and convert the input image into an output image using the modified deformed image; The one or more image operations mentioned above include cropping, geometric transformation, stitching, or a combination thereof.

14. The distortion correction device as described in claim 13, characterized in that, The geometric transformation is rotation, scaling, or a combination thereof.

15. The distortion correction device as described in claim 13, characterized in that, The processing unit does not perform any of the one or more image operations on the input image.

16. The distortion correction device as described in claim 13, characterized in that, The processing unit predetermines the modified deformed image by modifying the deformed image using one or more image operations, and the distortion correction unit uses the predetermined modified deformed image to convert the input image sequence into an output image sequence.

17. The distortion correction device as described in claim 13, characterized in that, The processing unit passes through: The deformed image is cropped to obtain multiple cropped portions from the deformed image; The geometric transformation is applied to the cut portion to obtain multiple transformed portions; as well as The multiple transformed parts are stitched together to form the modified deformed diagram.

18. The distortion correction device as described in claim 17, characterized in that, The processing unit further performs object detection on the input image to obtain one or more bounding boxes, crops the deformation map according to the one or more bounding boxes to obtain the plurality of cropped portions, and performs the geometric transformation on the plurality of cropped portions according to the one or more bounding boxes to obtain the plurality of transformed portions.

19. The distortion correction device as described in claim 13, characterized in that, While converting the first frame of the input image sequence, the distortion correction unit is modifying the deformed image of the second frame following the first frame of the input image sequence.

20. The distortion correction device as described in claim 13, characterized in that, The distortion correction unit further upsamples the modified deformed image to match the resolution of the output image before converting the input image.

Images