Image distortion transformation method and device

Abstract

The present application relates to an image distortion transformation method and device. An image distortion transformation method for transforming an original image having an original distortion profile imaged by an imager into a transformed image optimized for a distortion processing unit includes inputting an original image from the imager into a transformation unit, inputting an original image distortion profile into an original distortion profile memory of the transformation unit, and inputting a target distortion profile into a target distortion profile memory of the transformation unit. The target distortion profile is different from the original distortion profile. The method also includes transforming the original image into a transformed image by transforming the distortion profile of the original image from the original image distortion profile into the target image distortion profile, and outputting the transformed image from the transformation unit.

Description

[0001] This application is a divisional application of the invention patent application with national application number 201780023985.1, the application date of which is February 16, 2017, and the name of the invention is “Image distortion transformation method and device”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of currently pending U.S. Provisional Patent Application No. 62 / 295,724, filed February 16, 2016, entitled “Image Distortion Transformation Method and Apparatus,” the entire contents of which are incorporated herein by reference. Background Art

[0004] Embodiments of the present invention relate to image or imager distortion processing methods and apparatus for transforming an image, optimizing it for a distortion processing algorithm (e.g., a dewarping algorithm). Using the same method, embodiments of the present invention allow for the magnification of certain areas of interest in an image or imager field of view, thereby optimizing the storage, transmission, resolution, or display of the content.

[0005] Two-dimensional (2D) images are created by imagers such as narrow-angle imagers, panoramic imagers, wide-angle optical systems, fisheye optical systems, panomorph optical systems, or even images stitched from multiple narrow-angle images or images stitched from multiple panoramic images. Regardless of which imager is used, the image often contains optical distortion. This behavior is caused by the remapping of the three-dimensional (3D) environment into a 2D plane. As a result, the content in the image is deformed, and the exact distortion profile depends on the imager parameters. Typically, the distortion is more noticeable for panoramic imagers.

[0006] In order to properly display the image without distortion or feed the correct image to any image processing algorithm, the distortion in the panoramic image must be handled, such as for linear distortion (in U.S. Patent No. RE 44087 E) or custom symmetric and custom asymmetric distortion (in U.S. Patent Nos. 6844990 B2 and 6865028 B2), respectively. These existing distortion handling algorithms or dewarping algorithms (the action of correcting, removing, processing or modifying the distortion of an image) are implemented by software or hardware on different platforms (e.g., computers, smart phones, tablets, system on chip (SoC), etc.). As an example, Milestone System's IPIX plug-in is a dewarping software that corrects the distortion of fisheye lenses. Some dewarping algorithms use only linear distribution functions to correct different distortion profiles from different panoramic imagers. In this case, because the linear correction function used does not match the actual distortion profile of the panoramic image or imager, some distortion remains in the image after dewarping. In order to properly correct the distortion, the dewarping algorithm must use a distribution function (distortion profile) to take into account the distortion profile of the panoramic imager and the application user experience. This means that the deviation between the distortion profile and the dewarping algorithm distribution function must be controlled. One way could be to input the distortion profile as a mathematical function to the distortion processing algorithm to correct the panoramic image. However, this approach cannot be used in all cases, for example but not limited to, sometimes the distortion profile is confidential or does not exist in the image processing unit running the distortion processing algorithm. In addition, the distortion profile of a new imager cannot be easily added or updated in the existing software or hardware processing algorithm. This requires a software update, a firmware update or a hardware modification (e.g. an ASIC modification).

[0007] The ImmerVision Enables 2.0 algorithm corrects the distortion of a holographic image or imager. The library comprising the algorithm contains some distortion profiles (or distribution functions) corresponding to the holographic imagers or holographic lenses available at the time of the release of the algorithm in the image processing unit (library, ASIC, FPGA, etc.). If the distortion profile of the panoramic image or imager is in this list of pre-recorded distortion profiles, the user only needs to enter the distortion profile reference number and the processing algorithm will use the appropriate existing distortion profile. For example, at least some embodiments in US patent application US 2015 / 0281507 A1 use a mark on the image to record this distortion profile reference number or the distortion profile itself. In embodiments where only the reference number of the distortion profile is marked, the distortion processing unit can read the reference number from the mark and automatically use the corresponding distortion profile. This approach is more efficient than passing the entire distortion profile to the distortion processing algorithm. In some cases, this approach cannot be used because the distortion profile does not exist in the algorithm and the algorithm is therefore not designed to support this new type of distortion profile. An example of such a case is a panoramic image with a new distortion profile, created by, for example but not limited to, a new wide angle, fisheye or holographic lens or a new image processing algorithm. For this image, the distortion processing unit must be updated with the new distortion profile or a completely new algorithm before the distortion processing unit can correct, remove, process or modify the distortion of the panoramic image or imager. In some cases, the algorithm update cannot be easily performed, especially when the distortion processing algorithm is implemented in hardware. This limits the application of the distortion processing algorithm to different panoramic images or imagers.

[0008] Due to the large data size of images or videos, optimization of image storage or transmission resources is always preferred. The optimization method is to reduce the image or video size. For panoramic images or wide-angle images, U.S. Patent No. 8238695 proposes to trim a portion of the image data of the virtual camera from the entire image to reduce the image data size. This solution sacrifices the image data outside the virtual camera area, and it can only be used for the use case of the virtual camera. The image saved or transmitted contains only the image data of the virtual camera, while other image data from the rest of the field of view is lost. This is not suitable for the application of the entire panoramic image. Another alternative to keep the entire panoramic field of view is to uniformly reduce the entire image to a lower resolution and reduce the image size. However, this scaling is the same for the entire image, and the image quality is degraded throughout the image. An optimization method would be to use a higher resolution in important image areas and a lower resolution in other image areas, indicating that different scaling ratios are applied to different areas of the image. In this way, more pixels are used to emphasize important areas when reducing the image size. This nonlinear scaling causes new distortion in the generated image. This new distortion is different from the original distortion of the original panoramic image. Therefore, the distortion in the original panoramic image cannot be used to process this optimized image.

[0009] U.S. Patent No. 6865028 proposes projecting a non-disc-shaped panoramic image onto a rectangular image sensor. This includes but is not limited to elliptical images. Compared with circular images, elliptical images occupy more pixels on a rectangular image sensor, and therefore there are fewer black pixels in the output image. In this way, from the total file size of the image, a larger portion is used by useful image content compared to useless black content. The resulting image has a higher effective resolution. For image storage and transmission, this elliptical panoramic image containing fewer black pixels is more efficient than a circular panoramic image. However, U.S. Patent No. 6865028 only suggests optical components that implement this method of creating a holographic optical system, and the distortion profile is unique to each holographic optical system. In addition, the distortion profile is fixed after the holographic optical system is generated. The distortion profile can only be modified by designing a new lens, so the existing optical lens cannot be modified, and the distortion cannot be dynamically changed according to the application.

[0010] Dynamically changing distortion has been proposed in the past in US Patent Publication No. 2012 / 0314065 A1. However, the present invention uses a hardware optical system consisting of a deformable optical surface as a deformable mirror to change the distortion in real time. This construction is complex and cannot be used in existing panoramic imagers to dynamically modify the region of interest.

[0011] Since 2006, the Holographic Geometry Simulator (PGMS) software developed by ImmerVision simulates distortions in images to analyze image distortion effects. The PGMS is described in the paper "Novel hemispheric image formation: concepts and applications" (published in "Proc. SPIE 6994, Photon Management III, 699406 (April 25, 2008)"). The software is able to create different distortions on the image, including different preset distortions, nonlinear scaling distortions, or any customized distortions. However, the software is not used for panoramic image distortion processing, and it cannot transform the panoramic image with the new distortion profile in order to optimize it for the distortion processing algorithm. It also cannot pass the panoramic image distortion to the nonlinear scaling distortion to emphasize important areas in the image. Finally, it cannot be integrated into the panoramic image processing pipeline. Summary of the invention

[0012] In order to overcome all the above-mentioned problems, an embodiment of the present invention proposes an image distortion transformation method to transform images captured by different imagers so as to optimize for distortion processing algorithms. This allows the image distortion processing algorithm to process images captured by different imagers. Here, the imager includes but is not limited to a narrow-angle lens, a wide-angle lens, a fisheye lens, a holographic lens, a multi-lens system, a catadioptric system, a panoramic image generator by image stitching, or any image processing unit, etc. The image distortion processing algorithm is used to process image distortion for image display or any other image processing algorithm. The distortion processing algorithm can be implemented by software or hardware on different platforms (including but not limited to computers, smart phones, tablets, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), systems on chips (SoCs), etc.). The image distortion transformation method is used to transform the original image into an image with any kind of distortion profile that the image distortion processing unit has supported. This image distortion transformation method can be implemented by software or hardware on different platforms (including but not limited to computers, smart phones, tablets, FPGAs, ASICs, SoCs, etc.). This method can also be implemented by an optical system or any kind of system or associated device. This image distortion transformation method can be implemented in different locations (including but not limited to a camera device, an image capture device, a media server, a client computer, a display device, an image processing algorithm, or a cloud system).

[0013] More specifically, in at least some embodiments of the present invention, a panoramic raw image captured by a panoramic imager is transformed into an image with a well-known distortion profile that is already supported by a distortion processing algorithm. The distortion processing algorithm is able to process distortion according to this supported distortion profile.

[0014] In order to optimize image storage, transformation, resolution, quality or display, an effective way is to allocate more pixels to the region of interest in the image. More specifically, in at least some embodiments of the present invention, the target distortion profile is designed to increase the number of pixels occupied by the region of interest and reduce the number of pixels occupied by less important content. The region of interest has a higher resolution (or pixel density) than the less important content. The distortion transformation method can transform the original image into a transformed image with a suitable profile before the image is saved, transmitted, analyzed, processed or displayed. The distortion transformation method can change the magnification across the image, rather than the linear expansion and contraction of the original image. In this way, the important image area can be emphasized while the image size can be kept the same. In some embodiments, the present invention can be used to reduce the image while keeping more pixels in the region of interest than other parts of the image. In some other embodiments, this transformation can be performed dynamically in real time according to application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above summary and the following detailed description of the preferred embodiments of the present invention will be better understood by reading in conjunction with the accompanying drawings. For ease of illustration, the presently preferred embodiments are shown in the accompanying drawings. However, it should be understood that the present invention is not limited to the exact arrangements and means shown.

[0016] In the attached picture:

[0017] Figure 1 is a schematic diagram of an image transformation process for transforming an original panoramic image into a transformed panoramic image having a target distortion profile known to a distortion processing unit;

[0018] Figure 2 is a schematic diagram detailing an image transformation method within an image transformation unit;

[0019] Figure 3 showing several examples of original image formats and several examples of transformed images;

[0020] Figure 4 is an exemplary embodiment of the present invention, wherein the selection of the transformed image distortion profile is performed according to some parameters of the original panoramic imager;

[0021] Figure 5 is an exemplary embodiment of the present invention, wherein the selection of the transformed image distortion profile is dynamically changed according to changes in the image content of the original panoramic image;

[0022] Figure 6 is an exemplary embodiment of the present invention, wherein the image transformation unit generates a plurality of transformed images from the original panoramic image, the transformed images each having a different distortion profile for a given application;

[0023] Figure 7 is an exemplary embodiment of the present invention, wherein the image transform unit generates a plurality of transformed images from an original panoramic image, the transformed images each having a different target distortion profile input from different devices having different applications; and

[0024] Figure 8 is an exemplary embodiment of the present invention, wherein the image transform unit generates a transformed image from an original panoramic image, the transformed image changing a target distortion profile input from a display device in real time according to a position of a region of interest required for its application. DETAILED DESCRIPTION

[0025] As used in the claims and the corresponding parts of this specification, the terms "a" or "an" mean "at least one".

[0026] Figure 1 A schematic diagram of an image distortion transformation method for transforming a panoramic image from a panoramic imager into an image with a target distortion profile is shown. The capture device 100 is preferably constructed with a panoramic imager 110. Alternatively, the capture device 100 can be constructed with one or more narrow angle imagers 105. The capture device 100 can be any device to which a panoramic imager can be added to add a panoramic imaging feature, including but not limited to mobile phones, tablets, cars, robots, TVs, routers, GPS units, Internet boxes, home surveillance cameras, security cameras, motorcycle cameras, endoscopes, etc. When the panoramic imager 110 is integrated into the capture device 100, this device can capture an original panoramic image 120.

[0027] Here, the panoramic imager 110 is any device that is capable of capturing an image or portion of an image from a wide angle panoramic environment. The resulting panoramic environment preferably produces a field of view of approximately 120° or greater. Some examples of panoramic imagers 110 that can be used in the process are, but are not limited to, wide angle lenses, catadioptric imagers, multiple narrow field of view cameras, computer generated panoramic images, etc.

[0028] The original panoramic image 120 is an output image of the capture device 100. The panoramic image 120 contains a panoramic image content 125. There are some original image distortions in the panoramic image content 125. When the panoramic image distortion processing unit 160 does not know the distortion profile in the original panoramic image 120, the image distortion cannot be easily processed. An embodiment of the present invention proposes to add an image distortion transformation unit 130 so as to transform the original panoramic image 120 into an image with a target distortion profile (which is known to the distortion processing unit). In this way, the distortion in the original panoramic image 120 can be optimized for the distortion processing unit 160.

[0029] Both the original panoramic image 120 and the transformed panoramic image 140 can be any kind of panoramic images, including but not limited to circular or elliptical projected images, peripherally projected images, non-linearly scaled images, wide-angle images with continuous, discontinuous, symmetric or asymmetric, or any other projection.

[0030] The original panoramic image 120 is the input image of the image distortion transform unit 130. The image distortion transform unit 130 can be implemented by software or hardware. It can be implemented on any platform (including but not limited to computers, smart phones, tablets, FPGAs, ASICs, SoCs, etc.). It can also be implemented by an optical system or any kind of system or associated device. This method can be implemented in different locations (including but not limited to camera devices, image capture devices, media servers, client computers, panoramic display devices, panoramic image processing algorithms, or cloud systems). By targeting Figure 2 To further illustrate, the image distortion transform unit 130 outputs a transformed panoramic image 140. The transformed panoramic image 140 includes a target distortion contour.

[0031] More specifically, in at least some embodiments of the present invention, there are some common distortion profiles of holographic images (a type of panoramic image created by a holographic imager that creates a zone of interest). The holographic distortion processing software or hardware contains these common distortion profiles and can process any image mapped using these common distortion profiles. The distortion transformation method proposed in the present invention transforms the original panoramic image into a transformed image having one of the common holographic distortion profiles. In this way, the holographic distortion processing software or hardware can process the distortion of any original panoramic image after the distortion transformation unit.

[0032] Before the panoramic image is displayed or processed in the algorithm unit, the distortion in the transformed panoramic image 140 needs to be processed by the distortion processing unit 160. The transformed panoramic image 140 can be directly input to the algorithm unit 175 or to the distortion processing unit 160. Alternatively, the transformed panoramic image 140 can be output via image storage or different transmission methods 150 (including but not limited to the transfer of digital images through the Internet, transfer via TCP-IP protocol, USB, firewall, memory, socket, transfer using physical media (such as floppy disk, laser disk, disk, USB drive, hard drive) or transfer of the image by physically printing the image and then scanning it back as a digital file, etc.). When the image is transferred, it can be done internally within the device or externally to another device.

[0033] Because the distortion processing unit 160 knows the target distortion profile in the transformed panoramic image 140 , it is able to process the distortion and produce a processed image 180 .

[0034] The processed image 180 can be displayed on a display device 170. The display device 170 can be anything that partially or completely displays the result of the distortion processing unit, including but not limited to a computer or TV screen, a mobile phone, a tablet, a virtual reality (VR) headset, an augmented reality (AR) headset, a projector, etc. In addition, the processed image 180 or the transformed panoramic image 140 can also be used inside another algorithm unit 175 (which further processes the processed image to extract relevant information, including but not limited to: pedestrian detection, tracking and recognition; face recognition; road lane departure; hazard recognition; 3D position tracking; 3D reconstruction, etc.).

[0035] In at least some embodiments of the present invention, display device 170 or algorithm unit 175 can communicate with image distortion transform unit 130 via optional link 190 and change the transform unit's targeted distortion profile in real time as required by its application.

[0036] Figure 2 Show Figure 1 Detailed operations of the image distortion transformation unit 130 in FIG. The image distortion transformation method 230 is implemented inside the image distortion transformation unit 210. In addition, the image distortion transformation unit 210 further includes an original distortion profile memory 220 and a target distortion profile memory 225. The original image distortion profile 250 and the target image distortion profile 260 are stored in the two memory units, respectively. The image distortion transformation method 230 uses the original image distortion profile and the target image distortion profile to transform the original panoramic image 200 into a transformed panoramic image 240 distorted according to the target distortion profile. The image distortion transformation method 230 can include, but is not limited to, pixel mapping, interpolation, filtering, or different image processing algorithms.

[0037] The original panoramic image 200 is the input image of the image distortion transformation method 230. Each different original panoramic image 200 can contain a different original image distortion profile 250, which is created by a different panoramic imager. The image distortion transformation method 230 needs to know the distortion profile of the original panoramic image 200. The original image distortion profile 250 can be input to the original distortion profile memory 220 from some internal or external source (including but not limited to manual selection from a list of pre-saved image distortion profiles from a user interface, automatic analysis of the content of the original image (including reading tags or metadata in the original image), or automatic transmission from the imager). In these cases, the original image distortion profile 250 is written inside the original distortion profile memory 220. In other embodiments of the present invention, such as the case of an ASIC used for a specific transformation application, the original distortion profile can be hard-written inside the original distortion profile memory 220 and cannot be adjusted.

[0038] The image distortion transformation method 230 needs to know the target image distortion profile of the output transformed panoramic image 240, which can be processed by an existing distortion processing algorithm. The target distortion profile can be one of several public holographic distortion profiles, for example, but not limited to. In this case, the target image distortion profile 260 can be pre-stored in the target distortion profile memory 225, such as the case of an ASIC programmed for a single specific application. In addition, the target image distortion profile 260 can also be input to the target distortion profile memory 225 from some internal or external sources. The target distortion profile 260 can be a selection from a list of pre-stored target image distortion profiles within the target distortion profile memory 225 or a whole image distortion profile. The source of the target image distortion profile 260 includes, but is not limited to, an image distortion processing unit, a panoramic image display device, a face tracking or other image processing algorithm, etc. The image distortion transformation method 230 transforms the original image into a target image having a distortion profile optimized for an existing distortion processing algorithm. The output transformed panoramic image 240 with the target distortion profile can be processed later by existing distortion processing algorithms.Thus, the distortion processing unit can process any panoramic image after the image distortion transformation method 230 without any modification or update of the distortion processing unit.

[0039] Both the original image distortion profile 250 and the target image distortion profile 260 can be static distortion profiles. In some embodiments of the present invention, they can also be dynamic distortion profiles. The original image distortion profile 250 can be updated in real time, for example, when the distortion profile of the original panoramic image 200 changes over time for a dynamic panoramic imager without limitation. In another embodiment, the target image distortion profile 260 can be updated to a different target image profile required by the application without limitation.

[0040] Several examples of original and transformed images are shown in Figure 3 The original panoramic image of the image distortion transformation method 300 can be a circular panoramic image 310 or an elliptical panoramic image 317. In some other embodiments of the present invention, the image distortion transformation method 300 obtains the original multiple images from multiple narrow images 320 or obtains the original multiple images from multiple panoramic images as input, and stitches them into a single or double panoramic image with a target transformed distorted contour. In some other embodiments of the present invention, the input can also be an original equidistant rectangular image 330 or a single narrow image 335.

[0041] By means of the image distortion transformation method 300, the output transformed panoramic image can have different target distortions. Some examples of the output transformed panoramic images are 350, 360, 365, 370, 375, 380, 385 and 390. After the transformation, more pixels of information are recorded inside the region of interest. The present invention allows the preservation of all original information in the region of interest while reducing the total image size.

[0042] As a first example, some fisheye or holographic imagers create an image with raw panoramic image content 315 only in the central disk area (e.g., in raw panoramic image 310). The image is black outside the raw panoramic image content disk. With a rectangular image file (which is output by the image sensor, with a 16:9 or 4:3 aspect ratio), the black border is even larger than the raw image content disk. In such an image, the black border is not useful. When the black border becomes narrower and occupies fewer pixels, the useful image content will occupy more pixels and have a higher resolution.

[0043] A simple distortion contour transformation is to stretch a circular area with image content into an elliptical area, as in the transformed panoramic image A 350. The transformed image content 355 occupies more pixels in the image and there are fewer black boundary pixels than the original panoramic image content 315. More complex distortion contours can also be applied, including but not limited to super-elliptical shapes (as in the transformed panoramic image B 360) or nonlinear plano-celestial projections (as in the transformed panoramic image H 390). When panoramic images are transformed in this manner before being stored or transmitted, more image storage or transmission resources can be used on important content rather than useless black content.

[0044] As another example, the image display software or hardware can request the image distortion transform method 300 to increase the resolution of a region of interest when the user or device wishes to zoom in on that region. Better image quality can be achieved when the resolution is increased by the image distortion transform method 300 before the image is stored or transmitted, compared to digital scaling of the degraded image after the image is stored or transmitted. The image distortion transform method 300 can increase the resolution in certain locations, as in the transformed panoramic image C 365 with edge zooming, the transformed panoramic image D 370 with center zooming, and the transformed panoramic image E 375 with mid-zone zooming. The exact location of the region of interest is chosen according to the requirements from the user or application.

[0045] As another example, the image distortion processing algorithm is implemented by hardware. With this particular hardware, only circular symmetric distortion profiles are supported. When the original panoramic image does not have the same rotational symmetry as in the original elliptical panoramic image 317, the image distortion transformation method is able to transform the image into a transformed image with a rotationally symmetric distortion profile.

[0046] As another example, the target distortion profile can be any kind of projection distortion that can be processed by the image distortion processing unit. The transformed panoramic image can be: an elliptical panoramic image 350; a superelliptical 360; circular panoramic images 365, 370 and 375; or a parametrically projected panoramic image 390. In most embodiments according to the invention, the field of view of the original image and the field of view of the transformed image are similar. Alternatively, the transformed panoramic image can also be a dewarped portion of the full image in the region of interest with a smaller field of view as in the transformed image F 380. More specifically, the case of the transformed image F at 380 shows that in some embodiments of the invention, it is not mandatory to maintain the panoramic view by compressing the rest of the image, and we can combine this processing (which creates zones of increased resolution) with cropping of other less important parts of the image when required by the application.

[0047] As another example, when the input image is an original narrow image 335 with a certain distortion profile, the output transformed image G 385 can also be a narrow angle with a different distortion that creates a region of interest.

[0048] As another example, when the input images are the original multiple images 325 as in the case of stereo panoramic imaging, the output can be a transformed multiple images F 380 to allow a different transformed image to be displayed to each eye after processing while increasing the number of pixels in the region of interest.

[0049] Figure 4 An example embodiment of the present invention is shown, in which the selection of the distortion profile of the output transformed image is made according to some parameters of the original panoramic imager. In at least some embodiments of the present invention, the output transformed image distortion profile is selected according to some parameters of the original panoramic imager. Figure 3The target distortion profile of the image distortion transformation process can be a dynamic distortion profile required by the application, and the target distortion profile can change in real time according to any parameters from the original image or the imager. These parameters include, but are not limited to, device orientation, device location or altitude from a GPS device, date and time, ambient conditions (including climate or underwater imaging), or user selection. As a specific example that does not limit the scope of the present invention, the distortion transformation method can use different target distortion profiles for different orientations of the capture device. When the panoramic imager 400 is facing downward or upward, the image content on the border is more important because they are objects around the camera device. In this case, the image distortion transformation unit 420 can transform the original panoramic image into a transformed panoramic image 430 with higher resolution toward the edge of the image content. Alternatively, when the panoramic imager 410 is facing forward, the image distortion transformation unit 420 can transform the original panoramic image into a transformed panoramic image 440 with higher resolution in the central area, which contains objects in front of the camera device.

[0050] Figure 5 Another example embodiment of the present invention is shown, in which the target distortion profile is dynamically adjusted according to the scene content. The adjustment of the distortion profile can come from any source, including the user manually moving the region of interest or the algorithm unit performing specific object recognition or face tracking. In the example shown, there is a person moving in the field of view of the panoramic imager, and the region of interest follows him. Initially, the person 502 is on the left side of the object scene 500 imaged by the panoramic imager 530. The image distortion transformation unit 540 is therefore configured to create a region of interest with a higher magnification 552 in the left part of the image 550, while still maintaining a panoramic view by compressing the rest of the image. At a later time, the person 512 is now in the center of the object scene 510 imaged by the same panoramic imager 530. The image distortion transformation unit 540 is therefore configured to create a region of interest with a higher magnification 562 in the center part of the image 560, while still maintaining a panoramic view by compressing the rest of the image. At a later time, the person 522 is on the right side of the object scene 520 imaged by the same panoramic imager 530. The image distortion transform unit 540 is therefore configured to create a region of interest with a higher magnification 572 in the left portion of the image 570, while still maintaining a panoramic view by compressing the rest of the image.

[0051] Figure 6An exemplary embodiment of the invention is shown, in which the image distortion transformation unit 605 outputs multiple transformed images from the same original panoramic image 600. Alternatively, in another embodiment according to the invention, the multiple transformed images can also originate from multiple original panoramic images, as in the case of stereoscopic vision. In this specific example, which does not limit the scope of the invention, there are two transformed images 610 and 615, each of which has a different region of interest. The image 610 on the left has more pixels in the region of interest in the center of the image, while the image 615 on the right has more pixels in the region of interest on the right side of the image. These multiple transformed images are then stored or transmitted at 620 for use by the image distortion processing unit 630. When the image is transmitted, it can be done internally within the device or externally to another device. After the distortion processing, two processed images 640 and 645 are output, each image showing the portion of the image where the region of interest is in the transformed images 610 and 615. These processed images 640 and 645 are then used inside a single or multiple viewing devices 650. These devices can be, for example, virtual reality viewing devices. In at least some embodiments of the present invention, there is also an optional communication link 660 between the viewing device 650 and the image distortion transform unit 605, allowing the viewing device to modify the position and size of the region of interest in real time as required by the application.

[0052] Figure 7 An example embodiment of the present invention is shown in which an image distortion transform unit 705 outputs multiple transformed images 710 from the same original panoramic image 700. The output images may have any output transformed distortion according to the present invention. The target transformed image distortion profile is configured as required by the application and transmitted to the image distortion transform unit 705 via a communication link 750. The transformed images 710 are then stored internally at 720 or transmitted to a distortion processing unit 730. When the image is transmitted, it can be done internally within the device or externally to another device. The output from this distortion processing unit 730 is a plurality of display devices 740. According to its application, the display device 740 requests the accurate transformed distorted image in real time from the image distortion transform unit 700 via a communication link 750. Each different display device can have a different region of interest or a different transformed image distortion profile.

[0053] Figure 8An example embodiment of the invention is shown in which an image distortion transform unit 805 continuously generates transformed panoramic images 810 from an original panoramic image 800 and outputs them to an image storage or transmission 820 for use in a distortion processing unit 830. When the image is transmitted, it can be done internally within the device or externally to another device. The distortion processing unit 830 receives the transformed image, which contains panoramic content or a portion of the panoramic content with a larger field of view than required to be displayed. The distortion processing unit 830 extracts and dewarps the area of ​​the image centered on the area of ​​interest of the panoramic image requested by the display device 840. These display devices 840 display a dewarped view of the image centered on the area of ​​interest. When the display device requires a new area of ​​interest (such as, for example, the rotation of the head of a user using a virtual reality display device), the display device 840 communicates with the image distortion transform unit 810 via a communication link 850 in order to have a new transformed panoramic image 810 (with this new area of ​​interest). This process can take some time due to latency between the display device and the image distortion transform unit. However, because the present invention creates a transformed image that still has full panoramic content in the image or a portion of the panoramic content with a larger field of view than required to be displayed (with lower resolution outside the region of interest), the distortion processing unit 830 is able to instantaneously de-warp the content of the new region of interest and display the result on the display device at a lower resolution. After this waiting time period is over, the distortion processing unit eventually receives a transformed image with a new target distortion profile that has maximum resolution in the new region of interest. The resulting de-warped view displayed in the display device 840 is improved from a lower resolution view to a maximum resolution view. In addition, in some embodiments of the present invention, only the difference between the two transformed images from the image distortion transformation unit 805 is transmitted or stored, thereby optimizing storage or transmission and reducing system latency.

[0054] All of the above are drawings and examples of specific image distortion transform units and methods. In all of these examples, the imager can have any field of view from very narrow to extremely wide angle. These examples are not intended to be an exhaustive list or to limit the scope and spirit of the invention. Those skilled in the art will appreciate that changes may be made to the above embodiments without departing from the broad inventive concept thereof. It is therefore to be understood that the invention is not limited to the specific embodiments disclosed, but is intended to cover modifications within the spirit and scope of the invention as defined by the appended claims.

Claims

1. An image distortion transformation method for transforming an original image imaged by an imager having an original distortion profile into a transformed image, the method comprising: a) inputting the original image from the imager to a transform unit; b) inputting the original image distortion contour into the original distortion contour memory of the transform unit; c) inputting a target distortion contour into a target distortion contour memory of the transform unit, wherein the target distortion contour is different from the original distortion contour and the target distortion contour is dynamically adjusted according to scene content; d) transforming the original image into a transformed image by transforming the distortion profile of the original image from the original image distortion profile to the target distortion profile, the transformed image not being a linear expansion or contraction of the original image; and e) Outputting the transformed image from the transform unit.

2. The image distortion transformation method according to claim 1, wherein: Dynamic adjustments to the distortion contours come from algorithmic units that perform object recognition or face tracking.

3. The image distortion transformation method as claimed in claim 1, wherein: The raw image from the imager is a panoramic image.

4. The image distortion transformation method as claimed in claim 1, wherein: The original image and the transformed image have similar fields of view.

5. The image distortion transformation method as claimed in claim 1, wherein: The raw image from the imager consists of multiple images.

6. The image distortion transformation method according to claim 1, wherein: The target distortion profile is selected by the transform unit according to parameters of the imager.

7. The image distortion transformation method according to claim 6, wherein: The parameter is the orientation of the imager.

8. The image distortion transformation method according to claim 1, wherein: The transformed image is output to a processing unit.

9. The image distortion transformation method according to claim 8, wherein: The target distortion profile is supported by the processing unit.

10. The image distortion transformation method according to claim 8, wherein: The distortion contour is transferred from the transform unit to the processing unit, or the distortion contour is transferred from the processing unit to the transform unit.

11. The image distortion transformation method according to claim 10, wherein: The distortion profile is transmitted with the transformed image.

12. The image distortion transformation method of claim 1, further comprising a display device, wherein the distortion profile is transmitted from the display device to the transformation unit.

13. The image distortion transformation method as claimed in claim 1, wherein: Outputting the transformed image to a display device, the method further comprising: f) adjusting, by the display device, an input target distortion profile of the transformation method according to a specific application required by the display device.

14. The method of claim 13, wherein: The input target distortion profile of the transformation method is adjusted to modify the position, size or magnification of the region of interest.

15. The method of claim 1, wherein: The original distortion contour is input into the transform unit by manual selection through a user interface, automatically input based on content analysis of the original image, automatically transferred directly from the imager, or hard-coded into the original distortion contour memory.

16. The method of claim 12, wherein: The target distortion profile is input into the transform unit by manual selection through a user interface, automatically transmitted from a distortion processing unit, the display device or an algorithm unit, or hard-coded in the target distortion profile memory.

17. An image distortion transformation method used in an image distortion transformation unit in combination with a distortion processing unit, the method being used to create a region of interest in a transformed image, the region of interest being dynamically adjusted according to scene content, the method comprising: a) a distortion processing unit configured to process the distortion profile; b) An image distortion transformation unit transforms the original image into a transformed image according to the distortion contour, wherein the transformed image is not a linear expansion or contraction of the original image.

18. The image distortion transformation method of claim 17, wherein the dynamic adjustment of the region of interest comes from an algorithm unit that performs object recognition or face tracking.

19. An image capturing system for transforming an original image having an original distortion profile imaged by an imager into a transformed image, the system comprising a transform unit having an original distortion profile memory and a target distortion profile memory, the transform unit being configured to: a) receiving the raw image from the imager; b) inputting the original image distortion contour into the original distortion contour memory; c) inputting a target distortion profile into the target distortion profile memory, wherein the target distortion profile is different from the original distortion profile, and the target distortion profile is dynamically adjusted according to scene content; d) transforming the original image into a transformed image by transforming the distortion profile of the original image from the original image distortion profile to the target distortion profile, the transformed image not being a linear expansion or contraction of the original image; and e) Output the transformed image.

20. The image capture system of claim 19, wherein: Dynamic adjustments to the distortion contours come from algorithmic units that perform object recognition or face tracking.

Images