Method, system and apparatus for detecting objects in distorted images

Abstract

The present invention relates to a method, system and device for detecting an object in a distorted image. The method comprises: receiving (S302) an inverse of a mathematical representation of a distortion of a first distorted image (600); wherein the detection of the object comprises: sliding (S304) a sliding window (620) over the first distorted image (600), and for each of a plurality of positions (630, 634, 638) in the first distorted image (600): transforming (S306) the sliding window (620) based on the inverse of the mathematical representation of the distortion at the position (630, 634, 638); and using (S308) the transformed sliding window (720, 724, 728) in a sliding window algorithm to perform object detection at the position (630, 634, 638) in the first distorted image (600).

Description

Technical Field

[0001] The present invention relates to a method, a device and a system for detecting an object in a distorted image. Background Art

[0002] An important area of ​​camera application is the surveillance of locations. In surveillance applications, various different image processing algorithms are often used to process the video of the location being monitored. For example, it is common to implement algorithms that automatically detect motion in recorded video. Another example of an important feature is object detection in captured images. A typical approach is then to compare the captured image with images in a reference database. When the object detection algorithm matches features in the captured image and images in the reference database, the object is detected and identified.

[0003] However, there are several problems with this algorithm. For example, there are high requirements on the images in the reference database. For example, these images must reflect a wide variety of objects while depicting the objects in a recognizable way. Therefore, the objects are usually imaged under different lighting conditions and from a wide variety of directions. Therefore, the reference database usually contains a large number of reference images.

[0004] However, the captured images are rarely captured under ideal imaging conditions. For example, the captured images may suffer from low brightness or distortion. There are a range of different sources of image distortion, such as the use of wide-angle lenses (e.g., fisheye lenses and optical domes) and stitching techniques used to provide panoramic images.

[0005] Regardless of the source of distortion and its shape, distortion is a challenge when analyzing images. For example, many object detection algorithms suffer greatly when applied to distorted images because most algorithms are designed to be applied to non-distorted images. Therefore, it becomes computationally intensive for the processor to detect objects in distorted images.

[0006] Therefore, there is a need for improved algorithms for object detection in non-ideal images. Summary of the invention

[0007] In view of the above, an object of the present inventive concept is to eliminate or at least alleviate one or more of the above-mentioned deficiencies or problems in the art. Specifically, an object is to provide a method, system and device for detecting objects in distorted images.

[0008] According to a first aspect, a method for detecting an object in a first distorted image using a sliding window algorithm is provided. The method comprises: receiving an inverse of a mathematical representation of a distortion of the first distorted image; wherein the detection of the object comprises: sliding a sliding window over the first distorted image and for each of a plurality of positions in the first distorted image: transforming the sliding window based on the inverse of the mathematical representation of the distortion at the position; and performing object detection at the position in the first distorted image using the transformed sliding window in the sliding window algorithm.

[0009] Within the context of the present application, the term "distorted image" should be interpreted as an image with a distorted perspective. In a distorted image, straight lines in the scene are typically curved to some extent. In contrast, an image of perfectly straight lines has perfectly straight lines corresponding to the straight lines in the depicted scene. Within the context of the present application, two types of distortion sources are discussed: physical distortion sources and digital distortion sources. Non-limiting examples of physical distortion sources are wide-angle lenses, including fisheye lenses (e.g., f-theta lenses), optical domes, and imperfect rectilinear lenses. Imperfections in lenses may be due to manufacturing inaccuracies. Non-limiting examples of digital distortion sources are image stitching algorithms, for example, for producing panoramic images from multiple images. The distortion pattern may be irregular or regular (such as radial distortion). The distortion pattern of a captured image may be the result of one or a combination of distortion sources.

[0010] Within the context of the present application, the wording "sliding window algorithm" should be interpreted as an object detection algorithm that includes a sliding window. A sliding window is a rectangular area that initially has a predetermined width and a predetermined height, which moves across the image. Image features present in the area defined by the sliding window are compared to a database of reference features in order to detect objects in the image. The feature detection pattern in the sliding window may be based on a database of reference features. The sliding window algorithm may use multiple feature detection patterns, such that a first sliding window includes a first feature detection pattern, and a second sliding window includes a second feature detection pattern, and so on. Thus, the sliding window algorithm may detect multiple different features by using multiple different sliding windows and feature detection patterns. The sliding window algorithm may be a convolution-based algorithm.

[0011] Within the context of the present application, the expression "mathematical representation of the distortion" should be interpreted as a mathematical description of an image transformation that, when applied to a rectilinear image, results in an image distortion. It is to be understood that the aforementioned distortion may be mathematically represented as a polynomial, a matrix, or a lookup table. For example, the mathematical representation may be a polynomial / matrix describing the transfer function of a fisheye lens used when capturing the distorted image. The lookup table may include coordinates in the distorted image indexed by coordinates in the rectilinear (or non-distorted) image, and vice versa.

[0012] By means of the present method, a sliding window algorithm can be used to detect objects in a distorted image such as the first distorted image. Therefore, the image data associated with the first distorted image does not need to be transformed / deformed prior to object detection using the sliding window algorithm. Therefore, the computational costs associated with image transformations are reduced. For example, the need to transform a curved image into a rectilinear image prior to object detection can be reduced or completely removed. Reducing the need for image transformations can thereby reduce any unnecessary image cropping associated with such image transformations. Therefore, image features in a region that were removed due to image cropping can be included in the sliding window algorithm, and objects present in such a region can therefore be detected.

[0013] Further, since the first distorted image does not need to be transformed / deformed, the image data associated with the first distorted image does not need to be interpolated. Thus, the computational cost associated with the sliding window algorithm can be reduced because the sliding window algorithm does not need to include the image data generated in the image interpolation. The interpolated image data does not include additional information that is not already present in the image data associated with the captured image, so including the image data generated in the image interpolation in the sliding window algorithm only increases the computational cost without a corresponding increase in actual image information.

[0014] In addition, since the first distorted image does not need to be transformed, the method can be performed early in the image processing pipeline. Performing the method early in the image processing pipeline to detect the object can allow the detected object to be used as an input to subsequent steps in the image processing pipeline without delaying the subsequent steps in the image processing pipeline, thereby allowing a shorter processing time associated with the image processing pipeline. For example, the detected object can be used as an input for calculating encoder settings for a video stream formed by the image processing pipeline, and / or for drawing an overlay, such as a bounding box, in the video stream formed by the image processing pipeline. Another advantage associated with detecting the object early in the image processing pipeline is that subsequent steps in the image processing pipeline may not need to be performed for an analysis camera that is used only to detect the object. Therefore, since the analysis camera may not need to output a video stream, the power consumption of the analysis camera can be reduced.

[0015] The step of transforming the sliding window may include transforming a feature detection pattern of the sliding window.

[0016] Within the context of this application, the term "feature detection pattern" should be interpreted as a pattern that a sliding window algorithm uses to detect specific features. It is to be understood that a sliding window algorithm may include a plurality of different feature detection patterns. For example, a feature detection pattern may be used to detect edges with various angles in an image frame. A feature detection pattern may also be used to detect a person, a specific face of a person, or other objects such as a car, a dog, etc. in an image frame.

[0017] The feature detection pattern may be transformed based on the inverse of the mathematical representation of the distortion at the location of the sliding window.

[0018] The advantage of transforming the feature detection pattern of the sliding window is that the distorted features in the first distorted image can be detected. Therefore, the sliding window algorithm can detect the distorted object in the first distorted image.

[0019] A further advantage of transforming the feature detection pattern of the sliding window is that the feature detection pattern can be adapted to the spatial resolution of the first distorted image when the spatial resolution of the first distorted image can vary across the first distorted image. For example, in areas of low spatial resolution, a coarser feature detection pattern can be used in the sliding window algorithm and thereby reduce the computational cost associated with the sliding window algorithm.

[0020] Within the context of the present application, the term "spatial resolution" is to be understood as the spatial resolution of an image frame. In a distorted image acquired, for example, by a wide-angle lens or stitched from multiple image frames, different areas of the image have different spatial resolutions. In other words, areas of the same size of an image frame cover angles of different sizes of the field of view (FOV) of the camera. The spatial resolution can be specified at the pixel level of the image frame, or it can be determined at the pixel subgroup level (for example at the macroblock level). The spatial resolution can be expressed as the number of pixels per FOV angle or as the number of FOV angles per pixel. The skilled person is familiar with how to interchange between these expressions depending on the application. For example, in the implementation of the method according to the present application, one of these expressions may be preferably used. The spatial resolution distribution can be represented, for example, by a table indicating the spatial resolution distribution of pixels or pixel subgroups (for example macroblocks).

[0021] The step of transforming the sliding window may include transforming the size of the sliding window.

[0022] The size of the sliding window may be transformed based on the inverse of the mathematical representation of the distortion at the location of the sliding window.It is to be understood that the height of the sliding window may be transformed independently of the width of the sliding window.

[0023] An advantage of varying the size of the sliding window is that the size of the sliding window can be adapted to the spatial resolution of the first distorted image when the spatial resolution of the first distorted image can vary over the first distorted image. Thus, the computational cost associated with the size of the sliding window can be reduced.

[0024] The method may also include: using the transformed sliding window as a kernel in a first layer of a convolutional neural network.

[0025] Within the context of this application, the term "convolutional neural network" should be interpreted as an algorithm for image classification. The algorithm can be trained before using it for object detection. The training results in a database of convolution filters associated with specific image features. When a convolutional neural network is used for object detection, multiple convolutions are performed on the input image, wherein each of the multiple convolutions uses a different convolution filter. In other words, the first layer is a convolutional layer that applies a convolution operation (using a transformed kernel) to the input (image data of an image frame), passing the result to the next layer. Each convolution results in an image feature map associated with a convolution filter. The feature maps obtained from the multiple convolutions are then used to form the final output. The final output can then be used to detect objects in the input image.

[0026] An advantage associated with using the transformed sliding window as the kernel in the first layer of the convolutional neural network is that image transformation of the first distorted image may not be required. Thus, the computational cost associated with image transformation may be reduced.

[0027] The method may further include storing the transformed sliding window for each of a plurality of locations in the first distorted image.

[0028] An advantage associated with storing the transformed sliding window for each of the plurality of locations in the first distorted image is that the transformed sliding window may be used at a later time. For example, the transformed sliding window may be used for additional computations later in the image processing pipeline. Since the distortion between images is the same, there is no need to transform the transformed feature detection pattern in the sliding window and / or the size of the sliding window for each image frame. Storing the transformed sliding window for each of the plurality of locations in the first distorted image facilitates reusing the transformed sliding window in object detection in other distorted images captured in the same manner as the first distorted image, which in turn may reduce computational time and computational cost associated with object detection in the plurality of distorted images.

[0029] The transformed sliding window may be stored in a lookup table indexed by a position among the plurality of positions in the first distorted image.

[0030] An advantage associated with storing the transformed sliding window in a lookup table indexed by positions in a plurality of positions in the first distorted image is that it may allow for simplified retrieval of the transformed sliding window and thereby reduce associated computational costs.

[0031] Where the method is performed on a plurality of distorted images, the detection of the object in each of the plurality of distorted images may include using a transformed sliding window for object detection in a first distorted image.

[0032] Since the transformation of the sliding window may be performed once for a plurality of distorted images, the computational cost associated with the transformation of the sliding window may be reduced because the sliding window does not need to be transformed for each of the plurality of distorted images.

[0033] Furthermore, since the sliding window transformation can be performed for one of the multiple distorted images, the computational cost can be reduced compared to the transformation of each of the multiple distorted images in the prior art system. In other words, the computational cost associated with object detection in multiple distorted images can be reduced by the present method.

[0034] The method may further comprise encoding the plurality of transformed images into a transformed video stream.

[0035] The distortion may include optical distortion. The optical distortion may include barrel distortion, pincushion distortion, and / or moustache distortion. The optical distortion may include misalignment between the optical axis of the imaging optics and the image sensor. The optical distortion may include tangential distortion.

[0036] Distortion may include an image transformation applied to the image data, thereby forming a distorted image.

[0037] The image transformation may include an image filter. The image transformation may include image stitching. Multiple primary images may be stitched together to form a panoramic image. The distorted image may be the formed panoramic image. Those skilled in the art will appreciate that the formed panoramic image may include distorted features due to the image stitching.

[0038] An advantage associated with distortion including an image transformation applied to the image data to form a distorted image is that the distorted image associated with the image data can be filtered before the sliding window algorithm is applied to object detection. Therefore, certain features present in the image data can be reduced or removed prior to object detection.

[0039] An advantage associated with distortions including image stitching is that it may allow the present method to detect objects in panoramic images.

[0040] The step of transforming the sliding window may be hardware implemented. For example, the transformation of the feature detection pattern may advantageously be performed in hardware such as a graphics processing unit (GPU).

[0041] According to a second aspect, a computer program product is provided. The computer program product comprises a computer readable storage medium having instruction adapters to perform the method when executed by a device having processing capabilities.

[0042] The computer readable storage medium may be a non-transitory computer readable storage medium.

[0043] The above features of the method also apply to the second aspect where applicable. To avoid unnecessary repetition, please refer to the above.

[0044] According to a third aspect, there is provided a device configured to detect an object in a first distorted image using a sliding window algorithm. The device comprises: an image receiver configured to receive a first distorted image; a distortion receiver configured to receive an inverse of a mathematical representation of a distortion of the first distorted image; and at least one processor configured to: slide a sliding window over a plurality of positions in the first distorted image, and for each of the plurality of positions in the first distorted image, transform the sliding window based on the inverse of the mathematical representation of the distortion at the position; and use the transformed sliding window in the sliding window algorithm to perform object detection at the position in the first distorted image.

[0045] The above features of the method and / or computer program product also apply to the third aspect where applicable. To avoid unnecessary repetitions, please refer to the above.

[0046] The apparatus may further include a non-transitory storage medium configured to store the transformed sliding window for each of a plurality of locations in the distorted image.

[0047] The device may be a video camera.

[0048] According to a fourth aspect, a system is provided for detecting an object in a distorted image using a sliding window algorithm. The system comprises: a camera configured to capture a distorted image of a scene; and the device; wherein an image receiver of the device is configured to receive the distorted image of the scene captured by the camera.

[0049] The above features of the method, computer program product and / or device also apply to the fourth aspect when applicable. To avoid unnecessary repetition, please refer to the above.

[0050] Further scope of applicability of the present disclosure will become apparent from the detailed description given below. However, it should be understood that the detailed description and specific examples, although indicating preferred variations of the inventive concept, are given only by way of illustration, since various changes and modifications within the scope of the present invention will be apparent to those skilled in the art from the detailed description.

[0051] Therefore, it is to be understood that the inventive concept is not limited to the specific steps of the described method or the components of the described system, because such methods and systems can be changed. It is also to be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to be limiting. It must be noted that, as used in the specification and the appended claims, the articles "a", "the" and "said" are intended to mean that there are one or more elements, unless the context clearly indicates otherwise. Thus, for example, references to "a unit" or "the unit" may include multiple devices, etc. In addition, the words "comprise", "include" and similar words do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other aspects of the present invention will now be described in more detail with reference to the accompanying drawings showing embodiments of the present invention. The drawings should not be considered to limit the present invention to the specific embodiments; rather, they are used to explain and understand the present invention.

[0053] As illustrated in the figures, the sizes of layers and regions are exaggerated for illustrative purposes and, thus, are provided to illustrate the general structure of embodiments of the present invention. Like reference numerals refer to like elements throughout.

[0054] Figure 1A An apparatus configured to detect objects in a distorted image using a sliding window algorithm is illustrated.

[0055] Figure 1B Picture camera.

[0056] Figure 2A The illustration includes a scene of straight lines.

[0057] Figure 2B Diagram Figure 2A Distorted image of the scene in .

[0058] Figure 2C Figure 2 shows the feature detection pattern and multiple transformed sliding windows.

[0059] Figure 3 is a block diagram of a method for detecting an object in a first distorted image using a sliding window algorithm.

[0060] Figure 4 A system configured to detect objects in a distorted image is shown. DETAILED DESCRIPTION

[0061] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which presently preferred variations of the inventive concept are shown. However, the inventive concept may be implemented in many different forms and should not be construed as limited to the variations set forth herein; rather, these variations are provided for thoroughness and completeness and to fully convey the scope of the inventive concept to the skilled person.

[0062] Detecting objects in a distorted image can be problematic when features in the image are deformed. Therefore, one solution is to correct the distorted image before applying the object detection algorithm. Correction is the process of inverting the distorted image into a linearly projected image, for which the object detection algorithm works better. However, correction itself is a computationally intensive operation that not only burdens the processor, but also takes up valuable resources in the processor, such as time, power, and bandwidth. Furthermore, correction burdens the calibrator unit in the camera system, which is a limited resource, and therefore other processes that also need to access the calibrator may be affected.

[0063] The inventors have realized that by transforming the sliding window in the sliding window algorithm, objects can be detected directly in the distorted image. Therefore, using the inventive concept, there is no need to correct the distorted image before applying the sliding window algorithm to object detection. Figure 4 The inventive concept is described.

[0064] Figure 1A to Figure 1B The apparatus 100 is shown as being configured to detect an object in a first distorted image 600 using a sliding window algorithm. FIG. 2A to FIG. 2C to explain the function of the device.

[0065] The apparatus 100 comprises an image receiver 102. The image receiver 102 is configured to receive a first distorted image (see Figure 2B , reference numeral 600). The first distorted image 600 may be a frame in a video stream. The image receiver 102 may be configured to receive image data from an image sensor. The image receiver 102 may be an image sensor.

[0066] The device 100 further includes a distortion receiver 104. The distortion receiver 104 is configured to receive an inverse of a mathematical representation of a distortion of the first distorted image 600. The distortion may include optical distortion. A collimator may be used to determine the optical distortion. The distortion may be determined based on a distorted image of a known planar target. The known planar target may include a varying pattern and / or a repeating pattern. For example, the known planar target may include a repeating pattern of a known geometric shape. The repeating pattern may be a checkerboard pattern.

[0067] The distortion may include an image transform applied to the image data, thereby forming a distorted image. The image transform may be associated with the stitching of images used to form the panoramic image. In some variations, the distortion is a combination of optical distortion and an image transform applied to the captured image data.

[0068] The image receiver 102 and the distortion receiver 104 may be a single receiver.

[0069] The device 100 further comprises at least one processor 106. The at least one processor 106 is configured to: slide a sliding window 620 over a plurality of positions in the first distorted image 600, and for each position 630, 634, 638 of the plurality of positions in the first distorted image 600: transform the sliding window 620 based on an inverse of the mathematical representation of the distortion at the position 630, 634, 638; and perform object detection at the position 630, 634, 638 in the first distorted image 600 using the transformed sliding window 720, 724, 728 in a sliding window algorithm.

[0070] The sliding window 620 may include a feature detection pattern 700. A plurality of feature detection patterns may be received from a server (not shown) communicating with the device 100, or the plurality of feature detection patterns may be stored in the device 100. The plurality of feature detection patterns may be predetermined by a training process. The training process may use a plurality of images including features of interest. The training process may use a plurality of images not including features of interest. For example, the training process may use a plurality of images including a car and a plurality of images not including a car.

[0071] The training process may include an optimization technique for determining the kernel of a convolutional neural network (CNN) that will best match the features of interest. In a CNN, the first layer is always a convolutional layer using a sliding window algorithm and a set of defined kernels. In a typical CNN scenario, each convolutional layer has its own convolutional kernel group, and weights should be trained for the convolutional layer based on the object detection scenario of the CNN (features of interest, such as cars, people, etc.). As described above, for distorted images, the defined kernel may not always be sufficient. Using the inventive concepts described herein, the size and / or feature detection pattern of the kernel group of the convolutional layer of the CNN can be transformed based on the inverse of the mathematical representation of the distortion and the position in the distorted image. Therefore, the distorted image can be used as an input to the CNN, and the kernel of the CNN, in particular the kernel of the first layer, can be transformed alternatively, but additionally / alternatively also used for other convolutional layers of the CNN.

[0072] The at least one processor 106 may be further configured to stitch images received from an image sensor such as a camera to form a panoramic image. The first distorted image 600 may be the formed panoramic image.

[0073] The device 100 may further include: Figure 1A . The non-transitory storage medium 108 illustrated in . The non-transitory storage medium 108 may be configured to store the transformed sliding windows 720, 724, 728 for each of a plurality of positions 630, 634, 638 in the first distorted image 600. The non-transitory storage medium 108 may be further configured to store the distorted image received by the image receiver 102. The non-transitory storage medium 108 may be further configured to store the distortion and / or the inverse of the distortion associated with a particular camera and / or camera model. In the case where the distortion is stored in the storage medium 108, the inverse of the distortion may be calculated using at least one processor 106. The non-transitory storage medium 108 may be further configured to store a plurality of feature detection patterns.

[0074] The device 100 may further include: Figure 1A The encoder 110 illustrated in FIG. The encoder 110 may be configured to encode the transformed image into another video stream. The non-transitory storage medium 108 may be further configured to store the other video stream.

[0075] The device 100 may include Figure 1A The image receiver 102 , the distortion receiver 104 , the at least one processor 106 , the non-transitory storage medium 108 , and / or the encoder 110 may communicate via the data bus 112 .

[0076] like Figure 1B As illustrated in FIG. 1 , the device 100 may be a camera 200. The camera 200 may include a Figure 1B . The optical device 202 may be an imaging optical device. The imaging optical device may be a camera objective. The optical device may image the scene 500. The device 100 may be configured to generate a panoramic image of the scene 500. The at least one processor 106 may further be configured to stitch images to form a panoramic image of the scene 500.

[0077] Now refer to FIG. 2A to FIG. 2C The inventive concept is further described. Figure 2A The illustration shows a scene 500 including a plurality of straight lines 510, 512, 514, 516, 518. A straight line image of the scene 500 will reproduce the straight lines 510, 512, 514, 516, 518. However, the image is usually distorted, which is Figure 2B exemplified as barrel distortion. Figure 2B Diagram Figure 2A A distorted image 600 of the scene 500 in FIG. Figure 2B, straight lines 510, 512, 514, 516, 518 in scene 500 appear as curved lines 610, 612, 614, 616, 618 in distorted image 600. As illustrated in distorted image 600, the distortion varies in distorted image 600. For example, near the center of distorted image 600, straight line 514 in scene 500 is imaged as straight line 614 in distorted image 600. Near the edge of distorted image 600, straight lines 510, 518 in scene 500 are imaged as curved lines 610, 618 in distorted image 600. In other words, for Figure 2B , the distortion is smaller in the center of the distorted image 600 and larger toward the edge of the distorted image 600. Thus, the degree and shape of the distortion depends on the distortion itself and the location 630, 634, 638 in the distorted image 600.

[0078] exist Figure 2B In FIG. 6 , a sliding window 620 is shown. In order to correctly identify features in the distorted image 600 , the feature detection pattern 700 may be transformed based on the inverse of the distortion and the positions 630 , 634 , 638 in the distorted image 600 . Figure 2C In the example shown in , the feature detection pattern 700 is associated with a straight line. For example, applying the feature detection pattern 700 directly to the distorted image 600 will not correctly detect features associated with the straight line at the first position 630 and the third position 638 in the distorted image 600. However, for example, applying the feature detection pattern 700 directly to the distorted image 600 will correctly detect features associated with the straight line at the second position 634 in the distorted image 600. Therefore, in order for the sliding window algorithm to correctly identify features associated with the feature detection pattern 700, the sliding window 620 can be transformed based on the inverse of the distortion for each position 630, 634, 638 in the distorted image 600. This is illustrated by three transformed sliding windows 720, 724, 728, which include Figure 2C 634, 638 in the distorted image 600. Thus, applying the transformed sliding windows 720, 724, 728 in the sliding window algorithm to the distorted image 600 will correctly identify features associated with the feature detection pattern 700 for each location 630, 634, 638 in the distorted image 600.

[0079] It is to be understood that the lines 510, 512, 514, 516, 518 in the scene 500 are Figure 2A In the figure, Figure 2B Optical distortion and Figure 2CThe feature detection patterns 700 in FIG. 7 are merely examples, and they are described herein to explain the inventive concept. It is to be understood that different image features (e.g., real-world objects) and different distortions (e.g., pincushion distortion, mustache distortion, and / or image stitching) may be used to explain the inventive concept.

[0080] Figure 3 is a block diagram of a method S300 for detecting objects in a first distorted image 600 using a sliding window algorithm. The method S300 comprises receiving S302 an inverse of a mathematical representation of a distortion of the first distorted image 600.

[0081] The distortion may include optical distortion. The optical distortion may include barrel distortion, pincushion distortion, and / or moustache distortion. The optical distortion may include misalignment between the optical axis of the imaging optics and the image sensor.

[0082] The distortion may include an image transformation applied to the image data to form a distorted image. The image transformation may include image stitching. Image stitching may stitch multiple primary images to form a panoramic image. The distorted image may be a panoramic image.

[0083] It is to be understood that a distorted image may include optical distortions and image transformations applied to the image data.

[0084] Detection of the object includes sliding S304 a sliding window 620 over the first distorted image 600, and for each of a plurality of positions 630, 634, 638 in the first distorted image 600: transforming S306 the sliding window 620 based on an inverse of a mathematical representation of the distortion at the position 630, 634, 638, and performing object detection at the position 630, 634, 638 in the first distorted image 600 using S308 the transformed sliding window 720, 724, 728 in a sliding window algorithm.

[0085] The step of transforming S306 the sliding window 620 may include transforming S310 the feature detection pattern 700 of the sliding window 620 .

[0086] The step of transforming S306 the sliding window 620 may include transforming S312 the size of the sliding window 620 .

[0087] The step of transforming S306 the sliding window 620 may be hardware-implemented. The step of transforming S306 the sliding window 620 may be implemented in an application specific integrated circuit (ASIC). In other variations, the step of transforming S306 the sliding window 620 may be implemented in software in at least one processor 106 of the device 100.

[0088] The transformed sliding windows 720 , 724 , 728 may be stored in a lookup table indexed by the locations 630 , 634 , 638 of the plurality of locations in the first distorted image 600 .

[0089] The method S300 may further include: using the sliding windows 720, 724, 728 transformed in S314 as kernels in a first layer of a convolutional neural network.

[0090] The method S300 may further include: storing S316 the transformed sliding window 720 , 724 , 728 for each position 630 , 634 , 638 of the plurality of positions in the first distorted image 600 .

[0091] In the case where method S300 is performed on a plurality of distorted images, the detection of the object in each of the plurality of distorted images may include using the transformed sliding windows 720, 724, 728 for object detection in the first distorted image 600. The plurality of distorted images may be a video stream. Method S300 may further include encoding the plurality of transformed images into a transformed video stream.

[0092] Figure 4 The system 800 is configured to detect objects in a distorted image using a sliding window algorithm. The system 800 includes a camera 810 and the device 100, wherein the camera 810 is configured to capture a distorted image of a scene 500; wherein the image receiver 102 of the device 100 is configured to receive the scene 500 ( Figure 2A ). Camera 810 may include imaging optics 812. Image receiver 102 of device 100 may be configured to receive a distorted image of scene 500 via a wired or wireless communication interface. The distorted image may be first distorted image 600. System 800 may include a plurality of cameras configured to produce a panoramic image of scene 500. System 800 may be mounted in a single assembly.

[0093] A person skilled in the art realizes that the inventive concept is by no means limited to the preferred variants described above. On the contrary, many modifications and variations are possible within the scope of the appended claims.

[0094] Additionally, variations to the disclosed variations can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0095] The systems and methods disclosed above may be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks between the functional units mentioned in the above description does not necessarily correspond to the division of physical units; on the contrary, a physical component may have multiple functions, and a task may be performed collaboratively by multiple physical components. Some or all components may be implemented as software executed by a digital signal processor or a microprocessor, or as hardware or as an application-specific integrated circuit. For example, the transformation of the sliding window may be implemented in a GPU or an application-specific integrated circuit ASIC, and the sliding window algorithm running with the transformed sliding window may be implemented in software running on the central processing unit CPU of the device.

Claims

1. A method (S300) for detecting an object in a first distorted image (600) using a sliding window algorithm, the method (S300) comprising: receiving (S302) an inverse of a mathematical representation of the distortion of said first distorted image (600), wherein the mathematical representation of the distortion is an image transformation which, when applied to a rectilinear image of a scene, results in the first distorted image of the scene, and An object in the first distorted image is detected by sliding (S304) a sliding window (620) over the first distorted image (600), the sliding window (620) comprising a feature detection pattern (700), and for each of a plurality of locations (630, 634, 638) in the first distorted image (600): transforming (S306) the sliding window (620) based on the inverse of the mathematical representation of the distortion at the position (630, 634, 638), wherein the step of transforming (S306) the sliding window (620) comprises: transforming (S310) the feature detection pattern (700) of the sliding window (620) such that a final distortion of the feature detection pattern of the transformed sliding window (720, 724, 728) corresponds to the distortion at the position of the first distorted image (600); and The transformed sliding window (720, 724, 728) including the transformed feature detection pattern is used (S308) in the sliding window algorithm.

2. The method (S300) according to claim 1, wherein: The step of transforming (S306) the sliding window (620) includes transforming (S312) the size of the sliding window (620).

3. The method (S300) according to claim 1, further comprising: For each position (630, 634, 638) of the plurality of positions in the first distorted image (600), the transformed sliding window (720, 724, 728) is stored (S316).

4. The method (S300) according to claim 3, wherein: The transformed sliding window (720, 724, 728) is stored in a lookup table indexed by the location (630, 634, 638) of the plurality of locations in the first distorted image (600).

5. The method (S300) according to claim 3, performed on a plurality of distorted images, wherein: The detection of an object in each of the plurality of distorted images comprises using the transformed sliding window (720, 724, 728) for object detection in the first distorted image (600).

6. The method (S300) according to claim 1, wherein: The distortion includes optical distortion.

7. The method (S300) according to claim 1, wherein: The distortion comprises an image transformation applied to the image data, thereby forming a distorted image.

8. The method (S300) according to claim 1, wherein: The step of transforming the sliding window (620) is implemented in hardware.

9. A non-transitory computer-readable storage medium having an instruction adapter, the non-transitory computer-readable storage medium performing the method (S300) of claim 1 when executed by a device having processing capabilities.

10. A device (100) configured to detect an object in a first distorted image (600) using a sliding window algorithm, the device (100) comprising: An image receiver (102), the image receiver (102) being configured to receive the first distorted image (600); a distortion receiver (104) arranged to receive an inverse of a mathematical representation of a distortion of the first distorted image (600), wherein the mathematical representation of the distortion is an image transformation which, when applied to a rectilinear image of a scene, results in the first distorted image of the scene; and At least one processor (106) configured to detect an object in the first distorted image, the at least one processor (106) being arranged to: Sliding a sliding window (620) over a plurality of locations in the distorted image (600), the sliding window (620) comprising a feature detection pattern (700), and for each location (630, 634, 638) of a plurality of locations in the first distorted image (600): transforming the sliding window (620) based on the inverse of the mathematical representation of the distortion at the location (630, 634, 638), and transforming the feature detection pattern (700) of the sliding window (620) such that a final distortion of the feature detection pattern of the transformed sliding window (820, 724, 728) corresponds to the distortion of the first distorted image (600); and Object detection is performed at the location (630, 634, 638) in the first distorted image (600) using the transformed sliding window (720, 724, 728) including the transformed feature detection pattern in the sliding window algorithm.

11. The apparatus (100) of claim 10, further comprising a non-transitory storage medium (108) configured to store the transformed sliding window (720, 724, 728) for each of a plurality of locations (630, 634, 638) in the distorted image (600).

12. The device (100) according to claim 10, wherein The device (100) is a camera (200).

13. A system (800) configured to detect an object in a distorted image using a sliding window algorithm, the system (800) comprising: a camera (810), the camera (810) being configured to capture a distorted image of the scene (500); as well as The device (100) according to claim 10; The image receiver (102) of the device (100) is configured to receive the distorted image of the scene (500) captured by the camera (810).

Images