Self-learning distortion correction

Abstract

The present invention relates to self-learning distortion correction. A method for distortion correction in images captured by a non-rectilinear camera is provided, the method comprising acquiring a plurality of images of a scene captured by the camera over time, determining the position in the image of a base of an object that has moved across a horizontal surface in the scene, determining a boundary of the horizontal surface in the scene based on the determined position of the base, generating a three-dimensional model of the scene by defining one or more vertical surfaces around the determined boundary of the plane of the scene, and correcting the distortion of at least one of the images by projecting the image onto the three-dimensional model of the scene. Corresponding apparatus, computer programs and computer program products are also provided.

Description

Technical Field

[0001] The present disclosure relates to distortion correction of images of a scene. In particular, the present disclosure relates to self-learning distortion correction of images captured by a camera with a non-rectilinear lens, such as, for example, a fisheye lens. Background Art

[0002] In many cases, it is desirable for a single camera to be able to capture a larger portion of a scene, and a wide angle lens may be used for this purpose. Such wide angle lenses are typically non-rectilinear, and therefore produce strong optical distortions of the image, so that straight lines or other objects in the scene end up being bent or distorted in the image of the scene captured using such a lens. An example of a non-rectilinear lens is a so-called fisheye lens, which is able to produce, for example, a hemispherical image of a scene, and which typically has a wide viewing angle of more than, for example, 90 degrees, 180 degrees, or even 270 degrees.

[0003] An example of the situation described above is a surveillance camera mounted on, for example, the ceiling of a room and configured to capture most or all of the room below the ceiling using a fisheye lens. Such a camera provides improved situational awareness by being able to capture more of what is happening around the camera than, for example, a conventional pan / tilt / zoom (PTZ) camera.

[0004] However, as mentioned above, one disadvantage of cameras with such non-rectilinear lenses is that the images they produce are distorted (e.g., twisted). Although there exist various solutions for digitally "de-warping" images captured by, for example, a fisheye lens, such solutions generally require manual adjustment of multiple parameters before the obtained results are sufficiently satisfactory. Furthermore, in an installation comprising many such cameras, the manual effort required to adjust the de-warping parameters for each camera may quickly become time consuming.

[0005]

[0006] Therefore, there is a need for improved ways to facilitate the use of non-rectilinear lenses in, for example, surveillance camera installations. Summary of the invention

[0006] To at least partially meet the above needs, the present disclosure provides an improved method for distortion correction in an image of a scene captured by a camera with a non-rectilinear lens, an apparatus configured to perform such a method, and a computer program and a computer program product as defined in the attached independent claims. Various embodiments of the improved method, apparatus, computer program and computer program product are defined in the attached dependent claims.

[0007] According to a first aspect of the present disclosure, a method for performing distortion correction in an image of a scene captured by a camera having a non-rectilinear lens is provided. The method comprises: i) acquiring a plurality of images of a scene captured by the camera over time. The method further comprises: ii) determining the position of the bottom of one or more objects that have moved across a horizontal surface in the scene in at least some of the plurality of images. The method further comprises: iii) determining the boundary of a horizontal surface in the scene based on the determined position of the bottom. The method further comprises: iv) generating a three-dimensional (3D) model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface. The method further comprises: v) correcting the distortion of at least one of the plurality of images by projecting at least one image onto the three-dimensional model of the scene. The method may be implemented, for example, on a device / computer such as, for example, a device comprising a processing circuit.

[0008] As used herein, a "horizontal surface" is, for example, a floor or ground of a scene or any other surface on which, for example, people, animals, or other objects typically walk / move. Similarly, a "vertical surface" is, for example, a wall or other surface arranged around the periphery of a horizontal surface (such as a wall surrounding the floor of a room), and generally extending perpendicular to the floor. The term "bottom" may, for example, include feet (of an animal or person), wheels (of, for example, a robotic vacuum cleaner or the like), or any other part of an object that contacts a horizontal surface when the object moves in the scene.

[0009] Using a room as an example of a scene, conventional solutions for distortion correction of images of a room typically require an operator / user to manually indicate which parts of the image of the room belong to the floor of the room and which parts of the image belong to vertical surfaces of the room (such as, for example, walls) in order to build a sufficient 3D model of the room required for distortion correction. The solution of the present disclosure improves on this conventional technique by using the recognition that the bottom of objects moving in the scene (such as, for example, feet) are typically in contact with the floor, and therefore the outer edge / boundary of the floor can be learned / estimated by examining how the bottom moves over time in images of the room. By assuming that other parts of the image that are not within this outer edge / boundary are, for example, parts of the walls, a 3D model of the room can be automatically constructed without the need for manual segmentation of the image. As mentioned earlier in this document, this can be particularly useful in larger installations of multiple cameras because, using the solution of the present disclosure, each camera can self-learn how to model the room and then perform the required distortion correction.

[0010] In some implementations of the method, the non-rectilinear lens (of the camera) may be a fisheye lens.

[0011] In some embodiments of the method, the base may include a foot of an object (such as a human foot or an animal foot).

[0012] In some embodiments of the method, step ii) may include identifying one or more key points of one or more objects. As used herein, a "key point" may be, for example, an identified foot, knee, leg or any other feature of an object in the image, and the actual foot position may be estimated based on the position of these key points. For example, if it is known where, for example, the knees and / or hips of a person sitting on a sofa are, the position of the feet may be calculated even if the feet are not visible in the image of the scene. This may be a possible scenario, for example, if there is a table placed in front of the sofa and the position of the feet is allowed to be inferred in any way and used to determine the boundaries of the floor.

[0013] In some embodiments of the method, the scene may be a scene of a room, and the horizontal surface may be a floor of the room.

[0014] In some embodiments of the method, at least some of the plurality of images may be captured during a training period, and if the number of bottoms that move in a region of at least some of the plurality of images during a finite time interval exceeds a threshold, the region may be considered to belong to a horizontal surface of the scene. Requiring detection of multiple bottoms in the same region before classifying the region as part of the horizontal surface of the scene may, for example, increase confidence that the region is actually part of a horizontal surface.

[0015] In some embodiments of the method, the three-dimensional model may include a horizontal surface defined by the determined boundary, and one or more vertical surfaces may be projected in a direction perpendicular to the horizontal surface. For example, the horizontal surface may be a floor of a room, and the vertical surface may be (the surface of) a wall that is typically arranged in the room so that the wall forms a 90 degree angle with the floor.

[0016] In some embodiments of the method, if the scene is of a room, the camera may be arranged in the ceiling of the room. The camera may, for example, be arranged so that the optical axis of the camera faces directly towards the floor, for example if the ceiling extends in a plane parallel to the plane of the floor, so that, for example, the optical axis of the camera is perpendicular to the floor plane (or horizontal surface).

[0017] According to a second aspect of the present disclosure, there is provided a device for performing distortion correction in an image of a scene captured by a camera having a non-rectilinear lens. The device includes a processing circuit. The processing circuit is configured to cause the device to: i) acquire multiple images of a scene captured by the camera over time; ii) determine the position of the bottom of one or more objects that have moved across the horizontal surface in the scene in at least some of the multiple images; iii) determine the outer boundary of the horizontal surface of the scene based on the determined position of the bottom; iv) generate a three-dimensional (3D) model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface of the scene, and v) correct the distortion of at least one of the multiple images by projecting at least one image onto the three-dimensional model of the scene. Therefore, the device is configured to perform the steps of the method of the first aspect. The device may, for example, include a memory storing instructions, which, when executed by the processing circuit of the device, causes the device to perform the above steps i) to v). Therefore, in such an embodiment, the device is "configured to" perform the above steps i) to v) means that the processing circuit is configured to read and execute the instructions stored in the memory.

[0018] In some embodiments of the apparatus, the processing circuit is further configured to cause the apparatus to perform embodiments of the method of the first aspect discussed and disclosed herein.

[0019] In some embodiments of the apparatus, the apparatus may be a camera with a non-rectilinear lens, such as a fish-eye lens camera or the like.

[0020] According to a third aspect of the present disclosure, a computer program for performing distortion correction in an image of a scene captured by a camera having a non-rectilinear lens is provided. The computer program includes computer code, which, when executed on a processing circuit of the device, causes the device to: i) acquire multiple images of a scene captured by the camera over time; ii) determine the position of the bottom of one or more objects that have moved across a horizontal surface in the scene in at least some of the multiple images; iii) determine the outer boundary of the horizontal surface of the scene based on the determined position of the bottom; iv) generate a three-dimensional (3D) model of the scene by defining one or more vertical surfaces around the determined boundary of the horizontal surface of the scene, and v) correct the distortion of at least one of the multiple images by projecting at least one image onto the 3D model of the scene. Thus, the computer code causes the device to perform the steps of the method of the first aspect.

[0021] In some embodiments of the computer program, the computer code may further cause, when run on a processing circuit of an apparatus, to cause the apparatus to perform any embodiment of the method of the first aspect as discussed and disclosed herein.

[0022] According to a fourth aspect of the present disclosure, a computer program product is provided. The computer program product includes a computer-readable storage medium, and the computer program according to the third aspect (or any embodiment of the present disclosure discussed and disclosed herein) is stored on the computer-readable storage medium. As used herein, the computer-readable storage medium can be, for example, non-transitory, and can be, for example, provided as a hard disk drive (HDD), a solid-state drive (SDD), a USB flash drive, an SD card, a CD / DVD, and / or as any other storage medium capable of non-transitory storage of data. In other embodiments, the computer-readable storage medium can be temporary, and for example corresponds to a signal (electrical signal, optical signal, mechanical signal, or the like) present on, for example, a communication link, a wire, or a similar signal transmission tool.

[0023] Other objects and advantages of the present disclosure will be apparent from the following detailed description, the accompanying drawings and the claims. Within the scope of the present disclosure, it is contemplated that all features and advantages described with reference to, for example, the method of the first aspect are related to, applicable to and can be used in combination with the apparatus of the second aspect, the computer program of the third aspect and the computer program product of the fourth aspect, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Exemplary embodiments will be described below with reference to the accompanying drawings, in which:

[0025] Figure 1A schematically illustrates an image of a scene / room captured by a camera with a non-rectilinear lens;

[0026] Figure 1B Schematically illustrating the process of determining the Figure 1A The bottom of the moving objects in the scene;

[0027] Figure 1C to Figure 1G Schematically illustrating the process of performing the method according to various embodiments of the present disclosure based on Figure 1B The bottom of the determined Figure 1A The various steps of creating a 3D model of the scene;

[0028] Figure 1H Schematically illustrates as generated by various embodiments of the method according to the present disclosure, Figure 1A A distortion-corrected image of the scene;

[0029] Figure 2A Various embodiments of the method and apparatus according to the present disclosure are schematically illustrated using various functional units;

[0030] Figure 2B Schematically illustrated Figure 2A The process of the method, and

[0031] FIG. 3A to FIG. 3C Various embodiments of the apparatus according to the present disclosure are schematically illustrated.

[0032] In the accompanying drawings, unless otherwise stated, the same reference numerals will be used for the same elements. Unless explicitly stated to the contrary, the drawings only show the elements necessary to illustrate the example embodiments, and other elements may be omitted or merely suggested for clarity. As illustrated in the accompanying drawings, the (absolute or relative) sizes of elements and regions may be exaggerated or underestimated relative to their actual values ​​for illustrative purposes, and are therefore provided to illustrate the general structure of the embodiments. Specific embodiments

[0033] Figure 1A An image 100 of a scene is schematically illustrated. In this specific example, the scene is a room. The image 100 is captured by a camera with a non-linear lens (such as, for example, a fisheye lens), so that the image 100 is distorted, so that straight lines are shown as curves, etc. The camera is envisioned to be installed in the ceiling (not shown) of the room and is arranged so that the optical axis of the camera directly faces the floor 110 of the room. The floor 110 is surrounded by four walls 120a to 120d, which protrude vertically upward from the floor 110 and are such that they form four corners 130a to 130d at the locations where the floor 110 and the respective walls intersect. A door 140 is also provided in the wall 120a, through which an object can enter / leave the room. At the moment of capturing the image 100, the room is empty (except, for example, furniture, wall decorations or the like), i.e., there are no people, animals or other moving objects in the room.

[0034] As can be seen in image 100, the lines connecting the corners together along floor 110 appear bent / distorted due to the projection of the camera's non-rectilinear lens onto the camera's image sensor. This also applies to most other lines in image 100, which are distorted / bent in image 100 due to distortion introduced by the camera's lens. Therefore, it is desirable to correct such distortions so that, for example, lines that are straight in the real world also remain straight in the image depicting the scene. Using conventional techniques, in order to correct the distortion in image 100 introduced by the camera's non-rectilinear lens, an operator would have to manually identify which portion of image 100 belongs to floor 110 (i.e., the horizontal surface of the scene shown in image 100). For example, an operator may perform such a task (using a graphical user interface showing image 100) by, for example, tracing along a curve connecting the four corners 130a to 130d, and / or by manually indicating, for example, areas of image 100 corresponding to the four walls 120a to 120d and door 140, i.e., the vertical surfaces of the scene. Generally speaking, such a process is time consuming and may even be impossible to handle if there are many (eg, hundreds) cameras to be configured and prepared for distortion correction in a larger camera installation.

[0035] How the present disclosure solves this problem / problem will now be described in more detail with reference to the remaining drawings, first with reference to Figure 1B In the following, the terms "scene" and "room" will be used interchangeably. The same applies to the terms "vertical surface" and "wall" and the terms "horizontal surface" and "floor", which will also be used interchangeably.

[0036] Figure 1B Schematically illustrates a superposition of multiple images 101 of a room captured over time, wherein multiple footprints 150 indicate where the bottom of one or more objects moving across a horizontal surface / floor in the scene is located. From the images 101, it is apparent, for example, that one or more objects have entered through a door 140 and moved to an object in the room (such as a compartment, a wall or the like), and then left again through the door 140. It should be noted that the footprints 150 are not necessarily actual footprints left behind by a person (such as footprints caused by dirt, rubber marks or the like under a shoe), but are used to illustrate / symbolize the position of the bottom of these objects (such as, for example, feet) determined over time. For example, key point detection can be used as a possible means to estimate where such a bottom of an object is located in the image. If a person or other object does leave real footprints (such as dirt / rubber marks) on the floor 110 when moving, then these footprints can of course also be detected and used as an indication of where the bottom of these objects is located in the image.

[0037] Also refer to Figure 1C to Figure 1HIt will now be explained how the contemplated method uses multiple images 101 and a “footprint” 150 to ultimately perform distortion correction of one or more of the images 101 .

[0038] Figure 1C It schematically illustrates how a base moves in a scene over time, illustrated using footprints 150. The footprints 150 are drawn in an (x, y) coordinate system, such that each footprint 150 is located at a specific (x, y) coordinate.

[0039] Figure 1D Schematically illustrates the Figure 1C The same footprint 150 is shown in FIG. 1 , but with a dashed shape 160 indicating the extension of the footprint in the plane of the floor 110 .

[0040] Figure 1E It is schematically illustrated how a rectangular boundary 170 is defined based on the outer edges 160 of the footprints 150, based on known parameters of the camera. This can be done, for example, based on knowledge of how a non-rectangular lens refracts light from different angles. For e.g. a fisheye lens, how the lens refracts light is based on the distance between the pixel (the pixel on the image sensor of the camera used to capture the image of the scene) and the optical center of the lens (the center from which the optical axis of the lens / camera extends), and may not require, for example, knowledge of the mounting height of the camera above the floor 110. The boundary 170 can be found, for example, by correcting for such refraction for all footprints 150, or, for example, only the outermost footprints 150. What is important is not how this is done, but that the determined positions of the footprints 150 in the image of the scene are used to tell that these positions correspond to the floor 110, and not, for example, any of the walls 120a to 120d.

[0041] Figure 1F Schematically illustrates how, after identifying the boundary 170, the boundary 170 is used as the horizontal surface of the three-dimensional model 102. Figure 1F As shown in FIG. 1 , the three-dimensional model 102 is incomplete and needs to be completed with one or more vertical surfaces.

[0042] Figure 1G It is schematically illustrated that the three-dimensional model 102 is completed to form a complete three-dimensional model 103. In the model 103, vertical surfaces in the form of walls 172a to 172d have been added / defined, such that the walls 172a to 172d are defined around the boundary 170. Figure 1G In the example shown, walls 172a to 172d are added so that they extend vertically away from the plane of boundary 170, i.e., in the example Figure 1G . Figure 1GIn the example shown in , the three-dimensional model 103 includes a floor 171 defined by a boundary 170 and four walls 172a to 172d intersecting each other and the floor 171 at four corners 174a to 174d. The walls 172a to 172d have the same height h. As previously described, the three-dimensional model 104 is therefore created under the assumption that areas in the image where no objects move are statistically likely to belong to walls of, for example, a room. The parameter h can be adjusted as needed to simulate the height of, for example, a room.

[0043] Figure 1H Again the room is shown schematically from above, but Figure 1A The image 100 has now been projected onto the three-dimensional model 103. The resulting image 104 is a distortion corrected version of the image 100, as can be seen therein, for example, that the lines connecting the four corners 130a to 130d are no longer twisted / bent as before.

[0044] As contemplated herein, performing the projection of the image 100 onto the three-dimensional model 103 may be based, for example, on knowledge of the orientation between the floor 171 and the camera, which orientation may be obtained, for example, by assuming that the optical axis of the camera / lens is orthogonal to the floor 171. When performing the projection of the image 100 onto the three-dimensional model 103, the assumed position of the camera may be intentionally changed from its real position, for example, to correct for the fact that the camera is not exactly centered within the ceiling of the room or the like. Changing the assumed position of the camera may, for example, help to obtain a similar number of pixels on both sides (horizontally and / or vertically) of the generated image 104. In some embodiments, when performing the projection, it is also contemplated that the assumed height of the camera above, for example, the floor of the room may also be changed so that it does not match the real height. This may be of particular interest if, for example, one wants to change the perspective of the generated image 104. In general, by acquiring (from the position of the bottom of an object that has been moved across a horizontal surface) knowledge of which parts of the image 100 correspond to, for example, a wall and to the floor, respectively, a three-dimensional model may be constructed, which may then be used to perform the projection of the image 100 onto a three-dimensional model (such as the model 103). How exactly the projection is then performed may vary based on, for example, the desired perspective effect and, for example, the desired viewpoint (eg, the desired / assumed position of the camera relative to a horizontal surface).

[0045] The accuracy of the proposed method may of course depend on whether objects moving in the scene visit all parts of, for example, the floor. If no objects move through a particular part of the floor in the multiple images of the scene provided to the proposed method, the resulting three-dimensional model of the scene will likely be too small and objects outside the envisaged floor will appear distorted. However, if it is assumed that the likelihood of each area of ​​the floor being visited by at least one object increases over time, this effect may be reduced by analyzing more images of the scene. In addition, by also using key point detection as described above, the true position of the bottom of objects can also be identified in areas of the floor that are hidden from the camera, since, for example, furniture or other objects are arranged between the floor and the camera. This further improves the chances of obtaining an accurate estimate of the true floor of, for example, a room.

[0046] An envisioned use scenario is, for example, where multiple fisheye cameras are installed to cover a larger room or, for example, a corridor and manually configuring each camera would be very time consuming. By using the envisioned method, the cameras can be installed and then each camera is allowed to learn which parts of its image correspond to, for example, the floor and the walls. Each camera can then generate its own 3D model of the part of the scene it is responsible for capturing based on this and then use this 3D model to project the image it captures in order to perform distortion correction.

[0047] It should also be noted that the floor, as contemplated herein, does not necessarily need to be square or rectangular, but may have other shapes as well. For example, one or more walls of a room may be curved. Similarly, a wall does not necessarily need to intersect another wall at a 90 degree angle. The only assumption made is that the walls extend vertically from the floor.

[0048] Now we will also refer to Figure 2A and Figure 2B Further describing the envisioned method (such as previously referenced Figures 1A to 1H method as shown). Figure 2A Various embodiments of the method 200 for distortion correction contemplated herein are schematically illustrated using various functional units, and Figure 2B The flow of this method is schematically illustrated. Figure 2A It is also used to illustrate a proposed apparatus 300 for performing the method, which will be referred to later in this document. FIG. 3A to FIG. 3C This is described in more detail.

[0049] In the method 200, a camera 210 is arranged to capture a scene (e.g., a room). The camera 210 has a non-rectilinear lens, such as, for example, a fisheye lens, so that the field of view of the camera 210 is large enough for the camera 210 to capture a wider portion of the scene. A plurality of images 212 of the scene captured by the camera 210 are provided to a first determination module 220.

[0050] In step S201, the first determination module 220 receives a plurality of images 212 of the scene from the camera 210. The camera 210 may send a plurality of images at a time, or, for example, send one image at a time (e.g., once each image is captured). The first determination module 220 is configured to estimate / determine (in step S202) the position of the bottom of one or more objects that have moved across the horizontal surface in the scene. This may be performed, for example, by using key points and / or by directly identifying the bottom of one or more objects. For example, the first determination module 220 may be configured, for example, to identify a person's feet, an animal's feet, and / or, for example, a wheel of a robot vacuum cleaner or other objects that often move across the floor of the scene. Such identification of the bottom may be performed, for example, using general techniques for this purpose, and may be based, for example, on various machine learning algorithms trained to identify such features. Once the position of the bottom object in the scene is determined, the first determination module 220 sends the estimated position 222 to the second determination module 230. The first determination module 220 may also wait until it has determined multiple positions of the bottom, and then send these multiple positions to the second determination module 230 simultaneously.

[0051] The second determination module 230 receives multiple positions 222 of the bottom from the first determination module 220 in order to collect statistics on how the object moves in the scene over time, and specifically how the bottom of the object moves in the scene over time. Specifically, the second determination module 230 can focus on where the object moves, that is, which part of the room the respective bottom of the object visits. In step S203, the second determination module 230 uses such statistics to determine / estimate the boundary of the floor of the room. Once the boundary of the floor of the room is determined, the second determination module 230 sends the determined boundary 232 to the 3D modeling module 240. In some embodiments, before considering a specific area as the floor of the room, the second determination 230 module can wait until the number of bottoms moving in the specific area of ​​the image exceeds a threshold. As mentioned earlier in this article, this can increase the confidence that the specific area is actually part of the floor. As contemplated herein, "multiple bottoms" do not necessarily refer to the bottoms of different objects, but can also be one or more bottoms of the same object that are repeatedly identified as being located in the specific area. More generally, it is not necessarily necessary to distinguish between different objects, and the envisaged method can therefore also be implemented using methods that are only able to identify the bottom (but not to which specific object the bottom belongs).

[0052] The 3D modeling module 240 receives the determined boundary 232 from the second determination module 230 and is configured to generate (in step S204) a three-dimensional model of the room captured by the camera 210 by defining one or more walls around the determined boundary of the floor. Once the three-dimensional model is generated, the 3D modeling module 240 sends the three-dimensional model 242 to the correction module 250. The three-dimensional model can be defined, for example, by a plurality of coordinates representing the boundary and a plurality of coordinates representing the walls. In other embodiments, if it is assumed that the walls have a height according to a height value and that they extend vertically from the floor and along the boundary of the floor defined by a polygon, the floor can be represented, for example, as a polygon and the walls can be represented, for example, as only a height value h. Of course, other ways of representing / parameterizing the three-dimensional model can be envisioned.

[0053] The correction module 250 receives the three-dimensional model 242 from the 3D modeling module 240 and (in step S205) performs correction of the distortion of the one or more images 214 captured by the camera 210 by projecting one or more images onto the three-dimensional model received from the 3D modeling module 240. The one or more images (one or more images capturing the same room / scene) on which the correction module 250 applies the correction may, for example, be received directly from the camera 210 as the one or more images 214. In other embodiments, the one or more images to be subjected to distortion correction may, for example, be sent from any other module that has such images. After the distortion of at least one of the one or more images 242 has been corrected, the correction module 250 may, for example, output a corrected image 252. The corrected image 252 may, for example, be included as part of a video stream and may, for example, replace a corresponding uncorrected image or be provided in addition to the uncorrected image. The distortion-corrected image may not necessarily be the same image as one of the images used to generate the three-dimensional model, but may, for example, be an image of the scene captured at a later time than the plurality of images used to generate the three-dimensional model.

[0054] Now refer to FIG. 3A to FIG. 3C An apparatus for performing distortion correction on images captured using a camera with a non-rectilinear lens is described in more detail.

[0055] Figure 3AComponents of various embodiments of the apparatus 300 contemplated herein are schematically illustrated in the form of a number of functional units / modules / blocks. The apparatus 300 comprises a processing circuit 310. The processing circuit 310 is provided using any combination of one or more of a suitable central processing unit (CPU), a multiprocessor, a microcontroller, a digital signal processor (DSP), etc., and is capable of executing software instructions stored in a computer program product (not shown herein), for example, in the form of a computer-readable storage medium 320. The processing circuit 610 may further be provided as at least one application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0056] Specifically, the processing circuit 310 is configured to cause the apparatus 300 to perform operations such as described above. Figure 2A and Figure 2B The method 200 illustrated in FIG. 200 is a set of operations or steps disclosed in one or more of steps S201 to S205. For example, the storage medium 320 may store a set of operations, and the processing circuit 310 may be configured to obtain the set of operations from the storage medium 320 so that the apparatus 300 performs the set of operations. The set of operations may be provided as a set of executable instructions. Therefore, the processing circuit 310 is thereby arranged to perform the operations disclosed herein, for example, with reference to FIG. Figure 2A and 2B method.

[0057] The storage medium 320 may also include a persistent storage, which may be, for example, any single or combined form of magnetic storage, optical storage, solid-state storage, or even remotely mounted storage.

[0058] The device 300 may further include a communication interface 330 for communicating with other entities, functions, nodes, and devices, such as, for example, the camera 210. For example, the communication interface 330 may allow the device 300 to communicate with the camera 210 in order to receive a plurality of images of a scene for generating a three-dimensional model of the scene. Thus, the communication interface 330 may include one or more transmitters and receivers including analog and / or digital components. As will be described in more detail later herein, in some embodiments, the device 300 may be the camera 210 itself, and the communication interface 330 may, for example, include any necessary circuitry to, for example, allow the processing circuitry 310 of the device / camera 300 to access image data generated by an image sensor (not shown) of the device / camera.

[0059] The processing circuit 310 controls the overall operation of the device 300, for example, by sending data and control signals to the communication interface 330 and the storage medium / memory 320, by receiving data and reports from the communication interface 330, and by obtaining data and instructions from the storage medium 320. Other components of the device 300 and their related functions (as illustrated by the dashed box 340) can of course also be included, but any description thereof is omitted so as not to obscure the concepts presented herein. A communication bus 350 is included and is configured to allow the various units 310, 320 and 330 (and optionally also 340) to exchange data and information with each other as needed.

[0060] Figure 3B The components of the apparatus 300 according to various embodiments of the present disclosure are schematically illustrated in the form of a number of functional modules 301 to 305. The apparatus 300 comprises at least a Figure 2A and Figure 2B The described method 200 includes a first determination module 301 (such as, for example, the first determination module 220) of step S202, a second determination module 302 (such as, for example, the second determination module 230) configured to execute step S203, a 3D modeling module 303 (such as, for example, the 3D modeling module 240) configured to execute step S204, and a distortion correction module 304 (such as, for example, the correction module 250) configured to execute step S205.

[0061] In general, each functional module 301 to 304 can be implemented in hardware or software. Preferably, one or more or all functional modules 301 to 304 can be implemented by processing circuit 310, which can cooperate with communication interface 330 and / or storage medium 320. Therefore, processing circuit 310 can be arranged to obtain instructions as provided by functional modules 301 to 304 from storage medium 320, and execute these instructions, and thereby perform any steps of method 200 performed by device 300 as disclosed herein. If provided as hardware, each module 301 to 304 can be separated from other modules. In other embodiments, one, more or all of modules 310 to 304 can be implemented as the same physical module or parts of similar modules.

[0062] In some embodiments, the apparatus 300 may further include additional functional modules (illustrated by dashed box 305) required to perform other tasks of the apparatus 300, for example as defined in the appended dependent claims. A communication bus 352 (logical or physical) is provided to allow the various functional modules 301 to 304 (and optionally 305) to communicate / exchange data and information as required.

[0063] Figure 3C An embodiment of a camera 300 as contemplated herein is schematically illustrated, namely with reference to Figure 3A and Figure 3B The described apparatus 300 is actually the case of a camera 300 for capturing multiple images of a scene. Figure 3A As shown in , in some embodiments, the camera 300 may be a fisheye camera having a fisheye lens 370. The camera 300 has an optical axis 372, so that when an image of a scene is captured, a point of the scene aligned with the optical axis 372 preferably corresponds to a pixel in the center of the captured image. Figure 3C As shown, the camera 300 includes, for example, processing circuitry 310 as described above, and may also include, for example, a memory (not shown) 320, etc. The camera 300 is configured to be mounted, for example, in a ceiling of a room, such that the optical axis 372 is orthogonal to the floor of the room, for example.

[0064] Although not illustrated in any of the figures herein, the present disclosure also provides computer programs and computer program products as have been described herein. The computer program product comprises a computer-readable storage medium on which the envisaged computer program is stored. As has been described, the computer program comprises computer code which, when executed on a processing circuit (such as 310) of an apparatus (such as 300), executes, for example, a reference to Figure 2A and Figure 2B Steps S201 to S205 of the described method 200 enable the apparatus to perform any method disclosed and discussed herein. Thus, the computer program and / or computer program product may therefore provide means for performing, for example, any steps of the method 200 and / or enabling the envisaged apparatus 300 to operate as intended.

[0065] The computer program product may for example be provided as an optical disc such as a CD (Compact Disc) or a DVD (Digital Versatile Disc) or a Blu-ray Disc. The computer program product may also be embodied as a memory such as a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM) or an Electrically Erasable Programmable Read Only Memory (EEPROM), and more specifically as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory, or a flash memory such as a Compact Flash. Thus, the computer program may be stored in any manner suitable for a computer program product, i.e. on a computer readable storage medium.

[0066] Summarizing the various embodiments presented herein, the present disclosure provides an improved way of handling distortion correction in images captured by a camera with a non-rectilinear lens. Specifically, the present disclosure provides a way to self-learn such correction by observing how or specifically where the bottom of an object (such as a person's foot) moves in an image of a scene, so as to determine which parts of the image of the scene correspond to, for example, the floor of a room, and which other parts of the image correspond to, for example, the walls of the room, thereby generating a three-dimensional model that can be used to perform the correction. By allowing, for example, a camera to learn how to generate such a three-dimensional model over time, distortion in images captured by the camera can be corrected without requiring, for example, manual input from an operator, thereby facilitating, for example, the process of installing multiple such cameras.

[0067] Although features and elements may be described above in specific combinations, each feature or element may be used alone without the other features and elements, or in various combinations with or without the other features and elements. In addition, variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention based on a study of the drawings, the present disclosure, and the appended claims.

[0068] In the claims, the words "comprising" and "including" do not exclude other elements, and "a" or "an" does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be used to advantage.

[0069] Reference numerals list

[0070] 100 images of scenes

[0071] 101; 212 Multiple images of the scene

[0072] 102, 103; 242 3D model of the scene

[0073] 104;252 Distortion-corrected image of the scene

[0074] 110; 171 Horizontal surface / floor

[0075] 120a to 120d; 172a to 172d Vertical surface / wall

[0076] 130a to 130d; 174a to 174d corner

[0077] 140 doors

[0078] 150;222 The position of the bottom of the object in the scene over time

[0079] 160 Bottom border

[0080] 170; 232 Boundaries of horizontal surfaces

[0081] 200 Distortion Correction Method

[0082] S201 to S205 Method steps

[0083] 210 Camera with non-rectilinear lens

[0084] 212 Multiple images of the scene

[0085] 220; 301 First determination module

[0086] 230; 302 Second determination module

[0087] 240; 303 3D modeling module

[0088] 240; 304 Correction module

[0089] 300 Devices / Cameras

[0090] 305; 340 Optional function modules

[0091] 310 Processing Circuit

[0092] 320 Memory

[0093] 330 Communication Interface

[0094] 350; 352 Communication bus

[0095] 370 fisheye lens

[0096] 372 Optical Axis

Claims

1. A method for distortion correction in an image of a room captured from above by a camera having a non-rectilinear lens, comprising: i) acquiring a plurality of images of the room captured by the camera from above over time; ii) determining the positions of one or more bases of one or more objects in at least some of the plurality of images; iii) determining the position of the boundary of the floor of the room in the plurality of images based on the determined position of the bottom, wherein the area is considered to belong to the floor of the room only if the number of bottoms that move in the area in at least some of the plurality of images exceeds a threshold value during a limited time interval; iv) generating a three-dimensional model of the room by defining one or more walls around the determined boundary of the floor, wherein the walls extend vertically away from the plane of the boundary of the floor, and v) generating a distortion-corrected image of the room by projecting at least one of the plurality of images onto the three-dimensional model of the room.

2. The method according to claim 1, wherein: The non-linear lens is a fisheye lens.

3. The method according to claim 1, wherein: The bottom portion includes the foot of the object.

4. The method according to claim 1, wherein: The step ii) comprises determining the position of the bottom of the one or more objects that have moved across the floor of the room in the at least some of the plurality of images by identifying one or more key points of the one or more objects.

5. An apparatus for performing distortion correction in an image of a room captured by a camera having a non-rectilinear lens, the apparatus comprising a processing circuit configured to cause the apparatus to perform the method according to any one of claims 1 to 4.

6. The device according to claim 5, wherein: The device is the camera having the non-rectilinear lens.

7. A computer program product for performing distortion correction in an image of a room captured by a camera having a non-rectilinear lens, wherein: The computer program product comprises computer code which, when executed on a processing circuit of a device, causes the device to perform a method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, which, when the computer program is executed on a processing circuit of a device, causes the device to execute the method according to any one of claims 1 to 4.

Images