IMPROVED OBJECT DETECTION USING STRUCTURED LIGHT

Abstract

Object detection system (10, 50) for detecting objects in an area of ​​interest which has the following features: a camera (12, 52) configured to map an area of ​​interest in order to provide an image; a structured light source (14, 54) configured to provide a first essentially two-dimensional light curtain and a second essentially two-dimensional light curtain, which is essentially perpendicular to the first essentially two-dimensional light curtain, in a field of view (18) of the camera (12, 52), namely such that an intersection point of the first essentially two-dimensional light curtain with the second essentially two-dimensional light curtain represents a known reference point within the image, wherein each of the first essentially two-dimensional light curtain and the second essentially two-dimensional light curtain is formed by at least one projected laser beam having a wavelength that can be detected by the camera (12); and an image analysis component (20, 60) configured to detect objects in the area of ​​interest from at least one position of the at least one projected laser beam in the image and from a position of the intersection point in the image, the reference point being used to convert the image into world coordinates.

Description

Technical field

[0001] The invention relates to vehicle safety systems and in particular to the use of structured light to improve an object detection system. Background of the invention

[0002] Single-lens camera systems are limited in their ability to represent three-dimensional data within the camera's two-dimensional focal plane arrangement. The camera's image of the world appears flat, lacking any depth. Algorithmic techniques for recovering this lost depth are application-dependent and work only within extremely narrow limits—they are not generally robust or repeatable. For example, a structure of motion and temporal stereo fitting or adjustment can be used to reconstruct an image with depth from a single-lens camera with a difference position. The camera must move with coherence between different frames, and these movements must be tracked in real-world coordinates, typically using an inertial measurement unit.Other examples, such as chromatic reflection, Gouraud shading, and flat-earth models, require controlled lighting of the scene and a stationary camera position.

[0003] In this context, US 2007 / 0019181A1 discloses an object detection system that uses one or more thin, flat, structured light patterns projected into a volume of interest, together with digital processing hardware and one or more electronic imagers for examining said volume. Triangulation is used to determine the intersection of the structured light pattern with objects in the volume of interest. Potential applications include navigation and object avoidance systems for autonomous vehicles, security systems, and pet training systems.

[0004] US 2013 / 0338831 A1 discloses the use of vertical and horizontal light curtains in combination in a robotic vacuum, wherein the horizontal pattern is used to detect obstacles, and wherein the vertical pattern is used to detect a vertical free space for the vacuum.

[0005] DE 10 2007 054 906 A1 teaches a method for the optical measurement of the three-dimensional geometry of an object. In this method, a first pattern of parallel stripes is projected onto the object and a first image is captured. The pattern is then changed and a second image is captured. The data from the two images are then combined to obtain a three-dimensional representation of the object, using at least one matching point for the combination.

[0006] US 2009 / 0040532 A1 discloses a method for three-dimensional measurement of an object which uses a laser to determine a depth of points on an imaged surface. Summary of the invention

[0007] The object of the present invention is achieved by an object detection system according to claims 1 and 16, and by a method for detecting objects according to claim 11. The dependent claims relate to preferred embodiments of the invention. Brief description of the drawings Fig. Figure 1 illustrates an example of an object detection system for detecting objects within an area of ​​interest; Fig. Figure 2 illustrates an example of a system for detecting objects within a blind spot or an area behind the vehicle; Fig. Figure 3 illustrates a method for detecting objects within an area of ​​interest; and Fig. Figure 4 is a schematic block diagram illustrating an exemplary system of hardware components that can implement examples of the systems and procedures described here. Detailed description

[0008] The inventor has discovered that a single-lens camera, or a monocular camera, can be enhanced with structured light to improve its ability to detect and locate objects within its field of view. Specifically, one or more light curtains—that is, a continuous or intermittent two-dimensional shape formed by one or more projected laser beams—can be projected into the camera's field of view. Deflection or deformation of these light curtains can be detected and quantified to determine the presence and location of an object within the field of view. In one embodiment, multiple shapes can be used to define different areas through which an object can extend.In another embodiment, two or more non-parallel beams are used to create a three-dimensional shape within the field of view, allowing for a more precise determination of the object's position.

[0009] Fig. Figure 1 illustrates an example of an object detection system 10 for detecting objects within an area of ​​interest. The system 10 comprises a camera 12 configured to capture an image of the area of ​​interest and a structured light source 14 configured to project a two-dimensional shape 16 into a field of view 18 of the camera. This shape is formed by at least one projected laser beam with a wavelength detectable by the camera. In one embodiment, the camera is selected to capture an image within the visible band, and the laser operates at approximately 532 nm.

[0010] An image analysis component 20 is configured to detect objects within the area of ​​interest from at least one position of the at least one projected laser beam in the image, or more precisely, the location of a beam-ground intersection. In one implementation, this detection can be performed by transforming the camera image into a world coordinate domain and checking the laser's intersection line with the respective objects to determine if any objects are present. In one implementation, the intersection of two non-parallel lines can be used as a known reference point during the transformation. Alternatively, multiple parallel beams can be used, representing different distances from the vehicle, with the specific beams exhibiting a deflection that indicates the object's distance from the vehicle.

[0011] Fig. Figure 2 illustrates an example of a system 50 for detecting objects within a blind spot or area behind the vehicle, referred to here as the area of ​​interest. The system 50 includes a camera 52 configured to capture an image of the area of ​​interest. In the illustrated embodiment, the camera 52 is configured to detect incident light in the visible and near-infrared ranges. The camera 52 is positioned within the vehicle to effectively capture an image of the area of ​​interest. It will be clear that the camera 52 can be connected to the other components of the system 54 and 60 via a vehicle bus 53.

[0012] A structured light source 54 is configured to provide an essentially two-dimensional light curtain that travels in the region of interest. In the illustrated embodiment, the structured light source 54 comprises a laser diode 56 configured to provide at least one laser beam with a wavelength detectable by the camera 52, and an optical beam-shaping component 58 configured to generate the essentially two-dimensional light curtain from the at least one laser beam. It will be clear that the laser diode 56 can provide light in any of the visible, infrared, and ultraviolet regions. The optical beam-shaping component 58 can comprise a diffracting diffuser, a diffracting beam-shaping element, or any other suitable optical component.In the illustrated embodiment, the optical beam shaping component 58 can have a fixed diffracting beam shaping optic that generates the light curtain over a range of 46 degrees. Accordingly, it will be clear that the illustrated embodiment functions without any moving components in the optics.

[0013] The structured light source 54 can be mounted at a suitable location on the vehicle to project the light curtain into the area of ​​interest. While the inventor has found that co-arranging the structured light source 54 and the camera 52 can lead to less than optimal results, the positions of the structured light source 54 and the camera 52 can be selected differently to maximize the usefulness of the captured area of ​​interest for a given vehicle geometry. In one embodiment, both the structured light source 54 and the camera 52 can be mounted on an upper part of the vehicle and separated by a horizontal baseline. In another embodiment, the structured light source 54 and the camera 52 can be separated by a small vertical baseline.

[0014] It should further be noted that the structured light source 54 can be configured to provide additional light curtains in the area of ​​interest. For this purpose, the optical beam shaping component 58 can be configured to provide a multitude of parallel light curtains. For example, each light curtain can be projected to intersect the ground at lines that are substantially parallel to one side of the vehicle on which the structured light source is mounted, with each light curtain intersecting the ground at a specified distance from the vehicle, assuming a sufficiently flat ground at the point of intersection.Alternatively or additionally, the structured light source 54 can be configured to provide a light curtain that is substantially perpendicular to one or more other light curtains, such that the light curtain intersects the floor in a line substantially perpendicular to the line or lines of the other light curtains. It will be clear that the additional light curtains can be provided by additional structured light sources (not shown), rather than by providing multiple light curtains from a single component, as in [reference]. Fig. 2 illustrated.

[0015] An image captured by camera 52 is delivered to an image analysis component 60, which is configured to determine whether an object is present in the area of ​​interest. It will be clear that the image analysis component 60 can be implemented as software running on a microprocessor in the vehicle, as dedicated hardware or a component, or as a combination of software and dedicated hardware. Specifically, the image analysis component 60 determines from the captured image whether one or more objects are present in the path of the light curtain. The image analysis component 60 can be configured to align the image before analysis if no alignment is performed on the image at camera 52 or in a related component (not shown), and other image processing can be performed before analysis.

[0016] It will be clear that, in the absence of an object, the light curtains will form relatively straight lines upon contact with the ground. However, in the presence of an object within the light curtain's area, the intersection of the light curtain with the object will have a height relative to the original baseline. In one design, several parallel light curtains can be used to define different regions near the vehicle. In this design, the distance of the object from the vehicle can be determined, according to which the light curtain(s) will exhibit a deviation from the baseline.

[0017] In the illustrated embodiment, two perpendicular light curtains can be used such that their intersection represents a known reference point in the image. Using this reference point, the location and height of the object can be determined from the position, magnitude, and width of the deviation from the baseline. For this purpose, the image analysis system 60 includes a coordinate transformation component 62 configured to transform the location of each pixel within the image into a real-world coordinate system around the vehicle. In particular, for a pixel (x, y) within the image, where x is a column in which the pixel is located and y is a row in which the pixel is located, a position (X, Z) is determined as follows, where X represents a distance across the camera and Z represents a distance down the camera: X = Hxfxsin(ϕ)−y cos(ϕ), where f x a horizontal focal length of the camera is, f y where Φ is the vertical focal length of the camera, Φ is the camera tilt angle, and H is the height of the camera.

[0018] Where objects have been detected, a height calculation component 64 can determine the object's height from the image and its specific location. In particular, a height Y at a given location can be determined from its distance Z, the downward range of the camera, and a row y in which the pixel representing that location is situated, as follows: Y=Z[y cos(ϕ)−fysin(ϕ)]y sin(ϕ)+fy cos(ϕ)+H where f y where Φ is the vertical focal length of the camera, Φ is the camera tilt angle, and H is the height of the camera.

[0019] Once the object's position and height have been determined, this information can be provided to the user via audible, visual, or tactile feedback. In one implementation, the object can be highlighted in a rear-view or side-view display shown to the operator on a screen inside the vehicle.

[0020] In view of the preceding structural and functional features described above, a method according to various aspects of the present invention is better described by reference to Fig. 3. become clear. While, for the purpose of simplifying the declaration, the procedure of Fig. It should be noted, and it should be clear, that the present invention is not limited to the illustrated sequence, since some aspects of the present invention could occur in other sequences and / or simultaneously with other aspects than those shown and described here. Furthermore, not all illustrated features may be necessary to implement a method according to the present invention.

[0021] Fig. Figure 3 illustrates a method 100 for detecting objects within an area of ​​interest. In Figure 102, an essentially two-dimensional light curtain is projected into an area of ​​interest. The area of ​​interest can be selected, for example, a blind spot or an area behind the vehicle. It will be clear that the light curtain can be formed from visible, infrared, or ultraviolet light. In one embodiment, multiple light curtains can be projected into the area of ​​interest. For example, the additional light curtains can be parallel or perpendicular to the first light curtain, depending on the application.

[0022] At step 104, an area of ​​interest is captured by the camera to provide an image such that the intersection of the essentially two-dimensional light curtain with the floor is visible. It will be clear that the camera may be equipped with suitable filters to detect the light associated with the light curtain. At step 106, it is determined whether the intersection of the essentially two-dimensional light curtain with the floor deviates from an expected baseline. If so (YES), at step 108, it is reported that an object has been detected, and the procedure returns to step 102 to continue an evaluation of the area of ​​interest. If not (NO), the procedure returns to step 102 to continue the evaluation of the area of ​​interest.

[0023] In one implementation, the image can be used to determine additional information about an object detected in the area of ​​interest. In one implementation, multiple parallel light curtains can be used, and it can be determined that the object is at a first distance if a line formed by the intersection of a first essentially two-dimensional light curtain and the floor deviates from a first expected baseline, or that it is at a second distance if a line formed by the intersection of a second essentially two-dimensional light curtain and the floor deviates from a second expected baseline. It will be clear that more than two parallel curtains can be used in this way to increase the resolution of the distance determination.Alternatively, vertical light curtains can be projected, and the image can be subjected to a coordinate transformation to determine the position and height of any detected objects, as described, for example, in equations 1 and 2 above.

[0024] Fig.Figure 4 is a schematic block diagram illustrating an exemplary control system 200 of hardware components that can implement examples of the systems and methods disclosed herein, such as the object detection system described above. The system 200 can comprise various systems and subsystems. It can be implemented using a vehicle-based control device connected via a vehicle bus, or using an image analysis control device. The computer could include a microprocessor, or the desired control functions could be achieved using an application-specific integrated circuit (ASIC) arranged accordingly.

[0025] The control system 200 can include a system bus 202, a processing unit 204, a system memory 206, storage devices 208 and 210, a communication interface 212 (for example, a network interface), a communication link 214, a display 216, and input devices 218 (for example, accelerometers, camera inputs, etc.). The system bus 202 can be connected to the processing unit 204 and the system memory 206. The additional storage devices 208 and 210 can also communicate with the system bus 202. The system bus 202 connects the processing unit 204, the storage devices 206–210, the communication interface 212, the display 216, and the input devices 218. In some examples, the system bus 202 also connects an additional (not shown) port.

[0026] The processing unit 204 can be a microprocessor and / or can include an application-specific integrated circuit (ASIC) arranged to perform the desired processing and control functions. The processing unit 204 executes a set of instructions to implement the operations of examples disclosed herein.

[0027] The additional storage devices 206, 208, and 210 can store data, programs, instructions, database queries in text or compiled form, and any other information that may be necessary to operate a computer. The storage devices 206, 208, and 210 can be implemented as computer-readable media.

[0028] Additionally or alternatively, the system 200 can access an external data source or query source through the communication interface 212, which can communicate with the system bus 202 and the communication link 214.

[0029] During operation, the processing unit 204 executes one or more computer-executable instructions, which originate from the system memory 206 and the storage devices 208 and 210. The term "computer-readable medium," as used here, refers to a medium that participates in supplying instructions to the processing unit 204 for execution.

[0030] The above descriptions are examples of the present invention. It is naturally not possible to describe every conceivable combination of components or methods for the purpose of describing the present invention, but the person skilled in the art will recognize that many further combinations and permutations of this invention are possible. Accordingly, the present invention is intended to encompass all such changes, modifications, and variations that fall within the scope of the appended claims.

Claims

[1] Object detection system (10, 50) for detecting objects in an area of ​​interest, which has the following features: a camera (12, 52) configured to map an area of ​​interest in order to provide an image; a structured light source (14, 54) configured to provide a first essentially two-dimensional light curtain and a second essentially two-dimensional light curtain, which is essentially perpendicular to the first essentially two-dimensional light curtain, in a field of view (18) of the camera (12, 52), namely such that an intersection point of the first essentially two-dimensional light curtain with the second essentially two-dimensional light curtain represents a known reference point within the image, wherein each of the first essentially two-dimensional light curtain and the second essentially two-dimensional light curtain is formed by at least one projected laser beam having a wavelength that can be detected by the camera (12); and an image analysis component (20, 60) configured to detect objects in the area of ​​interest from at least one position of the at least one projected laser beam in the image and from a position of the intersection point in the image, the reference point being used to convert the image into world coordinates. [2] Object detection system (10, 50) according to claim 1, wherein the structured light source (54) comprises a laser diode (56) and an optical beam shaping component (58) configured to generate the substantially two-dimensional light curtain. [3] Object detection system (10, 50) according to claim 2, wherein the laser diode (56) is configured to generate light in the infrared band. [4] Object detection system (10, 50) according to claim 1, wherein the camera (12, 52) is mounted on a vehicle at a first location, and wherein the structured light source (14, 54) is mounted on the vehicle at a second location which is spatially remote from the first location. [5] Object detection system (10, 50) according to claim 1, wherein the image analysis component (20, 60) is configured to evaluate a section of the substantially two-dimensional light curtain with the floor in the image, wherein the presence of an object is indicated by a deviation of a line formed by the section from an expected baseline. [6] Object detection system (10, 50) according to claim 1, wherein the structured light source (14, 54) is further configured to provide a third substantially two-dimensional light curtain within the field of view of the camera (12, 52), wherein the third substantially two-dimensional light curtain is substantially parallel to the first substantially two-dimensional light curtain. [7] Object detection system (10, 50) according to claim 6, wherein the image analysis component (20, 60) is configured to evaluate respective sections of both the first substantially two-dimensional light curtain and the third substantially two-dimensional light curtain with the ground in the image, wherein the presence of an object at a first distance is indicated by a deviation of a line formed by the section of the first substantially two-dimensional light curtain and the ground from a first expected baseline, and wherein the presence of an object at a second distance is indicated by a deviation of a line formed by the section of the third substantially two-dimensional light curtain and the ground from a second expected baseline. [8] Object detection system (10, 50) according to claim 1, wherein the source (14, 54) for structured light is configured to have a distribution or propagation angle between 40 degrees and 50 degrees. [9] Object detection system (10, 50) according to claim 1, wherein the image analysis component (60) has a coordinate transformation component (62) configured to perform a coordinate transformation on the image such that for each pixel (x, y) within the image, where x is a column in which the pixel is located and y is a row in which the pixel is located, a position (X, Z) is determined as follows, where X represents a distance transverse region of the camera (12, 52) and Z represents a distance downward region of the camera (12, 52): and where f x a horizontal focal length of the camera (12, 52) is, where f ya vertical focal length of the camera (12, 52) where Φ is a camera tilt angle, and where H is a camera height. [10] Object detection system (10, 50) according to claim 1, wherein the object detection system (10, 50) has a height calculation component configured to determine a height Y at a given location downwards from its distance Z from the camera (12, 52) and a row y in which the pixel representing the location is arranged, as Y=Z[y cos(ϕ)−fysin(ϕ)]y sin(ϕ)+fy cos(ϕ)+H where f y a vertical focal length of the camera (12, 52) where Φ is a camera tilt angle and where H is a height of the camera (12, 52). [11] Method (100) for detecting objects within an area of ​​interest which has the following features: Projecting (102) a first essentially two-dimensional light curtain into an area of ​​interest; Projecting a second essentially two-dimensional light curtain into the viewing area (18) of a camera (12, 52) which is essentially perpendicular to the first essentially two-dimensional light curtain, such that an intersection point of the first essentially two-dimensional light curtain with the second essentially two-dimensional light curtain represents a known reference point within the image; Taking (104) an image of the area of ​​interest with a camera (12, 52) to provide an image such that a section of the essentially two-dimensional light curtain with the floor is visible, wherein the image is converted into world coordinates, and wherein the reference point is used to convert the image into world coordinates; and Determine (106) that an object exists by a deviation of a section of the essentially two-dimensional light curtain with the floor from an expected baseline and a position of the intersection point in the image. [12] The method of claim 11, which further comprises projecting a third substantially two-dimensional light curtain into the area of ​​interest, wherein the third substantially two-dimensional light curtain is substantially parallel to the first substantially two-dimensional light curtain. [13] Method according to claim 12, wherein determining (106) that the object is present comprises determining the presence of an object at a first distance if a line formed by the intersection of the first substantially two-dimensional light curtain and the floor deviates from a first expected baseline, and determining the presence of an object at a second distance if a line formed by the intersection of the third substantially two-dimensional light curtain and the floor deviates from a second expected baseline. [14] The method of claim 11, which further comprises applying a coordinate transformation to the image such that for each pixel (x, y) within the image, wherein x represents a column in which the pixel is arranged and y represents a row in which the pixel is arranged, a position (X, Z) is determined as follows, wherein X represents a transverse distance range of the camera (12, 52), and wherein Z represents a downward distance range of the camera (12, 52), and where f x a horizontal focal length of the camera (12, 52) is, where f y a vertical focal length of the camera (12, 52) is, where Φ is a camera tilt angle, and where H is a height of the camera (12, 52). [15] The method of claim 14, which further comprises determining a height Y at a given location from its distance Z downwards from the camera (12, 52), and a row y in which the pixel representing the location is situated, as Y=Z[y cos(ϕ)−fysin(ϕ)]y sin(ϕ)+fy cos(ϕ)+H where f y a vertical focal length of the camera (12, 52) is, where Φ is a camera tilt angle, and where H is a height of the camera (12, 52). [16] Object detection system (50) for detecting objects in a blind spot or an area behind the vehicle, which has the following features: a camera (52) configured to take an image of the blind spot or the area behind the vehicle; a structured light source (54) comprising a laser diode (56) and an optical beam shaping component (58) configured to provide a first substantially two-dimensional light curtain and a second substantially two-dimensional light curtain, which is substantially perpendicular to the first substantially two-dimensional light curtain, in a field of view of the camera (52), namely such that an intersection point of the first essentially two-dimensional light curtain with the second essentially two-dimensional light curtain represents a known reference point within the image, wherein each of the first essentially two-dimensional light curtain and the second essentially two-dimensional light curtain is formed by at least one projected laser beam having a wavelength that can be detected by the camera (52); and an image analysis component (60) configured to detect objects within the blind spot or the area behind the vehicle from at least one position of the at least one projected laser beam in the image and a position of the intersection point in the image by applying a coordinate transformation to the image such that for each pixel (x, y) in the image, where x represents a column in which the pixel is located and where y represents a row in which the pixel is located, a position (X,Y) where X represents a lateral distance range of the camera (52), and where Z represents a downward distance range of the camera (52), is determined as and where f x a horizontal focal length of the camera (52) is, where f ya vertical focal length of the camera (52), where Φ is a camera tilt angle, and where H is a height of the camera (52). [17] Object detection system (50) according to claim 16, wherein the camera (52) is mounted on a vehicle at a first location and wherein the structured light source (54) is mounted on the vehicle at a second location spatially remote from the first location. [18] Object detection system (50) according to claim 16, wherein the image analysis component (60) is configured to evaluate a section of the substantially two-dimensional light curtain with the floor in the image, wherein the presence of an object is indicated by a deviation of a line formed by the section from an expected baseline.

Images