Environmental detection system and vehicle

Abstract

This invention relates to an environmental detection system and a vehicle. The environmental detection system includes a first camera, a second camera, a light source, and an analysis and processing unit. The fields of view of the first camera and the second camera have a predefined overlapping area. The light source is configured to project at least one predefined pattern onto the overlapping area of ​​the respective fields of view of the first camera and the second camera. The analysis and processing unit is configured to identify at least one predefined pattern in a first image captured by the first camera and in a second image captured by the second camera; based on the patterns identified in the first image and the second image, respectively, to determine a first stitching line in the first image and a second stitching line in the second image corresponding to the first stitching line; and to stitch the first image and the second image together into a complete image based on the first stitching line and the second stitching line, wherein there is a substantially seamless transition between the first image and the second image in the complete image.

Description

Technical Field

[0001] The present invention relates to an environmental monitoring system and a vehicle having such an environmental monitoring system. Background Technology

[0002] Environmental detection systems for vehicles are known from the prior art. These systems are typically set up based on multiple sensors (e.g., ultrasonic sensors, lidar sensors, radar sensors, cameras, etc.) to detect the vehicle's environment.

[0003] Furthermore, panoramic vision systems for vehicles are known, which are configured based on images from multiple cameras to create an overall image from the individual images by stitching them together in a suitable manner and as seamlessly as possible. In this way, for example, displaying the vehicle along with a top-down view of its surroundings represented by such an overall image on the vehicle's display, parking processes, in particular, can be simplified or performed with enhanced safety based on the overall image. Summary of the Invention

[0004] According to a first aspect of the invention, an environmental monitoring system is provided, particularly an environmental monitoring system for vehicles, wherein the vehicle may be configured as, for example, a passenger car, a truck, a bus, a transport vehicle, a two-wheeled vehicle, or a rail vehicle.

[0005] The environmental monitoring system has at least one first camera and a second camera, which are configured, for example, as monocular and / or stereo and / or black and white and / or RGB cameras, or as different types of cameras. Furthermore, the first camera and the second camera can be configured as the same or different cameras. Discrepancies between the two cameras may also involve, for example, different resolutions and / or sensitivities and / or the optics used respectively.

[0006] In addition, the environmental monitoring system has a light source and an analysis and processing unit, wherein the analysis and processing unit may be constructed, for example, as a CPU, microcontroller, ASIC, or otherwise.

[0007] The fields of view of the first camera and the fields of view of the second camera have a predefined overlapping area. Such an overlapping area is generated, for example, by mounting the first camera in the front region of the vehicle and oriented along the forward direction of travel, while the second camera is arranged, for example, in the region of the left or right rearview mirror of the vehicle and oriented partially along the forward direction of travel and partially towards the left or right side of the vehicle, such that the field of view of the second camera has an overlapping area with the field of view of the first camera due to its partial orientation along the forward direction of travel.

[0008] The light source is configured to project at least one predefined pattern onto the overlapping area of ​​the respective fields of view of the first and second cameras, such that the pattern can be detected proportionally by both cameras. Correspondingly, the first and second cameras are configured to detect the wavelength or wavelength range of light generated by the light source using corresponding photodetectors and generate signals corresponding to the detected light, which can be provided for subsequent processing. Particularly in the preferred case where the light generated by the light source is in a wavelength range outside the visible wavelength range, the first and second cameras are preferably configured to detect light in both the visible wavelength range (for optical detection of the environment) and the invisible wavelength range (for optical detection of the pattern projected by the light).

[0009] This does not explicitly exclude the possibility that light generated by a light source is at least partially within the visible wavelength range. Furthermore, it is not necessary for the first and second cameras to detect the entire visible light wavelength range.

[0010] The analysis and processing unit is configured to identify the at least one predefined pattern in a first image captured by a first camera and in a second image captured by a second camera. For this purpose, the corresponding images preferably exist in the form of digital data, which can be generated by corresponding received signals from the respective photodetectors of the first and second cameras. Pattern recognition can be achieved, for example, based on image recognition methods known from the prior art and / or based on trained neural networks and / or based on the use of related methods, etc.

[0011] The analysis and processing unit is further configured to determine, based on patterns identified in the first image and the second image respectively, a first stitching line in the first image and a second stitching line corresponding to the first stitching line in the second image. The stitching lines may have different shapes depending on their respective optical characteristics and / or arrangement positions and / or orientations and / or further influences on the first and second cameras. For example, these stitching lines may be straight lines constructed vertically or diagonally relative to the images from the first and second cameras. Furthermore, the stitching lines may each have corresponding curvatures and / or consist of multiple segments with different constructions.

[0012] Finally, the analysis and processing unit is configured to stitch the first image and the second image into a whole image based on the first stitching line and the second stitching line, in which there is a substantially seamless transition between the first image and the second image.

[0013] According to the above description, the environmental detection system according to the present invention offers a particular advantage: by using at least one pattern, the images of the first camera and the second camera can be stitched together with particularly low computational cost, because the stitching position does not need to be calculated from the pure image content of the first camera and the second camera with correspondingly high computational cost as in the prior art.

[0014] Therefore, compared with existing technologies, analysis and processing units with lower computing power and / or lower memory requirements can be used, thereby saving costs.

[0015] In particular, it also has the advantage of reducing the time required to stitch together the images from the first and second cameras, allowing for faster output of the resulting overall image to the vehicle's display and / or subsequent processing of the overall image (e.g., in systems used for partially and / or fully automated driving operations of the vehicle).

[0016] Furthermore, the at least one pattern can be used in subsequent processing to perform 3D reconstruction of the detected environment based on 2D images from the first and second cameras.

[0017] The following shows a preferred extension of the present invention.

[0018] In an advantageous configuration of the invention, the predefined pattern is a predefined first pattern, and the environmental detection system is configured to project at least one predefined second pattern onto the overlapping area of ​​the fields of view of the first and second cameras. Furthermore, the analysis processing unit is configured to additionally determine the first and second stitching lines based on the second pattern. By projecting multiple predefined patterns (i.e., two, three, four, or more) onto the overlapping area, the orientation of the two cameras relative to each other can be determined more easily and / or faster, and / or the appropriate stitching lines can be determined more accurately.

[0019] A particular advantage is that the first and second patterns are constructed as clearly distinguishable patterns. This provides, for example, the advantage of facilitating the association between patterns in the images of the respective cameras, and thereby allowing for faster determination of stitching lines, depending on the situation. Especially when interference occurs in at least one camera when detecting at least one pattern (e.g., due to environmental influences such as precipitation or fog), this method can improve the reliability of the association between corresponding patterns in the images of the respective cameras.

[0020] In a particularly preferred embodiment of the invention, the analysis and processing unit is configured to match the first and / or second images to each other using one or more geometric transformations before stitching them into a complete image, based on patterns identified in the first and second images respectively. In this way, the transition between the first and second images in the complete image can be improved.

[0021] In another configuration of the invention, the light source is a laser light source and / or an infrared light source. Alternatively or additionally, the light source is configured to emit light in the visible and / or invisible wavelength range. As a further alternative or supplement, the light source is integrated into the first camera and / or the second camera, or arranged separately from the first and second cameras. In the case of separate arrangement, the light source may be located, for example, between the first and second cameras, on the roof of the vehicle, or at a different location. Alternatively or additionally, the light source has multiple individual light sources, each configured to project at least one pattern into the overlapping area. Further alternatively or additionally, the light source is configured to project the corresponding pattern movably and / or immovably relative to the overlapping area. In the case of movable projection, for example, it is conceivable to project one or more patterns using a lidar scanner present on the vehicle (e.g., a 360° lidar scanner that may be arranged on the roof of the vehicle). Here, it may be necessary, depending on the situation, to synchronize the movement of such a lidar scanner (or other movable light source) with the corresponding shooting time of the images in the first and second cameras. Furthermore, it is possible that the light source is configured to project the corresponding pattern only within the overlapping area. This avoids interference with other sensors of the vehicle and / or other road users by projecting the pattern onto other areas of the vehicle's environment. Furthermore, the energy consumption for projecting the pattern can be kept low by projecting the pattern only onto overlapping areas.

[0022] The corresponding pattern projected by the light source can be constructed, for example, as horizontal lines and / or vertical lines and / or diagonals and / or circles and / or ellipses and / or rectangles and / or polygons and / or grids and / or point clouds and / or constructed differently.

[0023] More preferably, the environmental detection system can be configured based on the pattern (or multiple patterns) to perform calibration of the first camera and / or the second camera. This can be performed, for example, once during vehicle production or repeatedly, so that changes in camera orientation relative to the vehicle can be repeatedly detected as they occur, and preferably compensated accordingly.

[0024] In another advantageous configuration of the invention, the environmental detection system is configured to adjust the corresponding pattern projected onto the overlapping region based on currently existing boundary conditions and / or based on a predefined time series. Such boundary conditions may, for example, take into account the vehicle's current speed and / or current visibility conditions (e.g., due to weather conditions) and / or the type and / or number of currently activated vehicle functions.

[0025] More preferably, the light source is a first light source, and the overlapping area of ​​the field of view of the first camera and the field of view of the second camera is a first overlapping area. The environmental detection system also has at least one third camera and at least one second light source, wherein the field of view of the third camera has a second overlapping area with the field of view of the first camera or the field of view of the second camera. The second light source is configured to project at least one predefined additional pattern (or two, three, four or more patterns) onto the second overlapping area, and the environmental detection system is configured, based on said additional pattern, to substantially seamlessly join a third image captured by the third camera with the first image and / or the second image to produce a holistic image from the first image, the second image, and the third image. It goes without saying that all the configurations of the present invention described above for stitching together the first and second images can also be applied to stitching the third image with the first and / or the second image. Furthermore, it should be noted that additional cameras can be provided and correspondingly additional overlapping areas can be provided to, for example, provide a complete panoramic view around the vehicle.

[0026] According to a second aspect of the invention, a vehicle is provided having an environmental detection system according to the first aspect of the invention. As described above, the vehicle may be configured as, for example, a passenger car, a truck, a bus, a transport vehicle, a two-wheeled vehicle, or a rail vehicle. The features, combinations of features, and the resulting advantages obviously correspond to those set forth in conjunction with the first mentioned aspects of the invention, and thus, to avoid repetition, please refer to the discussion above. Attached Figure Description

[0027] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. In the drawings: Figure 1 A schematic diagram illustrating one embodiment of an environmental detection system according to the present invention, integrated with a vehicle; Figure 2a An exemplary first image is shown, captured by an environmental detection system according to the present invention; Figure 2b An exemplary second image is shown, captured by an environmental detection system according to the invention; and Figure 2c The overall image generated from the first and second images is shown. Detailed Implementation

[0028] Figure 1 A schematic diagram of one embodiment of the environmental monitoring system 5 according to the present invention, which is combined with a vehicle configured as a passenger car, is shown.

[0029] The environmental monitoring system 5 includes a first camera 10, a second camera 15, a third camera 80, a first laser light source 20, a second laser light source 25, and an analysis and processing unit 30.

[0030] Cameras 10, 15, and 80 are configured as RGB cameras and are further configured to receive lasers emitted by laser sources 20 and 25 that are outside the visible wavelength range.

[0031] Cameras 10 and 15 are positioned on the vehicle and aligned in such a way that their respective fields of view 11 and 16 form a first overlapping area 40.

[0032] Cameras 10 and 80 are positioned on the vehicle and aligned in such a way that their respective fields of view 11 and 81 form a second overlapping area 45.

[0033] The first laser source 20 is configured to project a predefined first pattern 50 and a predefined second pattern 55 into a first overlapping region 40, while the second laser source 25 is configured to project a predefined third pattern 56 into a second overlapping region 45.

[0034] It should be pointed out that, Figure 1 The representation and quantity of patterns 50, 55, and 56 shown can deviate arbitrarily in actual implementation, and for simplicity, they are shown here only as simple geometric shapes.

[0035] The analysis and processing unit 30 is configured here as a CPU, which is set up to process the first image 12 captured by the first camera 10 (see... Figure 2a In the second image 17 captured by the second camera 15 (see...) Figure 2b Identify the first pattern 50 and the second pattern 55 in the process.

[0036] The analysis and processing unit 30 is also configured to process the first image 12 captured by the first camera 10 (see...) Figure 2a The third pattern 45 and the second pattern 55 are identified in the third image captured by the third camera 80.

[0037] Based on this recognition, the analysis and processing unit 30 is also configured to determine the first splicing line 60 in the first image 12 (see...). Figure 2a And find the second splicing line 65 in the second image 17 that corresponds to the first splicing line 60 (see...). Figure 2bAnd based on the first splicing line 60 and the second splicing line 65, the first image 12 and the second image 17 are spliced ​​together to form an overall image 70 (see...). Figure 2c There is a substantially seamless transition between the first image 12 and the second image 17 in the overall image.

[0038] The analysis and processing unit 30 is additionally configured to add a third image to the overall image 70 based on the third pattern 56, similar to the description above.

[0039] Advantageously, the analysis and processing unit 30 is configured to match the first image 12 and / or the second image 17 and / or the third image with each other by means of geometric transformation before assembling the overall image 70 based on the patterns 50, 55 and 56 identified respectively in the first image 12, the second image 17 and / or the third image.

[0040] Figure 2a An exemplary first image 12 is shown, taken by the environmental monitoring system 5 according to the present invention.

[0041] It should be pointed out that, Figure 2a , 2b The images shown in 2c are simplified and are not shown in perspective correctly.

[0042] also, Figure 2a It shows in Figure 1 Patterns 50 and 55 described herein are detected both in the first image 12 and in the second image 17 (see [image description]). Figure 2b It was detected in ).

[0043] also, Figure 2a The first splicing line 60 obtained by the analysis and processing unit 30 based on patterns 50 and 55 is shown.

[0044] Figure 2b An exemplary second image 17 is shown, captured by the environmental detection system 5 according to the invention, which is preferably captured at the same time as the first image 12.

[0045] also, Figure 2b It shows in Figure 1 Patterns 50 and 55 described in the first image 12 (see Figure 2a It was detected in ) and also in the second image 17.

[0046] also, Figure 2b The second splicing line 60, obtained by the analysis and processing unit 30 based on patterns 50 and 55, is shown.

[0047] Figure 2cAn overall image 70 generated from a first image 12 and a second image 17 is shown, wherein splicing lines 60 and 65 are additionally shown at which the first image 12 and the second image 17 are spliced ​​together.

Claims

1. An environmental monitoring system (5), which has: First camera (10) Second camera (15) Light source (20), and Analysis and processing unit (30). in, The field of view (11) of the first camera (10) and the field of view (16) of the second camera (15) have a predefined overlapping area (40). The light source (20) is configured to project at least one predefined pattern (50) into the overlapping region (40) of the respective fields of view (11, 16) of the first camera (10) and the second camera (15), and The analysis and processing unit (30) is configured to be used for, - Identify the at least one predefined pattern (50) in the first image (12) captured by the first camera (10) and in the second image (17) captured by the second camera (20). - Based on the patterns (50) identified in the first image (12) and the second image (17) respectively, the first splicing line (60) in the first image (12) is obtained and the second splicing line (65) corresponding to the first splicing line (60) in the second image (17) is obtained, and - The first image (12) and the second image (17) are stitched together into an overall image (70) based on the first stitching line (60) and the second stitching line (65), in which there is a substantially seamless transition between the first image (12) and the second image (17).

2. The environmental monitoring system (5) according to claim 1, wherein, The predefined pattern is a predefined first pattern (50), and The environmental detection system (5) is configured to project at least one predefined second pattern (55) into the overlapping region (40) of the fields of view (11, 16) of the first camera (10) and the second camera (15), and The analysis and processing unit (30) is configured to additionally determine the first splicing line (60) and the second splicing line (65) based on the second pattern (55).

3. The environmental monitoring system (5) according to claim 2, wherein, The first pattern (50) and the second pattern (55) are constructed as clearly distinguishable patterns (50, 55).

4. The environmental monitoring system (5) according to any one of the preceding claims, wherein, The analysis and processing unit (30) is configured to match the first image (12) and / or the second image (17) with each other by means of geometric transformation before assembling the overall image (70) based on the patterns (50, 55) identified in the first image (12) and the second image (17) respectively.

5. The environmental monitoring system (5) according to any one of the preceding claims, wherein, The light source (20) It is a laser light source and / or an infrared light source, and / or Configured for emitting light in the visible and / or invisible wavelength range, and / or Integrated into the first camera (10) and / or the second camera (15), or arranged separately from the first camera (10) and the second camera (15), and / or It has multiple individual light sources (20), and / or Set up to use the corresponding patterns (50, 55). - Projection is performed movably and / or immovably relative to the overlapping area (40), and / or - Projection is performed only within the overlapping region (40).

6. The environmental monitoring system (5) according to any one of the preceding claims, wherein, The corresponding patterns (50, 55) projected by the light source (20) are constructed as follows Horizontal lines and / or vertical lines and / or diagonals, and / or Circle and / or ellipse, and / or Rectangle, and / or Polygons, and / or Grid, and / or Dotted clouds.

7. The environmental monitoring system (5) according to any one of the preceding claims, wherein, The environmental detection system (5) is configured based on the patterns (50, 55) to perform calibration of the first camera (10) and / or the second camera (15).

8. The environmental monitoring system (5) according to any one of the preceding claims, wherein, The environmental monitoring system (5) is configured to be used for... Current boundary conditions, and / or Predefined time series To adjust the corresponding patterns (50, 55) projected onto the overlapping area (40).

9. The environmental monitoring system (5) according to any one of the preceding claims, wherein, The light source is the first light source (20). The overlapping area of ​​the field of view (11) of the first camera (10) and the field of view (16) of the second camera (15) is the first overlapping area (40). The environmental detection system (5) has at least one third camera (80) and at least one second light source (25). The field of view (81) of the third camera (80) has a second overlapping area (45) with the field of view (11) of the first camera (10) or the field of view (16) of the second camera (15). The second light source (25) is configured to project a predefined additional pattern (56) into the second overlapping area (45). The environmental detection system (5) is configured based on the additional pattern (56) to connect a third image captured by the third camera (80) substantially seamlessly with the first image (12) or the second image (17) to generate an overall image (70) from the first image (12), the second image (17) and the third image.

10. A vehicle having an environmental detection system (5) according to any one of the preceding claims.

Images