Environmental monitoring device and method for operating an environmental monitoring device

Description

[0001] The invention relates to an environmental monitoring device for vehicles, in particular for construction machinery or agricultural machinery, and a method for operating an environmental monitoring device.

[0002] There is a rapidly growing demand for 3D sensors in vehicles, particularly in agricultural and construction machinery. This demand stems from assistance functions for manned vehicles, but also increasingly from the corresponding functionality of unmanned vehicles. To cover the widest possible range of applications, hereinafter also referred to as "applications," cost-effective solutions are required for these sensors.

[0003] A wide field of view is fundamentally necessary for monitoring the vehicle's surroundings. However, a narrower field of view offers advantages such as improved depth accuracy and a greater range. Therefore, to cover all required applications, different device variants are needed that can display varying field of view angles.

[0004] This can be achieved, for example, by using different device variants depending on the application. However, this solution is costly, as a vehicle or application must be equipped with several devices operating in parallel. Alternatively, motorized zoom lenses can be used. However, stereo technology is highly dependent on the quality of the geometric calibration. With motorized zoom lenses, all adjustment options must be calibrated and reproducibly adjustable, resulting in significant effort and high costs.

[0005] US 2005 / 0265619 A1 discloses an environmental monitoring device with the features according to the preamble of claim 1. The system disclosed therein can, for example, be used as a monitoring system on a vehicle and comprises two cameras, each equipped with a fisheye lens. The cameras are connected to a computer which has two sets of image provisioning devices for a rectification method of the camera images.

[0006] It is an object of the present invention to propose an environmental monitoring device and a method for operating an environmental monitoring device which enables a wide range of applications to be carried out with low complexity and low cost.

[0007] This task is solved by the subject matter of the independent claims. Advantageous further developments are the subject matter of the dependent claims and also arise from the description and the drawings.

[0008] The environmental monitoring device according to the invention is intended for use in vehicles, in particular construction machinery or agricultural machinery, and comprises a stereo camera which has two fisheye lenses, an image sensor device with at least two sensor areas for detecting optical signals from the fisheye lenses in each of the sensor areas and for generating and outputting image data signals based on the respective optical signals, a processing unit for processing the image data signals output by the image sensor device and for providing the processed image data signals, in particular for use by an application currently executed by the environmental monitoring device, and a control unit.The sensor areas each have at least a first sub-area corresponding to a first opening angle within the maximum opening angle of the respective fisheye lens, and a second sub-area corresponding to a second opening angle within the maximum opening angle of the respective fisheye lens. The first and second opening angles differ from each other. The processing unit is adjustable to selectively process the image data signals according to only one of the sub-areas. The control unit is configured to adjust the processing unit, in particular to one of the sub-areas, so that the image data signals are processed according to only that sub-area, especially at any given time.

[0009] According to the invention, it is possible to read out image data signals from the image sensor device and then preferably process them in at least two different ways, namely once for the first sub-area and once for the second sub-area. The processing unit can therefore be selectively configured to process either image data signals from the first or the second sub-area. In this way, it is possible, for example, to achieve high accuracy in depth resolution in the first sub-area, while subsequently covering a large image field with the second sub-area. It is also possible to read out the image data signals separately for each sub-area, in particular only from those parts of the sensor areas that provide image data for the corresponding sub-area. The reading out for different sub-areas can then be performed sequentially.

[0010] The image sensor device can, in principle, either have a separate image sensor for each fisheye lens, or it can have a single image sensor onto which both fisheye lenses project. This means that the sensor areas can either be areas of a single image sensor, arranged side by side or even partially overlapping, or they can be areas of two different image sensors within the image sensor device. The sensor areas can, in particular, each be configured as a two-dimensional array, for example, as CMOS or CCD image sensors, which receive optical signals from the respective fisheye lens and generate electronic signals—the image data signals—based on these signals.

[0011] The first and second aperture angles differ from each other and are smaller than or equal to the maximum aperture angle. Thus, the first and second sub-areas also differ from each other. The maximum aperture angle of the fisheye lenses can be approximately 180° or only slightly less than 180°, ranging from 170° to 180°. The first aperture angle can, for example, be in the range of 60° to 100°, preferably 60° or 90°. The second aperture angle can, for example, be in the range of 100° to 160°, preferably 130° or 140°.

[0012] The sensor areas can be configured such that less than the maximum aperture angle of a fisheye lens is projected onto the sensor area. The aperture angle projected onto the sensor area can, in particular, correspond to the second aperture angle. This second sub-area can then correspond to the entire sensor area of ​​a fisheye lens.

[0013] The first sub-area and the second sub-area can preferably be rectangular. More preferably, the first sub-area can be completely contained within the second sub-area, with the first sub-area, for example, occupying half or a quarter of the area of ​​the second sub-area.

[0014] The fisheye lens can, in particular, be an equidistant fisheye lens.

[0015] By configuring the processing unit, the image data signals are processed according to only one of the sub-areas. The processing unit can thus be configured by the control unit so that the image data signals are processed only for this specific sub-area. This allows the control unit to determine whether image data is used for a small or a large field of view, as only the image data signals processed by the processing unit are made available for use by the application. The processing unit's configuration by the control unit can be particularly dependent on the application being run, so that image data can be acquired with either a large field of view or, in comparison, with a smaller field of view but improved depth accuracy and a greater distance range.

[0016] Preferably, the processing unit is adjustable during operation of the stereo camera to selectively process the image data signals according to only one of the sub-areas. "During operation" means that switching between the sub-areas corresponding to the respective opening angles is possible even during runtime. This means that image data signals corresponding to a large opening angle can initially be processed and made available for the application, and later according to a smaller opening angle, or vice versa, for example, if the application executed by the environmental monitoring device is changed during operation.

[0017] The resolution of the processed image data signals can be independent of the processing unit's setting for any of the sub-areas. For the best possible lateral object resolution, the highest possible image resolution is required, just as for the best possible depth resolution. However, apart from the processing power, the achievable image resolution is limited only by the lens and the resolution of the image sensor. With fisheye lenses, due to the barrel distortion, the scene information projected onto a pixel is higher at the edge of the image than in the center, and this effect increases with the angle of view. At the maximum angle of view, the maximum resolution of the processed image is therefore limited only by the edge regions of the input image, i.e., the image data signals output by the image sensor device, resulting in information loss in the center.With a comparatively smaller opening angle, however, all the information in the center of the image data signals received from the image sensor device can be used, since the peripheral areas are not needed. Put simply, with a comparatively large opening angle, the image data signals are reduced in size during processing, whereas with a comparatively small opening angle, the processed image can be approximately the same size as the image or image data signals received from the image sensor device.

[0018] The processing unit is advantageously designed to rectify the image data signals during processing. Rectification means that the image data signals are modified to correct, among other things, geometric aberrations. A rectified image has the property of a uniform image scale across the entire image area. An object at any point in the image is thus mapped onto the same number of pixels. In particular, rectification can include creating the stereo image, in which, firstly, the distortion of the respective fisheye lens is corrected, and secondly, the camera or lens pair is aligned with each other. Here, the two cameras are aligned so that horizontal lines in the scene are represented on a single image line in the rectified stereo image pair.

[0019] Rectifying the image data signals can involve stretching the image edges and / or compressing the image center. Furthermore, the rectification process can differ depending on the processing unit's setting for one of the image sub-areas. In particular, setting the unit to a sub-area corresponding to a smaller aperture angle requires less correction of aberrations than setting it to a sub-area corresponding to a larger aperture angle.

[0020] Rectification can also include changing the resolution of the image data signals. Scaling can follow rectification. In particular, the resolution of the processed image data signals, that is, after rectification and any necessary scaling, can be the same regardless of which sub-area the image data signals are being processed. Thus, image data signals with the same resolution are always available for further use, which significantly simplifies downstream processing, for example by FPGA (Field Programmable Gate Array) or ASIC (application-specific integrated circuit), and reduces computing power.

[0021] Thus, different image rectifications can generate 2D data and depth data with different properties. Either an image with a large field of view but limited depth resolution and lower lateral resolution can be produced for object detection, or an image with improved lateral and depth resolution but a limited field of view, with both images potentially having the same resolution.

[0022] The image sensor device is adjustable to selectively generate and output image data signals corresponding to only one of the sub-areas. The control unit can be configured to adjust the image sensor device accordingly, particularly to one of the sub-areas, so that the image data signals are generated and output based on the optical signals received only from that sub-area. Thus, for example, only the pixels of a sub-area can be read out. In this way, the generation and output of image data signals by the image sensor device can be limited to a predefined opening angle, thereby reducing the number of image data signals to be transmitted.

[0023] The method according to the invention serves to operate an environmental monitoring device, which comprises a stereo camera with two fisheye lenses and an image sensor device with at least two sensor areas for detecting optical signals from the fisheye lenses in each of the sensor areas and for generating and outputting image data signals based on the respective optical signals, wherein the sensor areas each have at least a first sub-area corresponding to a first opening angle within the maximum opening angle of the respective fisheye lens, and a second sub-area corresponding to a second opening angle within the maximum opening angle of the respective fisheye lens, wherein the first opening angle and the second opening angle differ from each other, and wherein the image sensor device is adjustable.to selectively generate and output the image data signals according to only one of the sub-areas.

[0024] The procedure consists of the following steps: Generating an application requirement based on an application currently being executed by the environmental monitoring device, receiving the image data signals output by the image sensor device and processing the image data signals, in particular at a given time, selectively according to only one of the sub-areas depending on the application requirement and providing the processed image data signals, in particular for use by the application.

[0025] According to the procedure, depending on the application being executed, an application request is generated. This request contains information about which application is currently being run by the environmental monitoring device, or which image data signals the application requires—that is, image data signals corresponding to which of the available opening angles within the maximum opening angle, for example, the first opening angle or the second opening angle. Thus, the application request defines whether image data signals with a large or small opening angle are required. Subsequently, the image data signals are processed only according to the portion corresponding to the desired opening angle.

[0026] Preferably, the image data signals are rectified during processing before being provided. For rectification, please refer to the explanations above. Rectification of the image data signals can include stretching the image edges and / or compressing the image center. Furthermore, depending on the processing of the image data signals, the rectification may differ in only one of these sub-areas. The resolution of the image data signals can also be changed during rectification.

[0027] Preferably, the procedure comprises the following steps: Processing and rectifying the image data signals according to only the first sub-area, followed by processing and rectifying the image data signals according to only the second sub-area, and combining the rectified image data signals according to the first sub-area and the rectified image data signals according to the second sub-area into combined image data signals.

[0028] This allows image data signals corresponding to a smaller and a larger aperture angle to be processed in parallel and then combined. The combining process can be made particularly simple and efficient if certain calibration parameters are restricted, requiring a multiple of the focal length (e.g., twice the focal length) for the different aperture angles. The resulting image data exhibits a lateral resolution and depth resolution in the image center that is several times the focal length (e.g., twice the focal length) compared to the image edge.

[0029] The invention further relates to a computer program comprising instructions which, when the program is executed by a computer, cause it to perform the method described above, and to a computer-readable storage medium comprising instructions which, when executed by a computer, cause it to perform the method described above.

[0030] The information provided above in connection with the environmental monitoring device applies equally to the method for operating an environmental monitoring device and vice versa. The same applies to the data processing device, the computer program, and the computer-readable storage medium.

[0031] The invention is explained below by way of example only, using a schematic embodiment shown in the drawings. The drawings show: Fig. 1 a schematic top view of a vehicle with an environmental monitoring device according to an exemplary embodiment, Fig. 2 a diagram showing the structure of the environmental monitoring device. Fig. 1 schematically illustrated, and Fig. 3 a schematic representation of the stereo camera with the fisheye lenses and the associated sensor areas of the environmental monitoring device. Fig. 1 .

[0032] In Fig. 1Figure 100 schematically depicts a vehicle 100, which can be configured as a construction or agricultural machine. The vehicle 100 has a surround-view monitoring device 10 at its front, which will be described in more detail below. It is also conceivable that surround-view monitoring devices 10 could be arranged at the sides or rear of the vehicle 100, either at the front or as an alternative.

[0033] How Fig. 2 As shown, the environmental monitoring device 10 comprises a stereo camera 11, which includes two parallel and aligned fisheye lenses 19a, 19b, an image sensor device 13, a control unit 15 and a processing unit 17.

[0034] The in Fig. 3The lens-shaped fisheye lenses 19a, 19b each have a maximum opening angle ω max, which can only be slightly less than 180° and is approximately in the range of 170° to 180°. Depending on the application to be performed by the environmental monitoring device 10, however, it may also be necessary to obtain image data for a smaller opening angle in order to achieve improved depth accuracy and a greater distance range. This is solved in the present embodiment as follows: In this example, the image sensor device 13 has two sensor areas 21a, 21b. Here, the sensor areas 21a, 21b are each assigned to their own image sensor. These two image sensors are part of the image sensor device 13. In other embodiments, however, it may also be possible for both sensor areas 21a, 21b to be provided by only one image sensor, which, however, is at least functionally subdivided within itself.

[0035] The two sensor areas receive and capture optical signals from their respective assigned fisheye lenses 19a and 19b; that is, sensor area 21a is assigned to fisheye lens 19a, and sensor area 21b is assigned to fisheye lens 19b. Electronic image data signals are generated and output based on the captured optical signals.

[0036] In the present example, each sensor area 21a, 21b is subdivided into sub-areas T1, T2, T3, where in this case sub-area T3 includes sub-area T2 and sub-area T2 in turn includes sub-area T1. However, it is also conceivable that the sub-areas T1, T2, T3 of the respective sensor area 21a, 21b are adjacent sub-areas of sensor area 21a, 21b without overlap. The subdivision of the sensor areas 21a, 21b into their sub-areas T1, T2, T3 is to be understood as follows: the sensor areas 21a, 21b are designed as a two-dimensional arrangement (array), wherein the sub-areas T1, T2, T3 each correspond to a part of this two-dimensional arrangement in which optical signals from the associated fisheye lens 19a, 19b are received according to a respective opening angle ω 1 , ω 2 , ω 3.

[0037] The first sub-area T1 corresponds to the first opening angle ω1 of the respective fisheye lens 19a, 19b. This means that the optical signals received in this sub-area T1 represent the objects located within the first opening angle ω1. Similarly, the second sub-area T2 corresponds to a second opening angle ω2 and the third sub-area T3 to a third opening angle ω3 of the respective fisheye lens 19a, 19b. The opening angles ω1, ω2, ω3 are all within the maximum opening angle ωmax of the fisheye lenses 19a, 19b. In this case, the third opening angle ω3 corresponds to the maximum opening angle ωmax, whereas the two opening angles ω1, ω2 are smaller than it.

[0038] The control unit 15 is connected to the processing unit 17 via a signal connection and can configure the processing unit 17 to one of the sub-areas T1, T2, or T3. This means that, depending on the setting, the processing unit 17, which is connected to the image sensor device 13 and receives the image data signals from the image sensor device 13, processes the image data signals only according to one of the sub-areas T1, T2, or T3. In other words, the processing unit 17 knows the pixel areas of the image data signals received from the image sensor device 13 that correspond to the sub-areas T1, T2, or T3 and uses only the pixel area of ​​the image data signals that corresponds to the respective sub-area T1, T2, or T3 for further processing.The processing unit 17 is configured by the control unit 15 primarily based on an application requirement, which includes information about which application is currently being executed by the environmental monitoring device 10, or which image data signals are required by the application—that is, image data signals corresponding to which of the available opening angles ω1, ω2, ω3. Thus, the processing unit 17 is set to one of the sub-areas T1, T2, T3, so that the image data signals are selectively processed only according to that sub-area T1, T2, T3, which corresponds to the desired opening angle ω1, ω2, ω3.

[0039] As part of the processing, processing unit 17 performs a rectification of the image data signals. This corrects geometric imaging errors, which are particularly noticeable with fisheye lenses with large aperture angles. In particular, rectification can include establishing the stereo normal case, in which, firstly, the distortion of the respective fisheye lens is corrected, and secondly, the camera or lens pair is aligned with each other.

[0040] During the rectification of the image data signals by processing unit 17, the image edges are stretched and the image center is compressed. However, the degree of rectification can vary depending on the opening angle. In particular, a more pronounced rectification is required at a comparatively large opening angle, where significant distortion occurs, than at a small opening angle.

[0041] Thus, different image rectifications can generate 2D data and depth data with different properties. Either an image with a large field of view but limited depth resolution and lower lateral resolution for object detection, or an image with improved lateral and depth resolution but a limited field of view, can be generated, with both images potentially having the same resolution. The present embodiment therefore results in an environmental monitoring device 10 that is comparatively simple and inexpensive to manufacture, and which can also be used for a wide range of applications, since appropriate image data can be provided depending on the application. Reference symbol list

[0042] 10 Environmental monitoring device 11 Stereo camera 13 Image sensor device 15 Control unit 17 Processing unit 19a Fisheye lens 19b Fisheye lens 21a Sensor area 21b Sensor area 100 vehicles T1 sub-area T2 sub-area T3 sub-area ω max maximum opening angle ω 1 opening angle ω 2 opening angle ω 3 opening angle

Claims

1. An environment monitoring apparatus (10) for vehicles (100), comprising a stereo camera (11) which has two fisheye lenses (19a, 19b), an image sensor apparatus (13) having at least two sensor regions (21a, 21b) for detecting optical signals of the fisheye lenses (19a, 19b) in a respective one of the sensor regions (21a, 21b) and for generating and outputting image data signals based on the respective optical signals, a processing unit (17) for processing the image data signals output by the image sensor apparatus (13) and for providing the processed image data signals, and a control unit (15) for setting the processing unit (17), wherein the sensor regions (21a, 21b) each have a first part region (T1), which corresponds to a first aperture angle (ω1) within the maximum aperture angle (ωmax) of the respective fisheye lens (19a, 19b), and a second part region (T2) which corresponds to a second aperture angle (ω2) within the maximum aperture angle (ωmax) of the respective fisheye lens (19a, 19b), wherein the first aperture angle (ω1) and the second aperture angle (ω2) differ from one another and characterized in that the first aperture angle (ω1) is smaller than the second aperture angle (ω2), in that the image sensor apparatus (13) can be set to selectively generate and output the image data signals corresponding to only one of the part regions (T1, T2) of the respective sensor region (21a, 21b), and in that the processing unit (17) can be set in order to selectively process the image data signals corresponding to only one of the part regions (T1, T2) of the respective sensor region (21a, 21b).

2. An environment monitoring apparatus (10) according to claim 1, characterized in that the processing unit (17) can be set during operation of the stereo camera (11) to selectively process the image data signals corresponding to only one of the part regions (T1, T2) of the respective sensor region (21a, 21b).

3. An environment monitoring apparatus (10) according to claim 1 or 2, characterized in that a resolution of the processed image data signals is independent of the setting of the processing unit (17) to one of the part regions (T1, T2).

4. An environment monitoring apparatus (10) according to any one of the preceding claims, characterized in that the processing unit (17) is configured to rectify the image data signals during the processing.

5. A method for operating an environment monitoring apparatus (10), said environment monitoring apparatus (10) comprising a stereo camera (11) having two fisheye lenses (19a, 19b) and an image sensor apparatus (13) having at least two sensor regions (21a, 21b) for detecting optical signals of the fisheye lenses (19a, 19b) in a respective one of the sensor regions (21a, 21b) and for generating and outputting image data signals based on the respective optical signals, wherein the sensor regions (21a, 21b) each comprise a first part region (T1), which corresponds to a first aperture angle (ω1) within the maximum aperture angle (ωmax) of the respective fisheye lens (19a, 19b), and a second part region (T2) which corresponds to a second aperture angle (ω2) within the maximum aperture angle (ωmax) of the respective fisheye lens (19a, 19b), wherein the first aperture angle (ω1) and the second aperture angle (ω2) differ from one another and characterized in that the first aperture angle (ω1) is smaller than the second aperture angle (ω2), wherein the image sensor apparatus (13) can be set to selectively generate and output the image data signals corresponding to only one of the part regions (T1, T2) of the respective sensor region (21a, 21b), and wherein the method comprises the following steps: - generating an application requirement based on an application currently executed by the environment monitoring apparatus (10), - receiving the image data signals output by the image sensor apparatus (13) and processing the image data signals corresponding to only one of the part regions (T1, T2) of the respective sensor region (21a, 21b) selectively in dependence on the application requirement, and - providing the processed image data signals.

6. A method according to claim 5, characterized in that the image data signals are rectified during the processing before the provision.

7. A method according to claim 6, characterized in that the rectification of the image data signals comprises stretching the image margins and / or compressing the image center.

8. A method according to claim 6 or 7, characterized in that the rectification differs depending on the processing of the image data signals corresponding to only one of the part regions (T1, T2).

9. A method according to any one of the claims 6 to 8, characterized in that the rectification comprises changing a resolution of the image data signals.

10. A method according to any one of the claims 5 to 9, characterized by the processing and rectification of the image data signals corresponding to only the first part region (T1), followed by the processing and rectification of the image data signals corresponding to only the second part region (T2), and the merging of the rectified image data signals corresponding to the first part region (T1) and the rectified image data signals corresponding to the second part region (T2) to form merged image data signals.

11. A computer program comprising commands that, on the execution of the program by a computer of an environment monitoring apparatus for vehicles, cause said environment monitoring apparatus to perform the method according to any one of the claims 5 to 10.

12. A computer-readable storage medium comprising commands that, on the execution by a computer of an environment monitoring apparatus for vehicles, cause said environment monitoring apparatus to perform the method according to any one of the claims 5 to 10.