Environmental perception system of a motor vehicle, method for operating an environmental perception system, computer program product and motor vehicle

Description

[0001] This document describes an environment detection system for a motor vehicle, a method for operating an environment detection system, a computer program product, and a motor vehicle.

[0002] Environmental detection systems of a motor vehicle, methods for operating an environmental detection system, computer program products and motor vehicles of the type mentioned above are known in the prior art.

[0003] Environment recognition systems are used in motor vehicles for various purposes, including autonomous driving systems and driver assistance systems such as speed limit warnings that read speed limits from signs, and lane assist systems that need to recognize lane markings.

[0004] Furthermore, light projection systems are known in motor vehicles that can generate light structures and project them into the environment, e.g. onto signs, walls or roadways.

[0005] One previously mentioned application is lane departure warning systems. A first known type is active lane departure warning systems, which actively assist the driver of a suitably equipped vehicle. Such systems intervene in the steering, for example, to prevent an unintentional lane change—that is, a lane change without the driver indicating an intention to change lanes by using a turn signal. Passive, purely warning systems are also known, such as those that generate a vibration in the steering wheel, alerting the driver that continuing in the same direction will result in leaving the lane. Furthermore, lane departure warning systems are known that combine both of the aforementioned systems and assist the driver through visual projections onto the road, for example, by projecting solid or streak lines onto the road.For example, a non-solid line can be overlaid with a projection of a solid line if a safe lane change is not possible, for instance, due to another vehicle in the destination lane. Furthermore, combinations of the aforementioned types of lane assist systems are known.

[0006] In the latter systems, which work with projections, it can happen that the projection interferes with a system for environmental detection, for example by means of a front camera, so that the projection overexposes a structure to be detected, e.g. it can be mistaken by the optical environmental detection system for an actual lane marking or the lane marking is marked as not recognizable.

[0007] US Patent 2019 / 293791 A1 discloses an image acquisition and control device for target detection and distance evaluation for vehicles. The device consists of an infrared light irradiation unit for illuminating the target and an infrared light detector receiver unit and a visible light receiver unit for receiving the light signal reflected from the target. The infrared and visible light reflected from the target is used to assess the distance to the target, to image the target's contour, and to detect backlighting, fog, and color. Fine-tuning the incident and received light conditions allows for the differentiation of these parameters after analyzing the superposition of incident and reflected signals during exposure times. This fine-tuning is performed by a controller once the detection unit has determined that the peripheral conditions correspond to a predefined set of parameters.The device generates visible light, IR and distance image data as well as an object ID.

[0008] US Patent 2019 / 095726 A1 describes a vehicle detection support system comprising a marker light device for projecting a marking and a camera device. The marker light device is controllable in a pulsed manner, with the frequency at which a marking is projected being synchronized to a frame rate of the camera.

[0009] From US 2016 / 034771 A1 a method and system for scanning a road surface is known, wherein a structured light pattern is projected into a path of the vehicle, which is captured by a camera and subsequently evaluated.

[0010] The task is therefore to further develop the environmental perception systems of a motor vehicle, methods for operating an environmental perception system, computer program products and motor vehicles of the type mentioned above in such a way that environmental perception systems which have projectors for emitting light structures can detect lane markings on a road more reliably.

[0011] The problem is solved by an environment detection system of a motor vehicle according to claim 1, a method for operating an environment detection system according to dependent claim 6, a computer program product according to dependent claim 10, and a motor vehicle according to dependent claim 11. Further embodiments and developments are the subject of the dependent claims.

[0012] The following describes an environment detection system for a motor vehicle, comprising at least one projector configured to project linear light structures onto an area of ​​a roadway in front of the motor vehicle, and at least one camera arranged to detect an area in front of the motor vehicle, wherein the camera is sensitive to at least a sub-range of the frequency spectrum emitted by the at least one projector in order to detect reflections of the light projected by the projector onto the area of ​​the roadway, wherein the projector has a modulatable light source arrangement, wherein a control unit is provided which is connected to the at least one camera and the at least one projector, wherein at least one timer is provided to pulse the projector and camera, and wherein the projector is operated in pulsed mode controlled by the timer.

[0013] The controller includes a camera data evaluation module designed to separately evaluate camera data from a first pulse state and a second pulse state. The evaluation module is further configured to identify overlapping areas in the at least two camera images and to recognize track markings in the camera images.

[0014] The control system is designed to change the light structures when the confidence level for lane marking detection falls below a predetermined value.

[0015] The camera data can be camera images, but it can also contain structured data, for example, detected objects.

[0016] The object could be, for example, a sign, a wall, or a roadway. The light structure could be linear or planar, among other things.

[0017] By synchronizing the timing of the projector and camera, with the projector operating in pulsed mode, it is possible for the camera to perceive the light structure emitted by the projector specifically according to the pulsed reproduction, thus enabling lane markings on the road to be distinguished from the pulsed light structures.

[0018] In a first further embodiment, it can be provided that the control system has a light modulation module which is configured to modulate the light emitted by the projector, so that the projector can emit at least a first light structure and at least a second light structure different from the first light structure, wherein the switching between the first light structure and the second light structure is controlled by means of the timer.

[0019] When different lighting structures are used, controlled by a timer, it is possible to clearly distinguish between a first and a second lighting structure. The timing provided by the timer allows for a temporal correlation between the changes between the different lighting structures and the images captured by the camera.

[0020] If camera data from the first pulse state and camera data from the second pulse state are evaluated separately, it is possible to compare the differences between the evaluated camera data and identify similarities and differences between the two sets of evaluated camera data. This allows for more reliable conclusions to be drawn about the information on track markings contained in the respective camera data. The camera data can represent camera images and / or camera images enriched with object information.

[0021] In a further refinement, the control system may include an evaluation module for camera data, which is designed to evaluate camera data from the first light structure and the second light structure separately.

[0022] If camera data from the first light structure and camera data from the second light structure are evaluated separately, it is possible to compare the differences between the evaluated camera data and identify similarities and differences between the two sets of data. This allows for more reliable conclusions to be drawn about the information on lane markings contained in the respective camera data. The camera data can represent camera images and / or camera images enriched with object information.

[0023] In a further, more advanced embodiment, it may be provided that a light timer is included for the projector and a sampling rate timer is included for the camera.

[0024] If a light timer is provided for the projector and a sampling rate timer for the camera, the system can be built more modularly. For example, a sampling rate timer in the camera can be used, which is addressed accordingly by the controller. Light timers are also frequently part of lighting modules. These light timers can also be addressed by the controller.

[0025] In a further refinement, it can be provided that a clock frequency of the light timer is an integer multiple of a clock frequency of the sampling rate timer, or a clock frequency of the sampling rate timer is an integer multiple of a clock frequency of the light timer.

[0026] If the clock frequency of the light timer is an integer multiple of the clock frequency of the sampling rate timer, or vice versa, it is possible to synchronize the projector and the camera accordingly, so that, for example, an image is always recorded when the projector is not illuminated, thus ensuring that the image captured by the camera is not disturbed by the light structure generated by the projector. This allows for the identification of the track marker.

[0027] In a further, more advanced embodiment, the light timer and sampling rate timer can be synchronized. In particular, it can be provided that a camera image is always generated when the projector is not emitting a light structure.

[0028] In a further refinement, it can be provided that a clock frequency of the light timer is a non-integer multiple of a clock frequency of the sampling rate timer, or a clock frequency of the sampling rate timer is a non-integer multiple of a clock frequency of the light timer.

[0029] If the corresponding clock frequencies are not integer multiples of each other, the clocks overlap with varying degrees of overlap. This results in camera images where the light structure is visible at full brightness, not at all, and / or at a reduced brightness. This variance allows differences and similarities in successive camera images to be identified, thus enabling more reliable detection of track markers.

[0030] A first independent subject matter relates to a method for operating an environment detection system of a motor vehicle of the aforementioned type, wherein a light structure is projected by a projector onto an area of ​​a roadway in front of the motor vehicle, wherein the projector is operated in pulsed mode, wherein at least one camera takes pictures of the area in front of the motor vehicle, wherein at least one camera picture in a first pulse state is evaluated by the control system, wherein at least one camera picture in a second pulse state is evaluated by the control system, wherein differences between the camera pictures of the first pulse state and the second pulse state are detected and evaluated.

[0031] In at least two camera recordings, overlapping areas are identified and lane markings are detected in the camera recordings, whereby a change in the light structure is made when a confidence level for the detection of lane markings falls below a predetermined value.

[0032] Using the appropriate method, it is possible to reliably detect lane markings on roads, even when a light structure is projected onto the lane marking using a projector.

[0033] By recognizing overlapping areas, two different images can be aligned to each other, and the recognition of lane markings based on the different light structures is possible with higher reliability.

[0034] Then the procedure can only be carried out situationally if the lane marking cannot be reliably detected.

[0035] In a first further embodiment, it can be provided that, in a further further embodiment, the first pulse state prevails for 90% to 99% of the time in a given time interval, with the second pulse state prevailing for the remaining time.

[0036] This makes it possible to create a calm and steady-looking image of the light structure for the driver.

[0037] In a further, more advanced embodiment, it can be provided that a sampling rate of at least one camera is synchronized with a clock frequency of the projector.

[0038] If the clock frequency of the light timer is synchronized, i.e., an integer multiple of the clock frequency of the sampling rate timer, or vice versa, it is possible to synchronize the projector and the camera accordingly, so that, for example, an image is always recorded when the projector is not illuminated, thus ensuring that the image captured by the camera is not disturbed by the light structure generated by the projector. This allows for the identification of the track marker.

[0039] In a further, more advanced embodiment, the light timer and sampling rate timer can be synchronized. In particular, it can be provided that a camera image is always generated when the projector is not emitting a light structure.

[0040] In a further, more advanced embodiment, it can be provided that the sampling rate of at least one camera and the clock frequency of the projector are asynchronous.

[0041] If the corresponding clock frequencies are asynchronous, meaning they are not integer multiples of each other, the clock cycles overlap with varying degrees of overlap. This results in camera images where the light structure is visible at full brightness, not at all, and / or at a reduced brightness. This variance allows for the identification of differences and similarities in successive camera images, thus enabling more reliable detection of track markers.

[0042] A first independent object relates to a device for operating an environment detection system of the aforementioned type, wherein means for pulsed operation of the projector are provided, wherein a camera is provided to take pictures of the area in front of the motor vehicle, wherein a control is provided to evaluate at least one camera image in a first pulse state and at least one camera image in a second pulse state, wherein the control is configured to detect and evaluate differences between the camera image of the first pulse state and the camera image of the second pulse state.

[0043] Another independent subject matter relates to a computer program product, comprising a computer-readable storage medium on which instructions are embedded which, when executed by at least one controller of an environment detection system, cause the at least one controller to be configured to execute the procedure of the aforementioned type.

[0044] The process can be executed on one or more computing units, so that certain process steps are executed on one computing unit and other process steps on at least one other computing unit, whereby calculated data can be transmitted between the computing units if necessary.

[0045] Another independent item concerns a motor vehicle with an environment detection system of the type described above.

[0046] Further features and details are described below, in which – possibly with reference to the drawing – at least one embodiment is described in detail. Identical, similar, and / or functionally equivalent parts are marked with the same reference numerals. These are shown schematically: Fig. 1 a top view of a motor vehicle with an environmental sensing system; Fig. 2 a control unit of the motor vehicle's environmental sensing system. Fig. 1 Fig. 3: A top view of a road with the motor vehicle from Fig. 1 ; Fig. 4 two time-intensity diagrams of a first embodiment, and Fig. 5 two time-intensity diagrams of a second embodiment.

[0047] Fig. 1 shows a top view of a motor vehicle 2 with an environment detection system 4 (components of the environment detection system 4 are outlined with dashed lines).

[0048] The environmental detection system 4 indicates in Fig. 1 Invisible projectors are arranged in headlights 6 and 8. These projectors are designed to project linear structures with sufficiently high contrast onto an object so that they are perceptible and distinguishable by a driver. This provides assistance to the driver of vehicle 2 when changing lanes, indicating, for example, when a lane change is permissible (e.g., by a projected dashed line) and when it is not (e.g., by a projected solid line when a hazard has been detected). The environmental perception system can also indicate the location of lane markings, for example, in poor visibility conditions such as on snow-covered roads, and so on.

[0049] The environmental detection system 4 also has a forward-facing camera 10 that can detect an area of ​​a roadway in front of the motor vehicle 2.

[0050] The headlights 6, 8 and the camera 10 are connected to a controller 12, the controller 12 comprising a processing unit 14 and a memory 16. A computer program is stored in the memory 16, which, when loaded and executed by the processing unit 14, performs the procedure described herein.

[0051] Using camera 10, images of the road in front of vehicle 2 are taken and evaluated in the control unit 12 by recognizing lane markings recorded on the road. In an alternative embodiment, this evaluation can be performed in camera 10. Special hardware can be used for this purpose, for example, special graphics processors that are specialized for image analysis and object recognition.

[0052] The light structures generated by the projectors and projected onto the road, for example lines, can obscure the lane markings, so that the projections can prevent the lane markings from being recognized, at least according to the type of lane marking, by camera 10 or control unit 12.

[0053] In this case, it is possible to modify the light structures with respect to their physical properties so that track markings can be detected in the images captured by camera 10. In the embodiment presented here, a change between different light structures takes place at regular time intervals.

[0054] In alternative embodiments, it may be possible to change the physical properties of the light structure only if the confidence level of the detection of lane markings by the corresponding projected light structures drops too low, for example below a threshold of 95%.

[0055] Fig. 2 shows a schematic representation of the control system 12.

[0056] The modules described below can be program components of the aforementioned computer program product. In alternative configurations, the modules can be implemented wholly or partially by dedicated components, such as processors and / or circuit boards.

[0057] Camera 10 is connected to an evaluation module 18 of the control unit 12. The evaluation module 18 is used to analyze the image information and to recognize the corresponding lane markings.

[0058] Furthermore, a light modulation module 20 is provided, which controls a projector. The projector 22 is arranged in the headlight 6.

[0059] The projector 22 has a matrix of light sources 23, which can be individually switched on and off by the light modulation module 20. The corresponding light sources are dimmable and can therefore emit different brightness levels. In other configurations, the light sources can also be color-variable.

[0060] Furthermore, a timer 24 is provided, which is connected to the evaluation module 18 and the light modulation module 20.

[0061] Evaluation module 18, light modulation module 20, timer 24, and memory 16 are connected to the processing unit 14. Images captured by camera 10 are processed by evaluation module 18, and the evaluated images, enriched with object information about objects identified in the captured images, are transferred to the processing unit 14. The processing unit 14 temporarily stores this information in memory 16.

[0062] The computing unit 14 then triggers a change in the light structure by controlling the light modulation module 20 accordingly, causing the projector 22 to emit a modified light structure.

[0063] Camera 10 takes another image with the correspondingly modified light structure. The image is processed by evaluation module 18 as described above and transferred to the processing unit 14. This additional image can also be stored or temporarily stored in memory 16.

[0064] The processing unit 14 can then retrieve the data for the first recording and, if applicable, the second recording from memory 16 and compare the two data sets with each other.

[0065] This process compensates for any offset between the two images caused by the different shooting location resulting from the movement of vehicle 2. This is achieved by determining an offset based on the time interval between the two images and the speed of vehicle 2, taking into account the optical properties of camera 10. One of the two images is then shifted relative to the other by this offset. If necessary, image distortion can also be corrected.

[0066] In an alternative embodiment, the offset can be compensated for by image recognition.

[0067] The differences found in it allow the detection of the light structures and the differentiation of the light structures from the lane markings on the road, so that reliable detection of the lane markings is possible.

[0068] Timer 24 controls both camera 10 via evaluation module 18 and projector 22 via light modulation module 20. Timer 24 is therefore a master clock that enables synchronous or asynchronous operation of camera 10 and projector 22.

[0069] Fig. 3 shows a representation of motor vehicle 2 from Fig. 1 on a street 26.

[0070] Road 26 has a lane marking 28, which is detected by camera 10 of the environment detection system 4.

[0071] The motor vehicle 2 emits a cone of light 30 via the headlight 6 and a light structure 32, here a solid line, at the height of the lane marking 28 via the projector 22 arranged therein. Depending on the color and brightness, it may become impossible for the camera 10 to reliably recognize the type of lane marking 28 due to the light structure 32.

[0072] By varying the light structure 32 through the pulsed operation of the light source 23 of the projector 22, it can be achieved that the track marking 28 stands out from the light structure.

[0073] Fig. 4 Two intensity-time diagrams are shown according to a first embodiment. The two diagrams are shown one above the other.

[0074] The upper diagram represents a sampling rate of 34 for camera 10, and the lower diagram represents a pulse rate of 36 for projector 22. The pulse rate of 36 is an integer multiple of the sampling rate of 34.

[0075] Sampling rate 34 and pulse rate 36 are synchronized such that the pulses of the projector 22 always fall within a period in which the camera 10 is not sampling, so that the camera 10 always records an image that does not contain any part of the light structure 32, since the projector 22 does not emit light at the respective recording times.

[0076] Fig. 5 shows two intensity-time diagrams according to a second embodiment.

[0077] In Fig. 5 A pulse rate of 38 is asynchronous to a sampling rate of 34, and the ratio between the sampling rate and the pulse rate is not an integer. As a result, the camera takes 10 shots, some of which show no light structure, some show a light structure 32 at full brightness, and some show a light structure 32 at reduced brightness, since it is only visible for part of the recording time and therefore appears darker. From this, the controller 12 can detect the track marker 28.

[0078] Although the subject matter has been illustrated and explained in detail by means of exemplary embodiments, the invention is not limited by the disclosed examples, and other variations can be derived from them by a person skilled in the art. It is therefore clear that a multitude of possible variations exist. It is also clear that the exemplary embodiments mentioned are merely examples and are not to be interpreted in any way as limiting, for example, the scope of protection, the possible applications, or the configuration of the invention.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims. Reference symbol list

[0079] 2 Motor vehicle 4 Environment detection system 6 Headlights 8 Headlights 10 Camera 12 Control unit 14 Processing unit 16 Memory 18 Evaluation module 20 Light modulation module 22 Projector 23 Light source 24 Timer 26 Road 28 Lane marking 30 Light cone 32 Light structure 34 Sampling rate 36 Pulse rate 38 Pulse rate Intensity tTime

Claims

1. Environment detection system of a motor vehicle (2), comprising at least one projector (22) adapted to project linear light structures (32) onto a region of a roadway (36) in front of the motor vehicle (2), and at least one camera (10) arranged to detect a region in front of the motor vehicle (2), wherein the camera (10) is sensitive to at least a partial region of the frequency spectrum output by the at least one projector (22) in order to detect reflections of the light (32) projected by the projector (22) onto the region of the roadway, wherein the projector (22) comprises a modulatable illuminant arrangement (23), wherein a controller (12) is provided, which is connected to the at least one camera (10) and the at least one projector (22) is connected, wherein at least one timer (24) is provided for clocking the projector (22) and camera (10), wherein the projector (22) is operated in a controlled pulsed manner by the timer (24), wherein the controller (12) has an evaluation module (14) for camera data, which is adapted to evaluate camera data from a first pulse state and from a second pulse state separately from one another, characterised in that the evaluation module (14) is designed to identify overlapping regions in the at least two camera images and to recognise track markers (28) in the camera images, wherein the controller (12) is adapted to perform a change of the light structures (32) drops below a predetermined value for the recognition of track markings (28).

2. Environment detection system according to claim 1, wherein the controller (12) comprises a light modulation module (20) adapted to modulate the light emitted from the projector (22) so that the projector (22) can emit at least one first light structure (32) and at least one second light structure different from the first light structure (32), wherein switching between the first light structure (32) and the second light structure (32) is controlled by the timer (24).

3. Environment detection system according to any preceding claim, wherein a light timer (24) is provided for the projector (22) and a sample rate timer (24) is provided for the camera (10).

4. Environment detection system according to any preceding claim, wherein a clock frequency of said light timer (24) is an integer multiple of a clock frequency of said sampling rate timer (24) or a clock frequency of said sampling rate timer (24) is an integer multiple of a clock frequency of said light timer (24).

5. Environment detection system according to any one of claims 1 to 3, wherein a clock frequency of the light timer (24) is a non-integer multiple of a clock frequency of the sampling rate timer (24) or a clock frequency of the sampling rate timer (24) is a non-integer multiple of a clock frequency of the light timer (24).

6. Method for operating an environment recognition system of a motor vehicle (2) according to one of the preceding claims, wherein a light structure (32) with a projector (22) is projected onto a region of a roadway in front of the motor vehicle (2), wherein the projector (22) is operated in a pulsed manner, wherein images of the region in front of the motor vehicle (2) are taken by the at least one camera (10), wherein at least one camera image is evaluated in a first pulse state by the control (12), wherein differences between the camera images of the first pulse state and the second pulse state are recognised and evaluated, wherein overlapping regions in the at least two camera images are identified and track markings (28) are thereby recognised in the camera images, characterised in that a change in the light structures (32) is carried out when a level of confidence for the recognition of track markings (28) drops below a predetermined value.

7. Method according to claim 6, wherein in a given time interval the first pulse state prevails for 90% to 99% of the time, the second pulse state prevailing for the remaining time.

8. Method according to any one of claims 6 and 7, wherein a sampling rate of the at least one camera (10) with a clock frequency of the projector (22) is synchronised.

9. Method according to any one of claims 6 to 8, wherein a sampling rate of the at least one camera (10) and a clock frequency of the projector (22) are asynchronous.

10. Computer program product comprising a computer readable storage medium (16) on which are embedded instructions which, when executed by at least one controller (12) of an environment recognition system according to claim 1, cause the at least one controller (12) to be adapted to execute the method according to any one of claims 6-9.

11. Motor vehicle with an environment recognition system according to any one of claims 1 to 5.