Method for evaluating an image taken by a camera of a vehicle and image processing device

Abstract

Method (600) for evaluating an image (300) recorded by a camera (110) of a vehicle (100), the method (600) comprising the following steps: - reading (610) the image (300) of the camera (110); - detecting (620) at least one object (305, 315) in a partial section (310, 320) of the image (300), wherein the partial section (310, 320) depicts a smaller area of ​​the vehicle surroundings (115) than the image (300); - transmitting (630) the partial section (310, 320) and / or information about the partial section (310, 320) of the image (300) to a driver assistance module (140, 150, 160); and - Using (540) the partial section (310, 320) of the image (300) and / or the information about the partial section (310, 320) of the image (300) instead of the entire image (300) in the driver assistance module (140, 150, 160) to evaluate the image (300) recorded by the camera (110); characterized in that in the detection step (620), at least one self-luminous object is differentiated from at least one externally illuminated object in order to define the partial section (310) of the image (300).

Description

State of the art

[0001] The present invention relates to a method for evaluating an image recorded by a camera of a vehicle, to a corresponding image processing device and to a corresponding computer program product according to the main claims.

[0002] The growing complexity of the requirements placed on the various video-based systems in vehicles necessitates the necessary "cooperation" between the individual systems. For example, it is known that lane detection information can be used for online calibration, the results of vehicle detection can be used to reduce the number of misclassifications caused by speed signs on trucks, for example, in the traffic sign recognition system, and so on. The number of cooperations between different systems mentioned here, which are only examples, can be expanded as desired.

[0003] DE 10 2007 034 196 A1 discloses a method for lane detection using a driver assistance system of a vehicle comprising a sensor system for lane detection. In this method, the sensor system for lane detection detects measuring points representing lane markings in a traffic area in front of the vehicle. Based on a reference model, an ideal number of measuring points MP is determined. A plausibility measure of the look-ahead distance is determined from a comparison between the number of actually detected measuring points and the ideal number of measuring points.

[0004] The document DE 10 2005 045 077 A1 discloses a method and a device for distributed analysis of images.

[0005] The document DE 10 2009 012 758 A1 discloses a device and a method for detecting at least one object. Disclosure of the invention

[0006] Against this background, the present invention presents a method for evaluating an image captured by a camera of a vehicle, an image processing device that uses this method, and finally a corresponding computer program product according to the main claims. Advantageous embodiments emerge from the respective subclaims and the following description.

[0007] The present invention provides a method for evaluating an image taken by a camera of a vehicle, the method comprising the following steps: - Reading the image from the camera; - Detecting at least one object in a partial section of the image, wherein the partial section depicts a smaller area of ​​the vehicle's surroundings than the image; - transmitting the partial section and / or information about the partial section of the image to a driver assistance module, wherein in particular the driver assistance module implements a driver assistance function that differs from a sole evaluation of the image recorded by the camera; and - Using the partial section of the image and / or the information about the partial section of the image instead of the entire image in the driver assistance module to evaluate the image recorded by the camera.

[0008] The present invention further provides an image processing device designed to perform or implement steps of the method according to the invention in corresponding devices. This embodiment of the invention in the form of a device also allows the object underlying the invention to be achieved quickly and efficiently.

[0009] A device, in particular an image processing device, can be understood here as an electrical device that processes sensor signals and outputs control signals depending on them. The device can have an interface that can be implemented in hardware and / or software. In a hardware implementation, the interfaces can, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the device. However, it is also possible for the interfaces to be separate integrated circuits or to consist at least partially of discrete components. In a software implementation, the interfaces can be software modules that are present, for example, on a microcontroller alongside other software modules.

[0010] Also advantageous is a computer program product with program code that can be stored on a machine-readable medium such as a semiconductor memory, a hard disk memory or an optical memory and is used to carry out steps of the method according to one of the embodiments described above when the program is executed on an image processing device or a device corresponding to a computer.

[0011] A camera can be understood as an optical recording unit for capturing a vehicle's surroundings in the spectral range of visible light. An object that is detected in a camera image can, for example, be a structural feature at the edge of the road (such as a delineator post or a street lamp) or on the road (such as a road marking) or even a moving object on the road (such as a vehicle). The camera image can be recorded in particular in the dark or at night, with the objects being characterized by a significantly increased brightness compared to the image background or other image areas. A partial section can be understood as a sub-area of ​​the image that depicts a smaller area of ​​the vehicle's surroundings than the camera image.In particular, a partial section can be considered to be an area of ​​the image which does not include at least one other section of the image. The partial section of the image forms an area around the object or objects in the vehicle’s surroundings recognised as one or more “objects” in the camera image. The partial section of the image can comprise an area around the object or objects recognised as “object(s)”. Information about the partial section of the image can be understood, for example, as a position and / or size of the section in which the object(s) are recognised in the image. Furthermore, the information about the partial section of the image can also relate to a parameter of the object, such as brightness and / or colour.In addition, the content of the camera image in the corresponding partial section can of course also be transmitted to the driver assistance module as information about the partial section. A driver assistance module can be understood as a device that performs a driver assistance function to relieve the driver of the vehicle when driving the vehicle. For example, a driver assistance function can implement automatic recognition of traffic signs, lane detection and / or multiple object detection in order to be able to detect one or more vehicles in front of the vehicle. The driver assistance module can then provide an output to the driver of the vehicle to make it easier for them to control the vehicle, for example by maintaining the lane the vehicle is currently in, even when cornering.The driver assistance function comprises at least one further functionality which differs from the sole examination and division of the camera image into different partial sections of the image recorded by the camera, such as the evaluation of the content of traffic signs and, in response thereto, the output of an information signal to the driver of the vehicle, the detection and extrapolation of a lane from detected lane markings and, in response thereto, the output of a lane following signal or the detection of braking of vehicles ahead and, in response thereto, the output of a warning signal or the output of an automatic braking signal to brake the vehicle.

[0012] The present invention is based on the realization that only a partial section of the image or information about the partial section, rather than the entire camera image, needs to be evaluated in the driver assistance module in order to actually utilize the driver assistance module or the function of the respective driver assistance module optimally. Essentially, an algorithm of the driver assistance module is based on the fact that a sometimes very large camera image must be searched for often small objects in order to provide the driver with a corresponding assistance function.If a rough analysis of the image for objects contained therein (particularly using an algorithm specifically optimized for object search) is performed at an early stage and one or more partial sections of the image are identified in which objects were detected, this partial section(s) can be transmitted to the relevant driver assistance module. The driver assistance module then only needs to examine a small section of the image, advantageously reducing the numerical or circuit-related data processing load in this driver assistance module.In the event that several different driver assistance modules are supplied with such a pre-processed camera image, i.e. a partial section relevant for the respective driver assistance module, this can result in a saving in computing effort in several of the driver assistance modules concerned, which results on the one hand in faster processing time and on the other hand in lower power consumption of the driver assistance modules concerned.

[0013] According to an advantageous embodiment of the present invention, a size and / or a position of the partial section in the image can be transmitted as information in the transmission step. A size and / or a position of the partial section of the image can represent a shape of the partial section of the image, such as a rectangular shape, an oval shape, or an orientation of the partial section, for example by specifying an angle of the partial section with respect to an upper or lower edge of the camera image. A position of the partial section of the image can, for example, represent coordinates of a corner point of the partial section in the camera image. Such an embodiment of the present invention offers the advantage that a position of the partial section in the camera image can be transmitted with very few parameters.

[0014] It is further advantageous if, in the detection step, a color and / or brightness of the object is detected, and in the transmission step, the detected color and / or brightness is further transmitted to the driver assistance module as information. Brightness can be understood as a luminosity and / or a pulsation or pulse frequency of the light of an area of ​​the camera image detected as an object. Color of the object can be understood as a spectral frequency of the light of the area of ​​the camera image detected as an object. Such an embodiment of the present invention offers the advantage that additional information can be transmitted as parameters to the relevant driver assistance module in a technically simple manner.

[0015] It is particularly advantageous if, in the recognition step, at least one traffic sign is recognized as an object in the partial section of the image, and in the transmission step, the partial section and / or the information about the partial section are transmitted to a traffic sign recognition module as a driver assistance module. Such an embodiment of the present invention offers the advantage that, as early as during the evaluation of the camera image, an object can be recognized as a traffic sign and fed to a recognition module specifically designed for recognizing traffic signs as a driver assistance module. A complete search of the entire camera image by the traffic sign recognition module with a correspondingly complex recognition algorithm can thus be avoided.

[0016] Furthermore, in the recognition step, at least one self-luminous object is distinguished from at least one externally illuminated object in order to determine the partial section of the image. A self-luminous object can be an object that actively emits light. Such a self-luminous object can be, for example, a headlight or a pair of headlights of an oncoming vehicle. In contrast, an externally illuminated object can be, for example, a traffic sign. Such an embodiment of the present invention offers the advantage of being able to distinguish, for example from the analysis of the spectral composition of the light detected by the camera, whether an object itself emits light or is illuminated (for example, by the headlights of the vehicle's own vehicle).Such differentiation enables early detection and / or classification of objects, allowing these objects to be fed to different driver assistance modules. For example, a section of the camera image in which a self-illuminating object was detected can be fed to a headlight control module, whereas a section of the camera image in which an externally illuminated object was detected can be fed to a lane detection module. These objects can be interpreted as guideposts, for example.

[0017] Furthermore, according to another embodiment of the present invention, in the detection step, at least one guide post and / or at least one street lighting device can be detected as an object in the partial section of the image, and in the transmission step, the partial section and / or the information about the partial section can be transmitted to a lane detection module and / or a lane guidance module and / or a lighting control system as a driver assistance module. Such an embodiment of the present invention offers the advantage of particularly favorable preprocessing of the read-in camera image, so that a lane detection module and / or a lane guidance module for the vehicle have to bear a lower processing load.A lane detection module can be understood as a driver assistance function in which the lane the vehicle will be traveling in is estimated in advance and which may support the driver in maintaining this lane by sending appropriate control signals to other vehicle components. A lane guidance module can be understood as a driver assistance function in which the vehicle is actively kept in a desired lane on the road.

[0018] In order to achieve a particularly low data processing load for a lane detection module, a lane guidance module, and / or a lighting control system, according to another embodiment of the present invention, in the recognition step, at least the taillights of a preceding vehicle can be recognized as an object in the partial section of the image, and in the transmission step, the partial section and / or the information about the partial section can be transmitted to a lighting control system and / or a lane detection module and / or a lane guidance module as a driver assistance module. The taillights of a preceding vehicle can be recognized as such, for example, based on their color and / or their distance from one another. In this case, the lighting control system can, for example, align a light cone of the vehicle in such a way that one or more preceding vehicles, whose taillights were recognized as an object, for example, are not dazzled.Alternatively or additionally, a lane detection module can also determine a lane course not illuminated by the headlights of the vehicle itself from a partial section of the camera image, in particular from a position of an object that has been classified as the taillights of a vehicle ahead.

[0019] The advantage of using the method proposed above is particularly great if, according to one embodiment of the present invention, an image that was taken by the camera in the dark is read in during the reading step. Darkness is understood to mean a brightness of the vehicle's surroundings that makes it necessary to drive with the headlights switched on (for example, in accordance with the road traffic regulations). Such an embodiment of the present invention offers the advantage that the evaluation capability of the driver assistance module is significantly increased, since the detection of objects is very difficult, especially in the dark. If a preliminary evaluation orIf the camera image is pre-processed using an algorithm specifically designed for camera images taken in the dark, this can significantly reduce the data processing load of the respective driver assistance module(s) to which the partial section is transmitted.

[0020] The approach described above can be used particularly advantageously if not just a single driver assistance module is supplied with a partial section of the camera image, but if several different driver assistance modules each receive their own partial section of the camera image that includes the detected objects relevant to the respective driver assistance function. In this way, a central, uniform preprocessing of the camera image can achieve a significant reduction in the data processing load in the downstream driver assistance modules, which further increases the efficiency of the approach proposed here. Thus, according to a further embodiment of the present invention, in the detection step, at least one further object can be detected in a further partial section of the image, wherein the further partial section depicts a smaller area of ​​the vehicle's surroundings than the image.Furthermore, in the transmitting step, the further partial section and / or further information about the further partial section of the image is transmitted to a further driver assistance module, wherein the further driver assistance module is different from the driver assistance module. Furthermore, in the using step, the further partial section is used instead of the entire image in the further driver assistance module to evaluate the image captured by the camera.

[0021] The invention is explained in more detail below with reference to the accompanying drawings. They show: Fig. 1 is a block diagram of a vehicle in which an embodiment of the present invention is used; Fig. 2 shows a block diagram of an embodiment of the image evaluation device according to the invention; Fig. 3 shows an embodiment of a camera image as it is taken by the camera Fig. 1 taken while looking at a vehicle environment; Fig. 4 shows another embodiment of a camera image as it is taken by the camera Fig. 1 taken while looking at a vehicle environment; Fig. 5 shows another embodiment of a camera image as it is taken by the camera Fig. 1 was taken while looking at a vehicle environment; and Fig. 6 a flowchart of an embodiment of the present invention as a method.

[0022] In the following description of preferred embodiments of the present invention, the same or similar reference numerals are used for the elements shown in the various figures and having a similar effect, whereby a repeated description of these elements is omitted.

[0023] Fig. 1 shows a block diagram of a vehicle 100 in which an exemplary embodiment of the present invention is used. The vehicle 100 comprises a camera 110 that captures a vehicle environment 115 from a viewing angle 120. In particular, the camera 110 can be configured to detect items and / or objects in the vehicle environment 115, particularly in the dark. The image captured by the camera 110 is subsequently evaluated in an image processing unit or image conditioning unit 130, wherein objects in the camera image are recognized and partial sections containing these objects are formed. These partial sections or information about these partial sections can then be transmitted to various driver assistance modules, such as a traffic sign recognition module 140, a lane detection module 150, or a light control module 160 for controlling the headlights of the vehicle 100.For example, the image processing unit 130 can use certain geometric shapes and / or colors in the image captured by the camera 110 to identify that traffic signs are present in a specific section of the image captured by the camera 110. A section containing such traffic signs in the camera image is then transmitted, for example, to the traffic sign recognition module 140. In another section of the camera image, reflectors of guide posts, for example, are detected, so that this section of the camera image is transmitted, for example, to the lane detection module 150.If, for example, headlights of an oncoming vehicle are recognized as an object, an image section of the camera image, which includes these objects recognized as headlights, can be sent to the light control module 160, which aligns the headlights or a light cone of the headlights in such a way that a driver of the oncoming vehicle is not dazzled.

[0024] The Fig. Figure 2 shows a block diagram of an exemplary embodiment of the image processing device or unit 130 according to the invention. The image processing device 130 comprises an interface 200 for reading the image from the camera 110. Furthermore, the image processing device 130 comprises a recognition unit 210 for recognizing at least one object in a partial section of the image. In addition, the image processing device 130 comprises a transmission interface 220 for transmitting the partial section and / or information about the partial section of the image to a driver assistance module. This driver assistance module can be, for example, the traffic sign recognition module 140, the lane detection module 150, or the light control module 160.

[0025] The approach proposed here refers in particular to the provision and consistent use of information currently only used internally in the “Vehicle Detection at Darkness” (VDD) system for the definition or restriction of search areas of other systems such as traffic sign recognition or lane detection assistance.

[0026] This makes the VDD system comparable to a so-called basic function at night, upon whose results further functions and systems can be built. By linking the VDD function—in particular the detection of objects in the vehicle's surroundings in the dark—with other (driver assistance) functions, the performance of the other (driver assistance) functions can be significantly increased, at least at night (i.e., generally in the dark) or in situations where vehicles are driving with their exterior lights switched on.

[0027] The use of additional information such as the position of classified traffic signs or their pulsing (i.e., information about pulsed / non-pulsed light objects in the vehicle's surroundings) in variable message signs can be advantageously used in the traffic sign recognition system to reliably detect traffic signs even at great distances (e.g., greater than 150 m) and to adapt a classification strategy for these signs accordingly. VDD can also classify objects recognized as traffic signs as "self-luminous" or "illuminated by their own headlights." The positions of the detected guide posts can be advantageously used in the lane detection system to significantly restrict the search area for the lanes or to stabilize or verify the plausibility of existing detections.Streetlights classified as objects by the VDD system are also of interest for lane detection during the color evaluation of the lane lines, since streetlights or tunnel lighting, for example, significantly complicate the yellow classification of lane lines. Any exclusion of image areas (i.e., partial sections of the camera image) provided by VDD for other (driver assistance) functions ultimately leads to a reduction in the overall system's computing time.

[0028] The Fig. 3 shows an embodiment of a camera image 300 as it is taken by the camera 110 Fig. 1 was taken looking at a vehicle environment. An example is shown in the Fig. 3 the detection of variable message signs 305 based on the color (usually red) and / or their pulsation when using LEDs of the variable message signs 305 by the VDD function. These variable message signs are detected in two partial sections 310 of the camera image 300, which are then forwarded to the traffic sign recognition mode 140 according to the Fig. 1 can be transmitted. The transmission can take place in such a way that the partial sections 310 themselves or only information about the size and position of the partial sections 310 in which the objects recognized as "traffic signs" are contained is transmitted to the traffic sign recognition module 140. The size of the partial areas or partial sections 310 can be selected such that a specific section with a border of a predefined size around the objects recognized as traffic signs is considered a partial section. A shape of the partial section, such as a rectangular shape, can also be selected in order to ensure the best possible processing of the partial sections in the driver assistance module.This enables early detection of traffic signs without the traffic sign recognition module 140 having to analyze the entire camera image 310, which requires significant numerical or circuitry effort. The partial sections 310 each comprise an area of ​​the camera image 300 that is smaller than the entire camera image 200.

[0029] Another example of the formation of partial sections from the camera image 300 is the detection of guidance posts, lateral lane markings 315 or even BotsDots (e.g. in tunnels or as road markings) by the VDD function (i.e. the image evaluation device 130), for example based on their color (usually yellow) of such detected objects or as an "externally illuminated object" in front of the vehicle. Such detection of structural measures in the vehicle environment 115 is also included in the Fig. 1, wherein the partial sections of the camera image that include these reflective elements are considered as further partial sections 320, which are, for example, sent to the lane detection module 150 according to Fig. 1. A lane detection function could thus intelligently adapt its own search areas based on this additional information.

[0030] The Fig. 4 shows a further embodiment of a camera image 300, which a camera 110 according to Fig. 1 of a vehicle environment. In this case, several traffic signs 305 are again detected, which are sent as partial sections 310, for example, to the traffic sign recognition module 140 according to the Fig. 1. It is also possible to create several different, separate partial sections 310 in which objects are recognized as traffic signs, with these individual or separate partial sections 310 both being transmitted to the traffic sign recognition module 140. Thus, not only do different partial sections have to be transmitted to different driver assistance modules; rather, several different partial sections can also be transmitted to a single driver assistance module.

[0031] The Fig. 5 shows a further embodiment of a camera image 300, which a camera 110 according to Fig. 1 of a vehicle's surroundings. Guide posts 315 are again visible, whereby, according to the approach presented above, a corresponding partial section 320 is formed, which has these guide posts 315 and which can be sent, for example, to the light control module 160 for better illumination of the curved roadway using cornering lights.

[0032] The Fig. 6 shows a further embodiment of a camera image 300, which a camera 110 according to Fig. 1 of a vehicle's surroundings. Guide posts 315 are again visible, whereby, according to the approach presented above, a corresponding partial section 320 is formed, which has these guide posts 315 and which can be sent, for example, to the light control module 160 for better illumination of the curved roadway using cornering lights.

[0033] Fig.6 shows a flowchart of an exemplary embodiment of the present invention as a method 600 for evaluating an image recorded by a camera of a vehicle. The method 600 comprises a step of reading 610 the image from the camera and a step of detecting 620 at least one object in a partial section of the image, wherein the partial section depicts a smaller area of ​​the vehicle's surroundings than the image. The method further comprises a step of transmitting 630 the partial section and / or information about the partial section of the image to a driver assistance module, in particular one that implements a driver assistance function that differs from solely evaluating the image recorded by the camera.Finally, the method 600 comprises a step of using 640 the partial section of the image and / or the information about the partial section of the image instead of the entire image in the driver assistance module in order to evaluate the image recorded by the camera.

[0034] The embodiments described and shown in the figures are selected only as examples. Different embodiments can be combined with each other completely or with regard to individual features. Furthermore, one embodiment can be supplemented by features of another embodiment.

[0035] Furthermore, method steps according to the invention can be repeated and carried out in a different order than that described.

[0036] If an embodiment comprises an “and / or” link between a first feature and a second feature, this can be read such that the embodiment according to one embodiment has both the first feature and the second feature and according to another embodiment has either only the first feature or only the second feature.

Claims

[1] Method (600) for evaluating an image (300) recorded by a camera (110) of a vehicle (100), the method (600) comprising the following steps: - reading (610) the image (300) of the camera (110); - detecting (620) at least one object (305, 315) in a partial section (310, 320) of the image (300), wherein the partial section (310, 320) depicts a smaller area of ​​the vehicle surroundings (115) than the image (300); - transmitting (630) the partial section (310, 320) and / or information about the partial section (310, 320) of the image (300) to a driver assistance module (140, 150, 160); and - using (540) the partial section (310, 320) of the image (300) and / or the information about the partial section (310, 320) of the image (300) instead of the entire image (300) in the driver assistance module (140, 150, 160) in order to evaluate the image (300) recorded by the camera (110); characterized bythat in the step of recognition (620) at least one self-luminous object is distinguished from at least one externally illuminated object in order to define the partial section (310) of the image (300). [2] Method (600) according to claim 1, characterized by that in the step of transmitting (630) a size and / or a position of the partial section (310, 320) in the image (300) is transmitted as information. [3] Method (600) according to one of the preceding claims, characterized by in that in the step of recognition (620) various features of the object (305, 315) are recognized and in the step of transmission (630) the extracted features are further transmitted as information to the driver assistance module (140, 150, 160). [4] Method (600) according to one of the preceding claims, characterized byin that in the step of recognition (620) at least one traffic sign (305) is recognized as an object in the partial section (310) of the image (300) and in the step of transmission (630) the partial section (310) and / or the information about the partial section (310) is transmitted to a traffic sign recognition module (140) as a driver assistance module. [5] Method (600) according to one of the preceding claims, characterized by in that in the step of recognition (620) at least one guide post (305) and / or at least one street lighting device is recognized as an object in the partial section (310) of the image (300) and in the step of transmission (630) the partial section (310) and / or the information about the partial section (310) is transmitted to a lane detection module (150) and / or a lane guidance module and / or light control system (160) as a driver assistance module. [6] Method (600) according to one of the preceding claims, characterized byin that in the step of recognition (620) at least rear lights of a vehicle driving ahead are recognized as an object in the partial section of the image (300) and in the step of transmission (630) the partial section (310) and / or the information about the partial section is transmitted to a light control system (160) and / or a lane detection module (150) and / or a lane guidance module as a driver assistance module. [7] Method (600) according to one of the preceding claims, characterized by that in the step of reading (610) an image (300) is read in by the camera (110) which was taken by the camera (110) in the dark. [8] Method (600) according to one of the preceding claims, characterized bythat in the step of recognizing (620) at least one further object (315) in a further partial section (320) of the image (300), wherein the further partial section (320) depicts a smaller area of ​​the vehicle surroundings (115) than the image (300), wherein in the step of transmitting (630) the further partial section (320) and / or further information about the further partial section (320) of the image (300) to a further driver assistance module (150, 160) takes place, in particular wherein the further driver assistance module (150, 160) differs from the driver assistance module (140), and wherein in the step of using (640) the further partial section (320) instead of the entire image (320) in the further driver assistance module (150, 160) takes place in order to to evaluate. [9] Image processing device (130) for processing an image (300) recorded by a camera (110) of a vehicle (100), wherein the image evaluation device (130) has devices (200, 210, 230) which are designed to carry out steps of a method (600) according to one of claims 1 to 8. [10] Computer program product with program code for carrying out steps of the method (600) according to one of claims 1 to 8 when the program is executed on an image processing device (130).

Images